Tag: tools

Fixing Issues after upgrading Proxmox 7 to 8

Flattr this!

My initial plan was to update all of my Proxmox nodes to the latest version by the end of this year. While most updates proceeded smoothly, I encountered two errors on one particular node.
Given that updating servers is a critical operation, especially when they are only remotely accessible via the network, I decided to document these errors and their solutions for future reference.

Proxmox Host does not come online again, after the reboot due to an update

The first issue arose after the mandatory reboot; the server failed to restart. Upon requesting a remote console connection, the boot process stalled with the following error message:

Boot stuck on “A start job is running for Network initialization (XXm / no limit)”

After consulting various posts on the Proxmox forum, I initially suspected a need to update my network configuration. However, my attempts proved unsuccessful, leading to multiple reboots into rescue mode.

Fortunately, I had the insight to consult the official “Proxmox upgrade 7 to 8” guide, where I ultimately discovered the solution to my issue:

Network Setup Hangs on Boot Due to NTPsec Hook

It appears that a bug may lead to the simultaneous installation of both ntpsec and ntpsec-ntpdate. This, in turn, causes the network to fail during boot, resulting in a hang.

The resolution involves disabling the ntpsec-ntpdate start script using the command chmod -x /etc/network/if-up.d/ntpsec-ntpdate and then rebooting, successfully resolving the issue.”

A container does not start and shows the error

The next issues happens with some containers, that don’t want to startup anymore.
The Proxmox UI displays the following error:

run_buffer: 322 Script exited with status 255  
lxc_init: 844 Failed to run lxc.hook.pre-start for container "105"  
__lxc_start: 2027 Failed to initialize container "105"  
TASK ERROR: startup for container '105' failed

After I started the container manually via the terminal, I got a more specific error:

root /etc/pve/lxc # lxc-start -n 105

lxc-start: 105: ../src/lxc/lxccontainer.c: wait_on_daemonized_start: 870 No such file or directory - Failed to receive the container state

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 309 To get more details, run the container in foreground mode

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

root /etc/pve/lxc # lxc-start -n 105 -F

lxc-start: 105: ../src/lxc/conf.c: run_buffer: 322 Script exited with status 255

lxc-start: 105: ../src/lxc/start.c: lxc_init: 844 Failed to run lxc.hook.pre-start for container "105"

lxc-start: 105: ../src/lxc/start.c: __lxc_start: 2027 Failed to initialize container "105"

lxc-start: 105: ../src/lxc/conf.c: run_buffer: 322 Script exited with status 1

lxc-start: 105: ../src/lxc/start.c: lxc_end: 985 Failed to run lxc.hook.post-stop for container "105"

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

Trying to mount the container disk also produces some more errors:

root /etc/pve/lxc # pct mount 105

mount: /var/lib/lxc/105/rootfs: wrong fs type, bad option, bad superblock on /dev/loop17, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
mounting container failed
command 'mount -o noacl /dev/loop17 /var/lib/lxc/105/rootfs//' failed: exit code 32

So I initially though, that the filesystem might be corrupt, so I also did try to check it:

root /etc/pve/lxc # pct fsck 105

fsck from util-linux 2.38.1
/var/lib/vz/images/105/vm-105-disk-1.raw: clean, 373288/4194304 files, 8047185/16777216 blocks

[  713.133949] loop17: detected capacity change from 0 to 134217728
[  713.137988] ext4: Unknown parameter 'noacl'

The last info provided me with the right clue:
It seems, that with proxmox 8, that container config did change slightly. Re-setting the Disk ACL to default did eventually work.


After that, the container was able to startup again

Fixing Error “failed, reason: getaddrinfo EAI_AGAIN” in Gitlab Builds

Flattr this!

Some days ago, I detected some new errors in one of our builds pipelines. The interesting part is, that there were no changes done, that might have caused this error.
One example for this error was an DNS resolution error using npm:

#9 [4/5] RUN npm i npm@latest
#9 71.06 npm ERR! code EAI_AGAIN
#9 71.06 npm ERR! syscall getaddrinfo
#9 71.06 npm ERR! errno EAI_AGAIN
#9 71.06 npm ERR! request to https://registry.npmjs.org/npm failed, reason: getaddrinfo EAI_AGAIN registry.npmjs.org
#9 71.06 
#9 71.06 npm ERR! A complete log of this run can be found in:
#9 71.06 npm ERR!     /root/.npm/_logs/2023-02-10T08_30_43_730Z-debug-0.log
#9 ERROR: executor failed running [/bin/sh -c npm i npm@latest]: runc did not terminate sucessfully

After some digging, I found out, that the only update might have come with an updated docker build behaviour, since the docker image, used in this job is docker:latest.

Some Google queries later, I found an issue realted to this topic. It points to a blog post about a strange behaviour of docker in regards to DNS. But since, I had no issues before, I ruled out, that this might be the case here.

And indeed, a look in the latest release notes pointed out the obvious:

Alias docker build to docker buildx build

docker/cli#3314

The easiest fix was to disable buildx alltogether via these ENV Vars:

build:
  image: docker:latest
  stage: build
  variables:
    DOCKER_BUILDKIT: 0
    COMPOSE_DOCKER_CLI_BUILD: 0
  services:
    - docker:dind
  before_script:
…
  script:
…

After this change, the build job did run without any errors.

Create CSV Reports from GIT Repositories containing your commits

Flattr this!

Some months ago, I got the need to run over several GIT Repositories and collect the work I did on each day. The play was to gather all the data and collect them in different CSV files.
Since I wasn’t able to find a ready script for this task, I guess it is a good candidate for a quick blog post :-).

The first part is a file folders.txt with a list of all GIT Repositories that we want to analyse. (All folders are Subfolders of a root Directory /Users/user/GIT. This root folder can be changed later on.)

cat folders.txt

tools
utils
customer1/project2
customer2/project1
customer2/project2
customer3/project1

The script does several things:

  1. Going through every repository and collecting Project,Date,Commit,Name,Email,Comment of each commit.
  2. We also need to do some filtering first to deal with characters in the commit messages, that might break the CSV later on.
  3. The last step is to split the complete log file into the different months.

At the moment the script does only run for one specific year, but that can be changed with adding another loop to run it for a bunch of years.

The Source of the Script is:

#!/bin/bash

#YEAR=$(date +"%Y")
HEADER=Project,Date,Commit,Name,Email,Comment
YEAR=2022
ROOT=$(PWD)
GIT_ROOT=$HOME/GIT
PROJECTS=$(cat folders.txt)
TMP_DIR=/tmp/csv
CREATOR="Philipp Haussleiter"
echo "" > /tmp/csv/all.csv
mkdir -p csv/$YEAR $TMP_DIR
rm -Rf $TMP_DIR/*

for PROJECT in ${PROJECTS}; do
    echo "creating log of ${PROJECT}"
    DIR=${GIT_ROOT}/${PROJECT}
    BASENAME=$(basename $DIR)
    cd ${DIR}
    git log --pretty=format:__${BASENAME}__,__%cs__,__%h__,__%an__,__%ae__,__%s__ > /tmp/csv/${BASENAME}.a.log
    cat /tmp/csv/${BASENAME}.a.log | sed -r 's/[\"]+/\"\"/g' > /tmp/csv/${BASENAME}.b.log
    cat /tmp/csv/${BASENAME}.b.log | sed -r 's/__+/\"/g' > /tmp/csv/${BASENAME}.log
    echo "" >> /tmp/csv/${BASENAME}.log
    cat /tmp/csv/${BASENAME}.log >> /tmp/csv/all.csv
    rm /tmp/csv/${BASENAME}.a.* /tmp/csv/${BASENAME}.b.*
    cd ${ROOT}
done

for MONTH in $(seq -f "%02g" 1 12); do
    FILE=csv/$YEAR/${YEAR}-${MONTH}.csv
    FILTER=${YEAR}-${MONTH}
    echo $HEADER > $FILE
    cat /tmp/csv/all.csv |grep "$CREATOR" |grep $FILTER >> $FILE
    echo $HEADER > csv/$YEAR/all.csv
    cat /tmp/csv/all.csv |grep "$CREATOR" >> csv/all.csv
    echo $FILE
done

After running the script for the years 2021 and 2022, you get a folder structure like this:

csv
├── 2021
│   ├── 2021-01.csv
│   ├── 2021-02.csv
│   ├── 2021-03.csv
│   ├── 2021-04.csv
│   ├── 2021-05.csv
│   ├── 2021-06.csv
│   ├── 2021-07.csv
│   ├── 2021-08.csv
│   ├── 2021-09.csv
│   ├── 2021-10.csv
│   ├── 2021-11.csv
│   └── 2021-12.csv
└── 2022
    ├── 2022-01.csv
    ├── 2022-02.csv
    ├── 2022-03.csv
    ├── 2022-04.csv
    ├── 2022-05.csv
    ├── 2022-06.csv
    ├── 2022-07.csv
    ├── 2022-08.csv
    ├── 2022-09.csv
    ├── 2022-10.csv
    ├── 2022-11.csv
    └── 2022-12.csv