Fixing Issues after upgrading Proxmox 7 to 8

Flattr this!

My initial plan was to update all of my Proxmox nodes to the latest version by the end of this year. While most updates proceeded smoothly, I encountered two errors on one particular node.
Given that updating servers is a critical operation, especially when they are only remotely accessible via the network, I decided to document these errors and their solutions for future reference.

Proxmox Host does not come online again, after the reboot due to an update

The first issue arose after the mandatory reboot; the server failed to restart. Upon requesting a remote console connection, the boot process stalled with the following error message:

Boot stuck on “A start job is running for Network initialization (XXm / no limit)”

After consulting various posts on the Proxmox forum, I initially suspected a need to update my network configuration. However, my attempts proved unsuccessful, leading to multiple reboots into rescue mode.

Fortunately, I had the insight to consult the official “Proxmox upgrade 7 to 8” guide, where I ultimately discovered the solution to my issue:

Network Setup Hangs on Boot Due to NTPsec Hook

It appears that a bug may lead to the simultaneous installation of both ntpsec and ntpsec-ntpdate. This, in turn, causes the network to fail during boot, resulting in a hang.

The resolution involves disabling the ntpsec-ntpdate start script using the command chmod -x /etc/network/if-up.d/ntpsec-ntpdate and then rebooting, successfully resolving the issue.”

A container does not start and shows the error

The next issues happens with some containers, that don’t want to startup anymore.
The Proxmox UI displays the following error:

run_buffer: 322 Script exited with status 255  
lxc_init: 844 Failed to run lxc.hook.pre-start for container "105"  
__lxc_start: 2027 Failed to initialize container "105"  
TASK ERROR: startup for container '105' failed

After I started the container manually via the terminal, I got a more specific error:

root /etc/pve/lxc # lxc-start -n 105

lxc-start: 105: ../src/lxc/lxccontainer.c: wait_on_daemonized_start: 870 No such file or directory - Failed to receive the container state

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 309 To get more details, run the container in foreground mode

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

root /etc/pve/lxc # lxc-start -n 105 -F

lxc-start: 105: ../src/lxc/conf.c: run_buffer: 322 Script exited with status 255

lxc-start: 105: ../src/lxc/start.c: lxc_init: 844 Failed to run lxc.hook.pre-start for container "105"

lxc-start: 105: ../src/lxc/start.c: __lxc_start: 2027 Failed to initialize container "105"

lxc-start: 105: ../src/lxc/conf.c: run_buffer: 322 Script exited with status 1

lxc-start: 105: ../src/lxc/start.c: lxc_end: 985 Failed to run lxc.hook.post-stop for container "105"

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

Trying to mount the container disk also produces some more errors:

root /etc/pve/lxc # pct mount 105

mount: /var/lib/lxc/105/rootfs: wrong fs type, bad option, bad superblock on /dev/loop17, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
mounting container failed
command 'mount -o noacl /dev/loop17 /var/lib/lxc/105/rootfs//' failed: exit code 32

So I initially though, that the filesystem might be corrupt, so I also did try to check it:

root /etc/pve/lxc # pct fsck 105

fsck from util-linux 2.38.1
/var/lib/vz/images/105/vm-105-disk-1.raw: clean, 373288/4194304 files, 8047185/16777216 blocks

[  713.133949] loop17: detected capacity change from 0 to 134217728
[  713.137988] ext4: Unknown parameter 'noacl'

The last info provided me with the right clue:
It seems, that with proxmox 8, that container config did change slightly. Re-setting the Disk ACL to default did eventually work.


After that, the container was able to startup again

Fixing nix Setup on MacOS Catalina

Flattr this!

With MacOS Catalina (10.15), Apple decided to decrease the possibilities of system users to install software applications within the system. That means, that it is not possible anymore to install software at specific location in your system, since most system folder ware mounted read-only at boot to improve the overall system security.

That leads to an installation error if you want to install Nix in MacOS Catalina /nix will not be writable on macOS Catalina #2925 .

In another Issue above there are three Options mentioned by user garyverhaegen-da to circumvent this problem, all with some more or less hefty drawbacks.

The good thing is, that with the use of the APFS System, it is very easy possible to setup a specific FS for nix without the use of an DMG Image and mount it under the correct /nix path.

These are the steps you can do to fix the problem:

!! This approach does only work if you have a Disk that is formated with APFS
(that should always be the case, if your OS is running of a SSD).

You can check this with:

% diskutil list | grep APFS

…
0:      APFS Container Scheme -                      +250.8 GB   disk1
…

Of you have an output here, everything is fine. Otherwise you might have a look into How to Convert to APFS .

1 Create another Volume on your Disk for NIX

This can be done via the Disk Utility Program,

or via the CLI:

% sudo diskutil apfs addVolume disk1 ‘APFS’ nix

Will export new APFS Volume "nix" from APFS Container Reference disk1
Started APFS operation on disk1
Preparing to add APFS Volume to APFS Container disk1
Creating APFS Volume
Created new APFS Volume disk1s6
Mounting APFS Volume
Setting volume permissions
Disk from APFS operation: disk1s6
Finished APFS operation on disk1

After that, you should see a volume nix if you are running diskutil list.
The great thing is, that this volume does not need to have a fixed size. APFS shares the free disk spaces over all configured volumes!

1b (optional) encrypt disk

If you want to, you can also activate Disk Encryption for te new Disk. First you need to find out the name of your new Disk (if you did not note it from the previous command ;-)).

% diskutil list | grep nix

4:                APFS Volume nix                      7.7 GB    disk1s6

% sudo diskutil apfs encryptvolume disk1s6 -user disk

You need to enter a passphrase for the encryption. You have to remember that passphrase once – you can add it to your key chain later on.

Passphrase for the new "Disk" user (672C4CFF-34C6-4407-83ED-294C1C42E161):
Repeat passphrase:

After that the disk encryption will start in the background:

Starting background encryption with the new "Disk" crypto user on disk1s6
The new "Disk" user will be the only one who has initial access to disk1s6
The new APFS crypto user UUID will be 672C4CFF-34C6-4407-83ED-294C1C42E161
Background encryption is ongoing; see "diskutil apfs list" to see progress

2 setup Mount Point

MacOS Catalina does not allow to create folders directly under your Root Path /. But we can use another method to have MacOS create that folder for us. To do this, we have to add an entry into the file /etc/synthetic.conf:

% sudo bash -c “echo nix >> /etc/synthetic.conf”

Now, the next time, the system starts, a mount point /nix will be created. The next task is to have our Volume mounted at Boot.

3 setup Mount

For the Mount Configuration, we need to the UUID of the Volume. We can find this via the diskutil tool:

% diskutil info /dev/disk1s6 | grep UUID

Volume UUID:               1D9389C1-5676-4077-88F5-8D5304A0B1A6
Disk / Partition UUID:     1D9389C1-5676-4077-88F5-8D5304A0B1A6

In previoues version, you would just edit /etc/fstab and adding an entry there. But here is a clear info in that file to only edit it via vifs. The Tool vifs works like vi so an edit is quite easy.

% sudo vifs

We have to enter instert mode whith prssing the Key I.

After that, we can just paste this line into the editor window:

UUID=1D9389C1-5676-4077-88F5-8D5304A0B1A6 /nix apfs  rw

!! This UUID is only valid for my volume – your mileage may vary 

We now have to write our changes: press ESC to leave the insert mode, type :w + ENTER to save the changes and :q + ENTER to exit the editor.

4 Reboot

Before we can start our nix Setup, we have to reboot the system.

If you have enabled the disk encryption, you will be asked for the passphrase after the system has restarted. You need to enter it once and mark “add to key chain”. The next time, the disk should be mounted automatically.

5 Setup Nix

You can now proceed with the installation of Nix. Everything should work as expected. Since it is a realy Mount Point and no Softlink, there should be no errors expected.

% sh <(curl https://nixos.org/nix/install)

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2399  100  2399    0     0   9913      0 --:--:-- --:--:-- --:--:--  9995
downloading Nix 2.3.4 binary tarball for x86_64-darwin from 'https://nixos.org/releases/nix/nix-2.3.4/nix-2.3.4-x86_64-darwin.tar.xz' to '/var/folders/y3/29k7dx8s50l4y_l2tr78b_vh0000gn/T/nix-binary-tarball-unpack.XXXXXXXXXX.Fe7OlFzr'...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    87  100    87    0     0    878      0 --:--:-- --:--:-- --:--:--   878
100 26.6M  100 26.6M    0     0  11.3M      0  0:00:02  0:00:02 --:--:-- 12.0M
Note: a multi-user installation is possible. See https://nixos.org/nix/manual/#sect-multi-user-installation
performing a single-user installation of Nix...
copying Nix to /nix/store.............................................
installing 'nix-2.3.4'
building '/nix/store/96xp8q08cm412ibj8bhrgivx3hgc2gfx-user-environment.drv'...
created 7 symlinks in user environment
installing 'nss-cacert-3.49.2'
building '/nix/store/qjabndqd4kdjwlpiphwxrk84acy7x60k-user-environment.drv'...
created 9 symlinks in user environment
unpacking channels...
created 1 symlinks in user environment
modifying /Users/philipp/.bash_profile...

Installation finished!  To ensure that the necessary environment
variables are set, either log in again, or type

  . $HOME/.nix-profile/etc/profile.d/nix.sh

in your shell.

5b Hint

To have the nix commands in your path, you have to add the following to your  .zshrc file, since Catalina uses ZSH as the default shell:

  . $HOME/.nix-profile/etc/profile.d/nix.sh