Fixing Issues after upgrading Proxmox 7 to 8

My initial plan was to update all of my Proxmox nodes to the latest version by the end of this year. While most updates proceeded smoothly, I encountered two errors on one particular node.
Given that updating servers is a critical operation, especially when they are only remotely accessible via the network, I decided to document these errors and their solutions for future reference.

Proxmox Host does not come online again, after the reboot due to an update

The first issue arose after the mandatory reboot; the server failed to restart. Upon requesting a remote console connection, the boot process stalled with the following error message:

Boot stuck on “A start job is running for Network initialization (XXm / no limit)”

After consulting various posts on the Proxmox forum, I initially suspected a need to update my network configuration. However, my attempts proved unsuccessful, leading to multiple reboots into rescue mode.

Fortunately, I had the insight to consult the official “Proxmox upgrade 7 to 8” guide, where I ultimately discovered the solution to my issue:

Network Setup Hangs on Boot Due to NTPsec Hook

It appears that a bug may lead to the simultaneous installation of both ntpsec and ntpsec-ntpdate. This, in turn, causes the network to fail during boot, resulting in a hang.

The resolution involves disabling the ntpsec-ntpdate start script using the command chmod -x /etc/network/if-up.d/ntpsec-ntpdate and then rebooting, successfully resolving the issue.”

A container does not start and shows the error

The next issues happens with some containers, that don’t want to startup anymore.
The Proxmox UI displays the following error:

run_buffer: 322 Script exited with status 255  
lxc_init: 844 Failed to run lxc.hook.pre-start for container "105"  
__lxc_start: 2027 Failed to initialize container "105"  
TASK ERROR: startup for container '105' failed

After I started the container manually via the terminal, I got a more specific error:

root /etc/pve/lxc # lxc-start -n 105

lxc-start: 105: ../src/lxc/lxccontainer.c: wait_on_daemonized_start: 870 No such file or directory - Failed to receive the container state

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 309 To get more details, run the container in foreground mode

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

root /etc/pve/lxc # lxc-start -n 105 -F

lxc-start: 105: ../src/lxc/conf.c: run_buffer: 322 Script exited with status 255

lxc-start: 105: ../src/lxc/start.c: lxc_init: 844 Failed to run lxc.hook.pre-start for container "105"

lxc-start: 105: ../src/lxc/start.c: __lxc_start: 2027 Failed to initialize container "105"

lxc-start: 105: ../src/lxc/conf.c: run_buffer: 322 Script exited with status 1

lxc-start: 105: ../src/lxc/start.c: lxc_end: 985 Failed to run lxc.hook.post-stop for container "105"

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start

lxc-start: 105: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

Trying to mount the container disk also produces some more errors:

root /etc/pve/lxc # pct mount 105

mount: /var/lib/lxc/105/rootfs: wrong fs type, bad option, bad superblock on /dev/loop17, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
mounting container failed
command 'mount -o noacl /dev/loop17 /var/lib/lxc/105/rootfs//' failed: exit code 32

So I initially though, that the filesystem might be corrupt, so I also did try to check it:

root /etc/pve/lxc # pct fsck 105

fsck from util-linux 2.38.1
/var/lib/vz/images/105/vm-105-disk-1.raw: clean, 373288/4194304 files, 8047185/16777216 blocks

[  713.133949] loop17: detected capacity change from 0 to 134217728
[  713.137988] ext4: Unknown parameter 'noacl'

The last info provided me with the right clue:
It seems, that with proxmox 8, that container config did change slightly. Re-setting the Disk ACL to default did eventually work.

After that, the container was able to startup again

Fixing Error “failed, reason: getaddrinfo EAI_AGAIN” in Gitlab Builds

Some days ago, I detected some new errors in one of our builds pipelines. The interesting part is, that there were no changes done, that might have caused this error.
One example for this error was an DNS resolution error using npm:

#9 [4/5] RUN npm i npm@latest
#9 71.06 npm ERR! code EAI_AGAIN
#9 71.06 npm ERR! syscall getaddrinfo
#9 71.06 npm ERR! errno EAI_AGAIN
#9 71.06 npm ERR! request to https://registry.npmjs.org/npm failed, reason: getaddrinfo EAI_AGAIN registry.npmjs.org
#9 71.06 
#9 71.06 npm ERR! A complete log of this run can be found in:
#9 71.06 npm ERR!     /root/.npm/_logs/2023-02-10T08_30_43_730Z-debug-0.log
#9 ERROR: executor failed running [/bin/sh -c npm i npm@latest]: runc did not terminate sucessfully

After some digging, I found out, that the only update might have come with an updated docker build behaviour, since the docker image, used in this job is docker:latest.

Some Google queries later, I found an issue realted to this topic. It points to a blog post about a strange behaviour of docker in regards to DNS. But since, I had no issues before, I ruled out, that this might be the case here.

And indeed, a look in the latest release notes pointed out the obvious:

Alias docker build to docker buildx build
docker/cli#3314

The easiest fix was to disable buildx alltogether via these ENV Vars:

build:
  image: docker:latest
  stage: build
  variables:
    DOCKER_BUILDKIT: 0
    COMPOSE_DOCKER_CLI_BUILD: 0
  services:
    - docker:dind
  before_script:
…
  script:
…

After this change, the build job did run without any errors.

Create CSV Reports from GIT Repositories containing your commits

Some months ago, I got the need to run over several GIT Repositories and collect the work I did on each day. The play was to gather all the data and collect them in different CSV files.
Since I wasn’t able to find a ready script for this task, I guess it is a good candidate for a quick blog post :-).

The first part is a file folders.txt with a list of all GIT Repositories that we want to analyse. (All folders are Subfolders of a root Directory /Users/user/GIT. This root folder can be changed later on.)

cat folders.txt

tools
utils
customer1/project2
customer2/project1
customer2/project2
customer3/project1

The script does several things:

Going through every repository and collecting Project,Date,Commit,Name,Email,Comment of each commit.
We also need to do some filtering first to deal with characters in the commit messages, that might break the CSV later on.
The last step is to split the complete log file into the different months.

At the moment the script does only run for one specific year, but that can be changed with adding another loop to run it for a bunch of years.

The Source of the Script is:

#!/bin/bash

#YEAR=$(date +"%Y")
HEADER=Project,Date,Commit,Name,Email,Comment
YEAR=2022
ROOT=$(PWD)
GIT_ROOT=$HOME/GIT
PROJECTS=$(cat folders.txt)
TMP_DIR=/tmp/csv
CREATOR="Philipp Haussleiter"
echo "" > /tmp/csv/all.csv
mkdir -p csv/$YEAR $TMP_DIR
rm -Rf $TMP_DIR/*

for PROJECT in ${PROJECTS}; do
    echo "creating log of ${PROJECT}"
    DIR=${GIT_ROOT}/${PROJECT}
    BASENAME=$(basename $DIR)
    cd ${DIR}
    git log --pretty=format:__${BASENAME}__,__%cs__,__%h__,__%an__,__%ae__,__%s__ > /tmp/csv/${BASENAME}.a.log
    cat /tmp/csv/${BASENAME}.a.log | sed -r 's/[\"]+/\"\"/g' > /tmp/csv/${BASENAME}.b.log
    cat /tmp/csv/${BASENAME}.b.log | sed -r 's/__+/\"/g' > /tmp/csv/${BASENAME}.log
    echo "" >> /tmp/csv/${BASENAME}.log
    cat /tmp/csv/${BASENAME}.log >> /tmp/csv/all.csv
    rm /tmp/csv/${BASENAME}.a.* /tmp/csv/${BASENAME}.b.*
    cd ${ROOT}
done

for MONTH in $(seq -f "%02g" 1 12); do
    FILE=csv/$YEAR/${YEAR}-${MONTH}.csv
    FILTER=${YEAR}-${MONTH}
    echo $HEADER > $FILE
    cat /tmp/csv/all.csv |grep "$CREATOR" |grep $FILTER >> $FILE
    echo $HEADER > csv/$YEAR/all.csv
    cat /tmp/csv/all.csv |grep "$CREATOR" >> csv/all.csv
    echo $FILE
done

After running the script for the years 2021 and 2022, you get a folder structure like this:

csv
├── 2021
│   ├── 2021-01.csv
│   ├── 2021-02.csv
│   ├── 2021-03.csv
│   ├── 2021-04.csv
│   ├── 2021-05.csv
│   ├── 2021-06.csv
│   ├── 2021-07.csv
│   ├── 2021-08.csv
│   ├── 2021-09.csv
│   ├── 2021-10.csv
│   ├── 2021-11.csv
│   └── 2021-12.csv
└── 2022
    ├── 2022-01.csv
    ├── 2022-02.csv
    ├── 2022-03.csv
    ├── 2022-04.csv
    ├── 2022-05.csv
    ├── 2022-06.csv
    ├── 2022-07.csv
    ├── 2022-08.csv
    ├── 2022-09.csv
    ├── 2022-10.csv
    ├── 2022-11.csv
    └── 2022-12.csv

Teaching Mailcow how to deal with Ham/Spam

The good must be put in the dish, the bad you may eat if you wish.
Cinderella

Mailcow is a groupware solutions, that is mainly used for email messaging. With Mailcow, you can setup your own Docker-based Mail-Server + Addons.

Mailcow uses rspamd to filter out Spam Messages.
However, after some time, there is a need to fine-tune the Spam (Spam Messages)/Ham (“good” Messages) filtering.

There is a documented method to learn Spam from existing emails within a directory, but especially for non-technical users, that might be hard to understand.

So I updated this method a little bit:

every user hast two folders rspamd/spam and rspamd/ham in their home directory.
Every user can new just drop new spam messages into the spam and false spam messages into the ham folder.
A cron Jobs runs every hour to parse the user directories for new files and updates the rspamd behaviour.

The script for SPAM learning looks like this (assumed that mailcow is installed in /opt/mailcow-dockerized):

#!/bin/bash

cd /opt/mailcow-dockerized
for u in $(ls /home); do
    ""mv /home/$u/rspamd/spam/* ./data/rspamd/spam/""
done
for file in ""./data/rspamd/spam/*""; do 
    docker exec -i $(docker-compose ps -q rspamd-mailcow) rspamc learn_spam < "$file"
done

""rm -Rf ./data/rspamd/spam/*""

There is also a similar script for HAM learning:

#!/bin/bash

cd /opt/mailcow-dockerized
for u in $(ls /home); do
    ""mv /home/$u/rspamd/ham/* ./data/rspamd/ham/""
done
for file in ""./data/rspamd/ham/*""; do 
    docker exec -i $(docker-compose ps -q rspamd-mailcow) rspamc learn_ham < "$file"
done
""rm -Rf ./data/rspamd/ham/*""

Both scripts will produce some output, so a good way of running it via cron, is to pipe the output into a log file.

Using the MacOS airport utility

Using the MacOS airport utility.

Sometimes you need to gather information abouth your current WiFi Connection of you Mac via CLI only (maybe you just have a remote SSH Connection to do so).

With the airport tool, there is a handy utility to perform most of the tasks, that you would normally do via the UI.

You can find that tool in /System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport. To run it, you need to have elevated access rights (run it with sudo).

Best thing is to create an alias first before using the tool:

alias airport='sudo /System/Library/PrivateFrameworks/Apple80211.framework/Versions/A/Resources/airport'

Display the WiFi Preferences

philipp@Imotep ~ % airport prefs
AirPort preferences for en0:

DisconnectOnLogout=NO
JoinMode=Strongest
JoinModeFallback=DoNothing
RememberRecentNetworks=YES
RequireAdminIBSS=NO
RequireAdminNetworkChange=NO
RequireAdminPowerToggle=NO
AllowLegacyNetworks=NO
WoWEnabled=NO

Listing all available WiFi Networks

philipp@Imotep ~ % airport  -s
Password:
            SSID BSSID             RSSI CHANNEL HT CC SECURITY (auth/unicast/group)
                  Network1 24:65:11:d3:bd:85 -88  1,+1    Y  -- RSN(PSK/AES/AES)
                  Network2 ac:22:05:1c:12:4d -83  40      Y  -- RSN(PSK/AES/AES)
                      Home 3c:a6:2f:78:22:cc -80  11      Y  DE RSN(PSK/AES/AES)
                      Home 3c:a6:2f:78:22:cb -78  40      Y  DE RSN(PSK/AES/AES)
          Vodafone Hotspot ae:22:15:1c:12:6f -77  1       Y  EU NONE
                  Network2 ac:22:05:1c:12:6f -77  1       Y  EU RSN(PSK/AES/AES)
                      Home b8:be:f4:87:2e:b0 -74  6,+1    Y  DE RSN(PSK,FT-PSK/AES/AES)
                      Home b8:be:f4:87:2e:b1 -73  48      Y  DE RSN(PSK,FT-PSK/AES/AES)
muenchen.freifunk.net/welt 66:b6:fc:72:c2:28 -51  6       Y  DE NONE
muenchen.freifunk.net/welt 9c:c9:eb:4f:a7:91 -59  44      Y  DE NONE

Listing a specific WiFi Network

Use airport -s <SSID>

philipp@Imotep ~ % airport -s Home
                            SSID BSSID             RSSI CHANNEL HT CC SECURITY (auth/unicast/group)
                          Home 3c:a6:2f:78:22:cc -81  11      Y  DE RSN(PSK/AES/AES)
                          Home 3c:a6:2f:78:22:cb -78  100     Y  DE RSN(PSK/AES/AES)
                          Home b8:be:f4:87:2e:b0 -74  6,+1    Y  DE RSN(PSK,FT-PSK/AES/AES)
                          Home b8:be:f4:87:2e:b1 -69  48      Y  DE RSN(PSK,FT-PSK/AES/AES)

Display the Metrics of your current connection

philipp@Imotep ~ % airport  -I
     agrCtlRSSI: -44
     agrExtRSSI: 0
    agrCtlNoise: -95
    agrExtNoise: 0
          state: running
        op mode: station
     lastTxRate: 144
        maxRate: 144
lastAssocStatus: 0
    802.11 auth: open
      link auth: wpa2-psk
          BSSID: e4:c3:2a:dd:36:f8
           SSID: Home
            MCS: 15
  guardInterval: 800
            NSS: 2
        channel: 9

Get the SSID of the currently connected WiFi

philipp@Imotep ~ %  ioreg -l -n AirPortDriver | grep IO80211SSID | sed 's/^.*= "\(.*\)".*$/\1/; s/ /_/g'

Home

Fixing nix Setup on MacOS Catalina

With MacOS Catalina (10.15), Apple decided to decrease the possibilities of system users to install software applications within the system. That means, that it is not possible anymore to install software at specific location in your system, since most system folder ware mounted read-only at boot to improve the overall system security.

That leads to an installation error if you want to install Nix in MacOS Catalina /nix will not be writable on macOS Catalina #2925 .

In another Issue above there are three Options mentioned by user garyverhaegen-da to circumvent this problem, all with some more or less hefty drawbacks.

The good thing is, that with the use of the APFS System, it is very easy possible to setup a specific FS for nix without the use of an DMG Image and mount it under the correct /nix path.

These are the steps you can do to fix the problem:

!! This approach does only work if you have a Disk that is formated with APFS
(that should always be the case, if your OS is running of a SSD).

You can check this with:

% diskutil list | grep APFS

…
0:      APFS Container Scheme -                      +250.8 GB   disk1
…

Of you have an output here, everything is fine. Otherwise you might have a look into How to Convert to APFS .

1 Create another Volume on your Disk for NIX

This can be done via the Disk Utility Program,

or via the CLI:

% sudo diskutil apfs addVolume disk1 ‘APFS’ nix

Will export new APFS Volume "nix" from APFS Container Reference disk1
Started APFS operation on disk1
Preparing to add APFS Volume to APFS Container disk1
Creating APFS Volume
Created new APFS Volume disk1s6
Mounting APFS Volume
Setting volume permissions
Disk from APFS operation: disk1s6
Finished APFS operation on disk1

After that, you should see a volume nix if you are running diskutil list.
The great thing is, that this volume does not need to have a fixed size. APFS shares the free disk spaces over all configured volumes!

1b (optional) encrypt disk

If you want to, you can also activate Disk Encryption for te new Disk. First you need to find out the name of your new Disk (if you did not note it from the previous command ;-)).

% diskutil list | grep nix

4:                APFS Volume nix                      7.7 GB    disk1s6

% sudo diskutil apfs encryptvolume disk1s6 -user disk

You need to enter a passphrase for the encryption. You have to remember that passphrase once – you can add it to your key chain later on.

Passphrase for the new "Disk" user (672C4CFF-34C6-4407-83ED-294C1C42E161):
Repeat passphrase:

After that the disk encryption will start in the background:

Starting background encryption with the new "Disk" crypto user on disk1s6
The new "Disk" user will be the only one who has initial access to disk1s6
The new APFS crypto user UUID will be 672C4CFF-34C6-4407-83ED-294C1C42E161
Background encryption is ongoing; see "diskutil apfs list" to see progress

2 setup Mount Point

MacOS Catalina does not allow to create folders directly under your Root Path /. But we can use another method to have MacOS create that folder for us. To do this, we have to add an entry into the file /etc/synthetic.conf:

% sudo bash -c “echo nix >> /etc/synthetic.conf”

Now, the next time, the system starts, a mount point /nix will be created. The next task is to have our Volume mounted at Boot.

3 setup Mount

For the Mount Configuration, we need to the UUID of the Volume. We can find this via the diskutil tool:

% diskutil info /dev/disk1s6 | grep UUID

Volume UUID:               1D9389C1-5676-4077-88F5-8D5304A0B1A6
Disk / Partition UUID:     1D9389C1-5676-4077-88F5-8D5304A0B1A6

In previoues version, you would just edit /etc/fstab and adding an entry there. But here is a clear info in that file to only edit it via vifs. The Tool vifs works like vi so an edit is quite easy.

% sudo vifs

We have to enter instert mode whith prssing the Key I.

After that, we can just paste this line into the editor window:

UUID=1D9389C1-5676-4077-88F5-8D5304A0B1A6 /nix apfs  rw

!! This UUID is only valid for my volume – your mileage may vary

We now have to write our changes: press ESC to leave the insert mode, type :w + ENTER to save the changes and :q + ENTER to exit the editor.

4 Reboot

Before we can start our nix Setup, we have to reboot the system.

If you have enabled the disk encryption, you will be asked for the passphrase after the system has restarted. You need to enter it once and mark “add to key chain”. The next time, the disk should be mounted automatically.

5 Setup Nix

You can now proceed with the installation of Nix. Everything should work as expected. Since it is a realy Mount Point and no Softlink, there should be no errors expected.

% sh <(curl https://nixos.org/nix/install)

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2399  100  2399    0     0   9913      0 --:--:-- --:--:-- --:--:--  9995
downloading Nix 2.3.4 binary tarball for x86_64-darwin from 'https://nixos.org/releases/nix/nix-2.3.4/nix-2.3.4-x86_64-darwin.tar.xz' to '/var/folders/y3/29k7dx8s50l4y_l2tr78b_vh0000gn/T/nix-binary-tarball-unpack.XXXXXXXXXX.Fe7OlFzr'...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    87  100    87    0     0    878      0 --:--:-- --:--:-- --:--:--   878
100 26.6M  100 26.6M    0     0  11.3M      0  0:00:02  0:00:02 --:--:-- 12.0M
Note: a multi-user installation is possible. See https://nixos.org/nix/manual/#sect-multi-user-installation
performing a single-user installation of Nix...
copying Nix to /nix/store.............................................
installing 'nix-2.3.4'
building '/nix/store/96xp8q08cm412ibj8bhrgivx3hgc2gfx-user-environment.drv'...
created 7 symlinks in user environment
installing 'nss-cacert-3.49.2'
building '/nix/store/qjabndqd4kdjwlpiphwxrk84acy7x60k-user-environment.drv'...
created 9 symlinks in user environment
unpacking channels...
created 1 symlinks in user environment
modifying /Users/philipp/.bash_profile...

Installation finished!  To ensure that the necessary environment
variables are set, either log in again, or type

  . $HOME/.nix-profile/etc/profile.d/nix.sh

in your shell.

5b Hint

To have the nix commands in your path, you have to add the following to your .zshrc file, since Catalina uses ZSH as the default shell:

  . $HOME/.nix-profile/etc/profile.d/nix.sh

How to disable the www-data user to send emails with postfix

Sometimes an insecure configuration allows spammer to use the www-data user to send emails with you postfix server.
Normally this is the case, when you get a bunch of error email from your mailserver, that some emails from www-data@hostname.tld could not be delivered.

To be sure, that this situation cannot exist, you can add the www-data user to a block list:
In /etc/postfix/main.cf just add

authorized_submit_users = !www-data, static:all

and restart you postfix server.
You will now get a lot of error in you log like this one:

postfix/sendmail[8563]: fatal: User www-data(33) is not allowed to submit mail

After some time, the spammers will hopefully loose interest in you server :-).

20min Handson ZFS

ZFS is often called the last word in file systems.
It is a new approach to deal with large pools of disks originally invented by Sun.
It was later then ported to FreeBSD, MacOS (only 10.5) and Linux.

This text should show some of the basic feature of ZFS and demonstrate them handson by example.

Prerequisites

-> FreeBSD
-> Solaris
-> MacOS (only Userland)

In our example we use

SunOS openindiana 5.11 oi_151a5 i86pc i386 i86pc Solaris.

as an environment.

But most commands also work on the other systems.

Since we do all the work within a VM, our commands have the pattern:

Input VM:

command

Output VM:

result

Pool Creation

The first information we need is the number of disk, present in our environment.
There are several ways to get a basic disk listing. Under (Open-)Solaris this can be done with:

Input VM:

format < /dev/null

Output VM:

AVAILABLE DISK SELECTIONS:
0. c4t0d0 
/pci@0,0/pci8086,2829@d/disk@0,0
1. c5t0d0 
/pci@0,0/pci1000,8000@16/sd@0,0
2. c5t1d0 
/pci@0,0/pci1000,8000@16/sd@1,0
3. c5t2d0 
/pci@0,0/pci1000,8000@16/sd@2,0
4. c5t3d0 
/pci@0,0/pci1000,8000@16/sd@3,0
5. c5t4d0 
/pci@0,0/pci1000,8000@16/sd@4,0
6. c5t5d0 
/pci@0,0/pci1000,8000@16/sd@5,0
7. c5t6d0 
/pci@0,0/pci1000,8000@16/sd@6,0
8. c5t7d0 
/pci@0,0/pci1000,8000@16/sd@7,0

about Pools

With ZFS it is possible to create different kinds of pools on a specific number of disk.
You can also create several pools within one system.

The following Pools are possible and most commonly used:

Type	Info	Performance	Capacity	Redundancy	Command
JBOD	Just a bunch of disks. In theory it is possible to create on pool for each disk in the system, although this is not quite commonly used.	of each disk	of each disk	–	zpool create disk1 pool1 zpool create disk1 pool2 …
Stripe	This is equivalent to RAID0, the data is distributed over all disks in the pool. If one disks fails, all the data is lost. But you can also stripe several Pools (e.g. two raidz pools) to have better redundancy.	very high	N Disks	no	zpool create disk1 disk2 pool1
Mirror	This is equivalent to RAID1, the data is written to both disks in the Pool. Restoring a pool (resilvering) is less efficient, since the data needs to be copied from the remaining disk.	normal	N-1 Disks	+1	zpool create mirror disk1 disk2 pool1
Raidz	This is equivalent to RAID5. One disk contains the parity data. Restoring a pool (resilvering) is less efficient, since the data needs to be copied from the remaining disks.	high	N-1 Disks	+1	zpool create raidz disk1 disk2 disk3 pool1
Raidz2	This is equivalent to RAID6. Two disks containing the parity data. Restoring a pool (resilvering) is less efficient, since the data needs to be copied from the remaining disk with parity data.	high	N-2 Disks	+2	zpool create raidz2 disk1 disk2 disk3 disk4 pool1
Raidz3	There is no real equivalent existing for that one. You have basically three disks with parity data.	high	N-3 Disks	+3	zpool create raidz3 disk1 disk2 disk3 disk4 disk5 pool1

You can also add hot-spares for a better fallback behaviour, SSDs for caching reads (cache) and writes (logs).
I also created a benchmark with various combinations.

create a basic Pool (raidz)

Input VM:

zpool create tank raidz c5t0d0 c5t1d0 c5t2d0
...
zpool status

Output VM:

  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0
            c5t1d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0

errors: No known data errors

(Raid5)

You already can access the newly created pool:

Input VM:

ls -al /tank

Output VM:

 
...
total 4
drwxr-xr-x  2 root root  2 2012-10-23 22:02 .
drwxr-xr-x 25 root root 28 2012-10-23 22:02 ..

create a basic Pool (raidz) with one spare drive

Input VM:

zpool create tank raidz1 c5t0d0 c5t1d0 c5t2d0 spare c5t3d0
...
zpool status

Output VM:

  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0
            c5t1d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0
        spares
          c5t3d0    AVAIL   

errors: No known data errors

List the availibe Layout

Input VM:

zpool list

Output VM:

NAME     SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
tank    1,46G   185K  1,46G         -     0%  1.00x  ONLINE  -

*The 1,5G does not reflect the real availible space. If you copy a 1G File to the Pool it will use 1,5G (1G + 512M Parity).

create a stripped pool

Input VM:

zpool create tank raidz1 c5t0d0 c5t1d0 c5t2d0 raidz1 c5t4d0 c5t5d0 c5t6d0

Output VM:

  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0
            c5t1d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0
          raidz1-1  ONLINE       0     0     0
            c5t4d0  ONLINE       0     0     0
            c5t5d0  ONLINE       0     0     0
            c5t6d0  ONLINE       0     0     0

errors: No known data errors

(Raid50 = Raid5 + Raid5)

deal with disk failures

Input VM:

zpool create tank raidz1 c5t0d0 c5t1d0 c5t2d0 spare c5t3d0

Failure Handling

Input Host:

echo /dev/random >> 1.vdi

Wait for it…
or Input VM:

  pool: tank
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: resilvered 66K in 0h0m with 0 errors on Tue Oct 23 22:14:19 2012
config:

        NAME          STATE     READ WRITE CKSUM
        tank          DEGRADED     0     0     0
          raidz1-0    DEGRADED     0     0     0
            spare-0   DEGRADED     0     0     0
              c5t0d0  DEGRADED     0     0    64  too many errors
              c5t3d0  ONLINE       0     0     0
            c5t1d0    ONLINE       0     0     0
            c5t2d0    ONLINE       0     0     0
        spares
          c5t3d0      INUSE     currently in use

errors: No known data errors

Input VM:

zpool clear tank
...
zpool detach tank c5t0d0
zpool replace tank c5t0d0 c5t7d0

Create File systems

Input VM:

zfs create tank/home
zfs create tank/home/user1
...
chown -R user:staff /tank/home/user1
...
zfs get all tank/home/user1
...
zfs set sharesmb=on tank/home/user1
...
zfs set quota=500M tank/home/user1

Copy File from MacOS into SMB Share.

Snapshot

Input VM:

zfs snapshot tank/home/user1@basic
...
zfs list
...
zfs list -t snapshot

Output VM:

NAME                              USED  AVAIL  REFER  MOUNTPOINT
rpool1/ROOT/openindiana@install  84,0M      -  1,55G  -
tank/home/user1@basic                0      -  42,6K  -

Input VM:

zfs snapshot -r tank/home@backup
...
zfs list -t snapshot

Output VM:

NAME                              USED  AVAIL  REFER  MOUNTPOINT
rpool1/ROOT/openindiana@install  84,0M      -  1,55G  -
tank/home@backup                     0      -  41,3K  -
tank/home/user1@basic                0      -  42,6K  -
tank/home/user1@backup               0      -  42,6K  -

Input VM:

zfs clone tank/home/user1@basic tank/home/user2

Output VM:

tank/home/user2          1,33K   894M  70,3M  /tank/home/user2

Restoring Snapshots

Delete ZIP File in SMB-Share.

Input VM:

ls -al tank/home/user1
...
zfs rollback tank/home/user1@backup

Output VM:

ls -al tank/home/user1

Resizing a Pool

Input VM:

zpool list
...
zpool replace tank c5t0d0 c5t4d0
zpool replace tank c5t1d0 c5t5d0
zpool replace tank c5t2d0 c5t6d0
...
zpool scrub tank
...
zpool list

Output VM:

NAME     SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
tank    1,46G   381K  1,46G     1,50G     0%  1.00x  ONLINE  -

Input VM:

zpool set autoexpand=on tank

Using ZFS for Backups

Bash-Script

rsync -avrz --progress --delete /Users/user root@nas.local::user-backup/
backupdate=$(date "+%Y-%m-%d")
ssh root@nas.local zfs snapshot tank/backup@$backupdate

solving Security Error while starting Java WebStart (e.g. IPMI Remote)

Most of the IPMI Systems out there still using good old Java based Remote Applications to connect to the remote console.
Sine Java 8 update 111, the MD5 singing algorithm was marked as insecure (aka disabled) by Oracale (see Relase Notes for that Release ” Restrict JARs signed with weak algorithms and keys”).
You will get an “Security Error while using MD5withRSA Signature”:

The only solution to fix this error is to have your Hardware Vendor to update the IPMI Firmware with JARs, signed with a more up to date singing algorithm. A work around is to re-enable MD5 for the time being. For that you need to get into your Browser Java Installation.
On my Mac this is in

/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home

You need to edit lib/security/java.security and remove MD5 from jdk.jar.disabledAlgorithms.

Change

jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024

jdk.jar.disabledAlgorithms=MD2, RSA keySize < 1024