Installing a cloud server with full disk encryption

I am installing a new server on Kimsufi. I want full-disk & swap encryption. Let’s do it.

Guides:

• OpsBlog guide on installing Linux with full-disk encryption and DropBear SSH on Kimsufi

  DropBear allows SSH connection before boot, to remotely decrypt the main partition.

• Debian guide on installing Debian from a Linux system

Overview of steps

• Boot the server in rescue mode.

• Wipe filesystem, format partitions:

  • A “cleartext” /boot partition

  • An encrypted / partition

  • An encrypted swap partition

    Docs:

    • dm-crypt/Swap on Arch wiki
    • cryptsetup How-to: see “2.3 How do I set up encrypted swap?”

• Install and configure Debian (including disk cryptography and decryption through SSH).

• Boot the server & setup automated decryption from a remote (in case of reboot).

Detailed process

Boot the server in rescue mode

• Go to the Kimsufi admin panel, click “NetBoot”, select “Rescue”, pick “rescue64-pro”, clock “Next”, “Confirm”, and then click the “Reboot” button on the admin panel.

• Kimsufi will mail you the root password for you to log into the server in rescue mode.

• Will likely cause an error from your SSH client, since the fingerprint of the server differs from the usual one.

Wipe filesystem & format partitions

Target partition table:

        +-----------+--------------------+-----------+--------------------+
Name:   | /dev/sda1 | /dev/sda2          | /dev/sda3 | /dev/sda4          |
        |           |                    |           |                    |
        |           |                    |           |                    |
        |           | +------------------+           | +------------------+
        |           | | /dev/mapper/swap |           | | /dev/mapper/main |
Format: | none      | | swap             | ext4      | | ext4             |
Size:   | 1MiB      | | 8GiB             | 512MiB    | | remaining space  |
Flags:  | bios_grub | |                  |           | |                  |
Mount:  |           | | swap             | /boot     | | /                |
        +-----------+-+------------------+-----------+-+------------------+

We use GPT partition table layout (without UEFI), which demands a small bios_grub partition at the beginning of the drive (stores GRUB’s core.img). Hence /dev/sda1. See:

• https://en.wikipedia.org/wiki/BIOS_boot_partition
• https://www.gnu.org/software/grub/manual/grub/html_node/BIOS-installation.html#BIOS-installation
• https://wiki.archlinux.org/index.php/partitioning#Partition_table
• https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Disks#Partition_tables

1. Wipe filesystem, format partitions

   • Wipe all information about previous filesystem:

     DANGER!!! DANGER!!! WIPES EVERYTHING FROM YOUR DISK!!!

     wipefs -a /dev/sda
     

   • Create GPT disk layout

     # Create GPT layout
     parted -a optimal /dev/sda mklabel gpt
     # Create first 1MiB bios_grub partition 
     parted /dev/sda -a optimal mkpart bios_grub 0% 1MiB
     parted /dev/sda set 1 bios_grub on
     # Create third 8GiB swap partition
     parted /dev/sda -a optimal mkpart swap 1MiB 8001MiB
     # Create second 512MiB /boot partition
     parted /dev/sda -a optimal mkpart boot 8001MiB 8513MiB
     # Create last / partition using all remaining space 
     parted /dev/sda -a optimal mkpart main 8513MiB 100%
     # Set first partition as bootable
     

   • Format partitions

     • The ext4 boot partition

       Format it like so:

       mkfs.ext4 /dev/sda3
       

     • The encrypted swap partition

       All configuration of the encrypted swap partition comes after the OS installation.

     • The encrypted main partition

       # Set it up using cryptsetup (I searched for good parameters):
       cryptsetup --type luks2 --cipher aes-xts-plain64 --hash sha256 --iter-time 2000 --key-size 512 luksFormat /dev/sda4
       
       # Get the UUID, save it for later:
       cryptsetup luksDump /dev/sda4 | grep UUID | awk '{print $2}'
       # Example output: 09762476-ba7c-4732-8856-44a716c23339
       
       # Decrypt the partition:
       cryptsetup open /dev/sda4 main
       
       # Format it to ext4:
       mkfs.ext4 /dev/mapper/main
       

2. Install and configure Debian

   • Mount the partitions

     # Decrypt the main partition (if not already done):
     cryptsetup open /dev/sda4 main
     # mount it:
     mount /dev/mapper/main /mnt 
     # Mount the boot partition:
     mkdir /mnt/boot
     mount /dev/sda3 /mnt/boot
     

   • Bootstrap latest stable Debian into mounted partitions

     debootstrap --arch amd64 stable /mnt http://ftp.fr.debian.org/debian/
     

   • Mount system partitions

     mount -o bind /dev /mnt/dev
     mount -t proc proc /mnt/proc
     mount -t sysfs sys /mnt/sys
     mount -t devpts devpts /mnt/dev/pts
     

   • Chroot

       chroot /mnt /bin/bash
     

   • Fill out crypttab and fstab (UUID is the output of above cryptsetup luksDump)

     /etc/crypttab

     cat << EOF > /etc/crypttab
     # <name> <device>                                  <password>   <options>
     main     UUID=09762476-ba7c-4732-8856-44a716c23339 none         luks
     swap     /dev/sda2                                 /dev/urandom swap,noearly,cipher=aes-xts-plain64,size=512
     EOF
     

     /etc/fstab

     cat << EOF > /etc/fstab
     # <filesystem>   <dir> <type> <options>          <dump> <pass>
     /dev/mapper/main /     ext4   errors=remount-ro  0      1
     /dev/sda3        /boot ext4   defaults           0      2
     /dev/mapper/swap none  swap   sw                 0      0
     EOF
     

   • Do the /proc/mounts -> /etc/mtab symlink

     ln -s /proc/mounts /etc/mtab
     

     Why? Linux From Stratch and our guide on OpsBlog both say it’s needed.

   • Choose a hostname & DNS domain

     hammerhead & hammerhead.luxeylab.net

   • Fill in network-related files

     Beware of your ethernet interface name! It’s only after failing with eth0 that the boot logs informed me that the iface name was eno1.

     /etc/network/interfaces

     cat << EOF > /etc/network/interfaces
     ######################################################################
     # /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)
     # See the interfaces(5) manpage for information on what options are
     # available.
     ######################################################################
     
     # loopback interface
     auto lo
     iface lo inet loopback
     
     # eno1 through DHCP
     auto eno1
     iface eno1 inet dhcp
     EOF
     

     /etc/resolv.conf

     cat << EOF > /etc/resolv.conf 
     # OVH public DNS
     nameserver 213.186.33.99
     # https://servers.opennicproject.org/edit.php?srv=ns2.he.de.dns.opennic.glue 
     nameserver 172.104.136.243
     EOF
     

     /etc/hosts

     cat << EOF > /etc/hosts
     127.0.0.1 localhost
     127.0.1.1 hammerhead hammerhead.luxeylab.net 
     
     # The following lines are desirable for IPv6 capable hosts
     ::1     localhost ip6-localhost ip6-loopback
     fe00::0 ip6-localnet
     ff00::0 ip6-mcastprefix
     ff02::1 ip6-allnodes
     ff02::2 ip6-allrouters
     ff02::3 ip6-allhosts
     
     213.186.33.116 ws.ovh.com
     198.245.48.4 ws.ovh.ca
     5.135.179.11 hammerhead hammerhead.luxeylab.net 
     EOF
     

     /etc/hostname

     echo "hammerhead" > /etc/hostname
     

     /etc/timezone

     dpkg-reconfigure tzdata
     

   • Update APT repositories in /etc/apt/sources.list

     cat << EOF > /etc/apt/sources.list
     deb http://ftp.fr.debian.org/debian buster main
     deb-src http://ftp.fr.debian.org/debian buster main
     
     deb http://ftp.fr.debian.org/debian-security/ buster/updates main
     deb-src http://ftp.fr.debian.org/debian-security/ buster/updates main
     
     deb http://ftp.fr.debian.org/debian buster-updates main
     deb-src http://ftp.fr.debian.org/debian buster-updates main
     EOF
     

   • Configure locales & keyboard

     # Refresh packages list
     apt update
     # Install & configure locales 
     apt install locales && dpkg-reconfigure locales
     # Install & configure keyboard 
     apt install console-setup console-data && dpkg-reconfigure keyboard-configuration
     

   • Pick up a kernel & install it

     apt-cache search linux-image
     # I selected the meta-package `linux-image-amd64` which is the latest 
     apt install linux-image-amd64 linux-headers-amd64
     

   • Install additional necessary software

     apt install cryptsetup dropbear grub-pc man-db ssh 
     

     Because we use a GPT table layout but no UEFI, we install grub-pc and not grub-efi-amd64.

     Do think about installing bash-completion and ufw someday.

   • Set up SSH keys and the rest (where <PUBLIC_SSH_KEY> is your client SSH public key)

     mkdir /root/.ssh && chmod 600 /root/.ssh
     echo "<PUBLIC_SSH_KEY>" > /root/.ssh/authorized_keys
     echo "<PUBLIC_SSH_KEY>" > /etc/dropbear-initramfs/authorized_keys
     

   • Configure dropbear-initramfs (SSH on boot)

     sed -i 's/#DROPBEAR_OPTIONS=/DROPBEAR_OPTIONS=\"-p 2222 -c \/bin\/cryptroot-unlock\"/' /etc/dropbear-initramfs/config
     

     This changes the SSH listen post to 2222, and enforces that the only command run from dropbear-initramfs is cryptroot-unlock.

   • Configure sshd custom listen port (SSH after boot)

     sed -i 's/#Port 22/Port 2223/' /etc/ssh/sshd_config
     

   • [Optional] Authorize root password login

     # Remove -s (disable password login) from dropbear-initramfs
     sed -i 's/local flags=\"Fs\"/local flags=\"F\"/' /usr/share/initramfs-tools/scripts/init-premount/dropbear
     

   • Update GRUB and initramfs

     update-grub && update-initramfs -u
     

   • Clean up before leaving

     exit
     umount /mnt/{boot,dev/pts,dev,proc,sys}
     umount /mnt
     cryptsetup luksClose main
     

3. Boot the server & setup automated decryption from a remote

   • I configured my laptop’s ~/.ssh/config like so:

     Host hammerhead-decrypt
         User root
         Hostname <SERVER_URL_OR_IP>
         Port 2222
         IdentityFile <PUBLIC_SSH_KEY_PATH>
         # Prevents signatures mismatch
         UserKnownHostsFile ~/.ssh/known_hosts_hammerhead-decrypt
     Host hammerhead
         User root
         Hostname <SERVER_URL_OR_IP>
         Port 2223
         IdentityFile <PUBLIC_SSH_KEY_PATH>
     

   • And here is how I connect to the server:

     laptop$ ssh hammerhead-decrypt 
     Please unlock disk main: <PASSWORD>
     # [...]
     Connection to <SERVER_URL_OR_IP> closed.
     laptop$ ssh hammerhead 
     # [...]
     root@hamerhead:~# echo A winner is you!
     A winner is you!
     

Automatically decrypt the drive from a remote server

It is desirable to have a daemon running on a remote server, to automatically decrypt the drive when the encrypted server reboots without warning.

The remote server is called a key escrow. One must be particularly careful about the escrow’s security, since it holds the decryption keys for our server.

Trinity recommends Tang and Clevis.