atomfs – convenience program for mounting OCI-squash layers

Project Machine is the home for our "OCI-native operating system". We started this work in 2017.  As many people, including cyphar and Tycho have pointed out, the .tar.gz nature of OCI layer files has many problems.  So starting early in 2019 Tycho started adding support to stacker for building OCI images with squash layers.

When we started - long before OCI artifacts support etc was discussed - there was some concern of whether squash based layers would have support issues.  Docker registries would not store them, for instance.  So Tycho coined 'atomfs' (*1) as the name for OCI images using squash layers.  Point being - it's not a filesystem, just a way of making use of filesystems.  We (wrote, and) used zot to store them, enabled stacker to build them, and used our internal OS build and management tool, atomix, to extract or mount them.  But while stacker and zot are external tools, we didn't have an external tool to make mounting these images easy.

To rectify this, we present atomfs: a program which can be used to easily examine atomfs images, that is, OCI images with squashfs layers.

If you are root host, it will use the kernel to mount each squash layer, with fsverity if the image specifies verity root hashes.  If you are root in a container, it will try to use squashfuse.  If that is not available, then it will use unsquashfs to unpack the full filesystem. The layers are then combined using overlay into one mount.  This is the step which requires root (*2), at least in your namespace, to accomplish.  So if you are not root, you'll need to unshare a user namespace.

Here are two asciicinema sessions showing some usage.

Happy spelunking!

---

*1 - atomix is the name of the internal project which begat project-machine.
*2 - we could use fuse-overlay for this, but requiring a new namespace has a certain tidyness, so we are still considering.

It’s like…

Let’s say you had one of those huge keyrings, with 50 keys on it. (Let’s say 41 keys – number chosen either randomly or bc its the current number of capabilities) On the plus side, when you want to let someone use one, you can magically, instantly make a copy to give to that person. On the down side, that person can do the same (give a copy to *anyone*), once you hand it to them. Further on the down side, well, you have 41 keys.

Now if you want to send the valet to take your car out, you have to give him a key to the elevator, key to the garage entry door, key to the car door, key to the car ignition, and a key to the garage-exit. You’ve done this yourself plenty so you know what keys to give.

But other parts of the building were designed by other people, who have placed gates in various places you’ve never visited, and know nothing about. So now you need the maid to go clean the fifth bedroom. How do you know what keys to give her?

You can start her on her way with the key to the elevator and the key to the bedroom. But she needed a key to pass a gate in the first hallway. So she comes back… all she knows to say is EPERM! She can’t even tell you where the gate is. So you walk with her to the first gate (strace), see the gate, figure out which key it needs, give it to her. Send her on her way.

Of course she hits another gate, comes back, and says EPERM!

Now, you’re not a quitter. And this is important. So you’ll keep trying until you figure out all the keys she needs. Unfortunately, there are 30 other people – chauffeurs, mechanics, janitors, butlers, maids, some delivery people, a gardener… They all need access to various places.

It gets worse. Some gates are only locked at certain times. So you think you’ve gotten them sorted out, but at 2am it turns out half of them are newly blocked. Oh, and next week the person in charge of the second floor decides a few more gates are really needed. Hopefully their newsletter will tell you where, and which keys they’ll use – but they’re busy, so probably not.

So, sure, you’re not usually a quitter, but it sure will be temping to give the full 41 keys to anyone who comes by.

OCI based linux

Containers are most commonly distributed in two ways:

1. ‘Image based’: lxc and lxd distribute their container images as full images, a simple representation of root filesystem and some configuration info.
2. OCI: based on the original docker format, this has become an open standard for publishing not only container images, but any artifacts.

Our products are created, distributed, and installed as OCI. All services run as containers. Each container rootfs is re-created from its OCI image at every start. A physical machine’s rootfs is also shipped as an OCI image, and is recreated on every boot. A system representation therefore consists of a manifest specifying the OCI references for services to run. To make this secure,

1. Images must be verifiable. An fs extraction step, such as un-tarring, prevents us from verifying the result on next boot without re-extracting. Therefore we distribute OCI layers as squashfs instead of tarballs, and mount them using overlayfs.
2. Squashfs layers ship with their dmverity root hash in the image manifest.
3. The system manifest which lists the content-addressed OCI images is signed with a product key.
4. The certificate for a product’s manifest signing public key is stored with the system manifest. All product manifest signing certificates are signed by one manifest signing CA.
5. The manifest signing CA certificate is stored in initrd.
6. The initrd, ‘smooshed’ together with the kernel and kernel command-line into one kernel.efi, are signed with a kernel signing key.
7. The TPM keys for root filesystems and machine-identifying unique key are only unlocked for the pcr7 resulting from (our shim and) a kernel signed with the right kernel signing key certificate.

In this way, we can ship a single ‘kernel.efi’ for all TPM-enabled hardware and VM products. To protect different groups’ products from each other, products are provisioned with a product ID, which must match product ID in the product manifest signing certificate. Each machine is also provisioned with a unique keypair, supporting secure cluster bringup and remote attestation.

This allows us to use OCI as the source for (verifiably) securely installed and booted products. We can install the OS on a host in the traditional way, or we can pxe-boot specifying on the kernel command-line an OCI URL to a layer containing the manifest to boot into.

We hope to present the full solution (with source) at FOSDEM 2023.

# References
1. For more details on the OCI specification, see https://github.com/opencontainers/image-spec/blob/main/spec.md.
2. The very code for generating and mounting squashfs based OCI images is at https://github.com/project-stacker/stacker and https://github.com/project-stacker/stacker/tree/master/atomfs.
3. The in-development replacement for atomfs is puzzlefs, at https://github.com/anuvu/puzzlefs and https://github.com/anuvu/puzzlefs/blob/master/doc/index.md.
4. The TPM-based unattended encrypted filesystem solution was presented in full at LSS 2021: ‘Securing TPM secrets in the datacenter’: https://www.youtube.com/watch?v=wfJDmfPP1OA.

Openconnect (anyconnect) on Ubuntu Jammy

Sorry, I should have posted this weeks ago to save others some time.

If you are running openconnect-sso to connect to a Cisco anyconnect VPN, then when you upgrade to Ubuntu Jammy, openssl 3.0 may stop openconnect from working. The easiest way to work around this is to use a custom configuration file as follows:


cat > $HOME/ssl.cnf
openssl_conf = openssl_init

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Options = UnsafeLegacyRenegotiation
EOF


Then use this configuration file (only) when running openconnect:


OPENSSL_CONF=~/ssl.cnf openconnect-sso --server=your-server.whatever.com


sxmo on pinephone

If you are looking for a new phone that either respects your privacy, leaves you in control, or just has a different form factor from the now ubiquitous 6″ slab, there are quite a few projects in various states of readiness

Freedom:

• vollaphone
• oneplus
• pinephone
• librem 5
• fairphone

Different form factors:

• unihertz atom and titan
• gemini and cosmo

Earlier this year I bought a pinephone, braveheart edition. I’ve tried several OSes on it. Just yesterday, I tried:

• sailfish: looked great, but it would not recognize sim, and crashed when launching browser.
• ubports (ubuntu touch): looked good, texting worked, but crashed when launching app store and would not ring on incoming calls.
• mobian: nice set of default apps, but again would not ring on incoming calls.

So I’m back to running what I’ve had on it for a month or two – sxmo, the suckless mobile operating system. It’s an interesting, different take on> interacting with the phone, and I quite like it. More importantly, for now it’s the most reliable as a communication devvice. With it, I can

• make and receive calls and texts.
• send texts using vi :).
• easily send/receive mail using mbsync, mutt, and msmtp.
• easily customize using scripts – editing existing ones, and adding new ones to the menu system.
• use a cozy, known setup (dwm, st, tmux, sshd)
• change call and text ringtone based on the caller – few other phones I’ve had have done that, not one did it well.
• have a good browsing experience.
• use both wifi and 4G data. I’ve not hotspotted, but can see no reason why that will be a problem.

The most limiting thing about this phone is the battery. It drains very quickly, charges slowly, and if I leave the battery in while turned off, it continues to discharge until, after a day, it doesn’t want to turn back on. An external battery charger helps enormously with this. There is also an apparent hardware misfeature which will prevent the modem from waking the cpu during deep sleep – this will presumably be fixed in later hardware versions, remember mine is the braveheart .

New library: libsubid

User namespaces were designed from the start to meet a requirement that unprivileged users be able to make use of them. Eric accomplished this by introducing subuid and subgid delegations through shadow. These are defined by the /etc/subuid and /etc/subgid files, which only root can write to. The setuid-root programs newuidmap and newgidmap, which ship with shadow, respect the subids delegated in those two files.

Until recently, programs which wanted to query available mappings, like lxc-usernsexec, have each parsed these two files. Now, shadow ships a new library, libsubid, to facilitate more programatic querying of subids. The API looks like this:

struct subordinate_range **get_subuid_ranges(const char *owner);
struct subordinate_range **get_subgid_ranges(const char *owner);
void subid_free_ranges(struct subordinate_range **ranges);

int get_subuid_owners(uid_t uid, uid_t **owner);
int get_subgid_owners(gid_t gid, uid_t **owner);

/* range should be pre-allocated with owner and count filled in, start is
 * ignored, can be 0 */
bool grant_subuid_range(struct subordinate_range *range, bool reuse);
bool grant_subgid_range(struct subordinate_range *range, bool reuse);

bool free_subuid_range(struct subordinate_range *range);
bool free_subgid_range(struct subordinate_range *range);

The next step, which I’ve not yet begun, will be to hook these general queries into NSS. You can follow the work in this github issue </p.

Customizing ubuntu phone: terminal top keyboard row

This is one of those “for my own reference on my next (pine) phone” posts, but anyone using ubuntu phone (ubports.com) may find it useful.

I use mutt (in a libertine ‘container’) on the ubuntu phone for sending email. The terminal keyboard is not bad, but one annoying thing I’ve found is that the auxilliary keyboard rows were not optimal for use in vi. The top row, for those who haven’t seen it, is a single row of extra buttons. There are several top rows to choose from, i.e. Fn keys, control keys, scroll keys, and there’s even one where buttons type out full commands like ‘clear’, ‘rm’ and ‘chmod’.

The main buttons I use are the up arrow, tab, and escape. But escape is in the ‘fn’ row, while tab and up arrow are in the scroll list. So I kept having to switch between different rows. To switch rows, you hold down a button until a popup appears allowing you to choose. This is suboptimal.

To fix this, I went into /opt/click.ubuntu.com/com.ubuntu.terminal/0.9.4/qml/KeyboardRows/Layouts and edited ScrollKeys.json. I removed the _key suffix for all the labels, which just take up space so that fewer buttons show up in one line. I copied the escape key entry from FunctionKeys.json as the first entry in ScrollKeys.json. Then I moved all other entries which preceded the tab key to the end of the file (adjusting the trailing ‘,’ as needed). Finally, I copied ScrollKeys.json to AScrollkeys.json, to make this the first keyboard row whenever I fire up the terminal. (The file ~/.config/ubuntu-terminal-app/ubuntu-terminal-app.conf supposedly orders these, but it is re-written every time the terminal starts!)

Perhaps I should add a row for ‘|’, ‘!’, and a few others which I’m always going to the second number screen for. But for now, this should speed things up.

First experience with Ubuntu Touch

For the past few weeks I’ve been using a nexus 4 running ubuntu touch as, mostly, my daily driver. I’ve enjoyed it quite a bit. In part that’s just the awesome size of the nexus 4. In part, it’s the ubuntu touch interface itself. If you haven’t tried it, you really should. (Sailfish ambiances are so much prettier, but ubuntu touch is much nicer to use – the quick switch to switch between two apps, for instance. Would that I could have both.). And in part it’s just the fact that it really feels like – is – a regular ubuntu system.

There have been a few problems. The biggest has been to do with email. I need a phone to do only a few things well – texts, calls, and imap email. The only imap mailer available by default, dekko2, looks very nice and is promising, but was simply not yet reliable for me. It would simply stop getting updates for hours, with no warning, for instance. So I’ve taken to using an ubuntu-push notification system for email notifications, and mutt and offlineimap in a libertine container for reading and sending. The notification system is based on https://forums.ubports.com/topic/3126/facebook-messenger-push-notifications/2 and the python mailbox library. It runs on my mail server, checks for new mail, and, if there is any, sends a push notification to my phone. The code I’m using is here on launchpad and here on github. It can certainly stand to be made a bit smarter (the seenmsgs list should be pruned, for instance, and maildir and mh folder support should be trivial to add for those cool cats who use those). Using this service instead of having the phone try to check for emails not only ends up being very reliable, but also saves a lot of battery life.

All in all this could definately work as my permanent new phone! Now if I could just get my hands on a pinephone or librem 5. The nexus 4 hardware is great, but it would be awesome being able to run an uptodate, upstream kernel. More than that – now that my experiment has succeeded, I probably need to stop, because running the ancient kernel simply is not as safe as I’d like. But I digress.

A huge thanks to Mark and the original touch team for creating it, and to the ubports team for keeping it going.

Nice job, everyone!

Running your own mail server

Motivation

Not too long ago there was some hubub around https://myaccount.google.com/purchases. In brief, if you use google mail, it tracks your purchases through receipts received in email. Now, some people see this as no big deal or even a feature. Others see it as a privacy invasion, and are reminded that all their data can be mined by the email provider and possibly third parties. Of those, some advoate getting a paid email provider. Agreed, that provides less incentive to monetize your data… but only a bit. Eventually, any company, however good its initial intentions, goes through leadership changes, is bought out, or goes bankrupt. At that point, your data is one of the assets being bargained with.

The other alternative, of course, is to run your own mail server. I won’t lie – this is not for everyone. But it’s not as bad as some make out. I recently reinstalled mine, so I wrote down the steps I took, and will leave them here. I’ve been holding onto this for at least 6 months hoping to eventually run through them again to work out some of the finer details. That hasn’t happened yet, so I’ll just post what I have now as a start.

Running your own mail server is not free. In particular, you’ll need to pay for a domain name ($10-15/year), and some place to run the mail server. If you have an always-on machine at home, and stable IP address, then you can run it there. You can pay for a tiny cloud instance on amazon/rackspace/digitalocean/etc. There are cheaper options (including “one year free” amazon micro instances), but a small digitalocean instance will be $5/month. Personally, I keep a large server online for running many VMs and containers, and run the mail server there.

You will also need a certificate. That’s now easy and free with letsencrypt.

There are also some non-monetary costs. You may get a bit more spam. And once in awhile, you may run into a case where your mail server is being rejected by another.

On the other hand, the server is entirely yours. You can create as many accounts for individual purposes as you like. You can point multiple domain names at it, so that you don’t give away your primary one for every little purchase you make. Ten, twenty years from now, you can still have all your friends’ and family emails in the same place, in the same format. This last one is too often overlooked, yet one of the best advantages of open source for all applications.

Setup

I picked up a new hosted server, and installed Ubuntu 18.04 on it. First thing I did was go to my dns provider and register a name for it, and set up the new mx record to point to it. The details of this vary a bit depending on your dns provider, so I won’t go into detail here (I’ll do a post if people ask for clarification). If you’re looking for a provider, I do recommend zoneedit.

I installed lxc and created a new container in which to run my mailserver:

apt-get -y install lxc1
lxc-create -t download -n mail -- -d ubuntu -r bionic -a amd64

I gave it a static ip address through dnsmasq:

echo "dhcp-host=mail,10.0.5.155" >> /etc/lxc/dnsmasq.conf
echo "LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf" >> /etc/default/lxc-net
sudo systemctl stop lxc-net
sudo systemctl start lxc-net

The point of the static ip address is to facilitate forwarding mail related ports into the container. I did this with a script started at boot by systemd:

cat > /usr/bin/container-ports-fwd << EOF
nic=enp0s31f6
iptables -t nat -A PREROUTING -p tcp -i ${nic} --dport 25 -j DNAT --to-destination 10.0.5.155:25
iptables -t nat -A PREROUTING -p tcp -i ${nic} --dport 465 -j DNAT --to-destination 10.0.5.155:465
iptables -t nat -A PREROUTING -p tcp -i ${nic} --dport 993 -j DNAT --to-destination 10.0.5.155:993
iptables -t nat -A PREROUTING -p tcp -i ${nic} --dport 587 -j DNAT --to-destination 10.0.5.155:587
EOF
chmod 755 /usr/bin/container-ports-fwd
cat > /etc/systemd/system/container-ports-forward.service << EOF
[Unit]
Description=Bring up port forwards for lxc
After=lxc-net.target
Before=lxc.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/container-ports-fwd

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable container-ports-fwd
systemctl start container-ports-fwd

I also installed and ran letsencrypt on the host:

sudo apt -y install letsencrypt
letsencrypt -d mail.example.org -m me@my.mail certonly

Next I started up the container and installed the basic mail tools:

sudo lxc-start -n mail
sudo lxc-attach -n mail apt -y install dovecot-imapd postfix mutt

New since my last mail server install is the removal of dovecot-postfix in favor of the mail-stack-delivery package:

sudo lxc-attach -n mail apt -y install mail-stack-delivery

After this I copied the letsencrypt keys into the container

lxc-attach -n mail -- mkdir -p /etc/letsencrypt/live/mail.example.org
cp /etc/letsencrypt/live/mail.example.org/* /var/lib/lxc/mail/rootfs//etc/letsencrypt/live/mail.example.org

and edited
/etc/postfix/main.cf and /etc/dovecot/conf.d/10-ssl.conf to point to those using these lines:

smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.org/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.org/privkey.pem

This is enough to be able to send and receive mail. Personally, I want to run this server in a uid-mapped container and from a luks-encrypted device. While you can setup the whole container that way from the start, for simplicity of examples, you could at this point copy over and uid-shift the container contents to a new device, and update the container configuration accordingly.

Some notes which I should elaborate on later:

SPF
• postscreen setup
• /etc/postfix/master.cf, i.e. uncomment smtps
• /etc/dovecot/conf.d details

My phone’s not dumb, it just looks it.

Well actually it is a little.

For my money, the height of the smartphone age was 2009-2011. That brought us the Nokia n900 and Nokia n9. Both brilliant for their own reasons. There were devices before that which I’d be happy to have back. But nothing since then. Sure, the Ubuntu Edge or Neo900 would have been great. But they never came to be.

So my current phone is chosen to match my lifestyle: A rugged Sonim XP5 . It runs Android. Without the full store, but you can sideload apps. It has no touchscreen, and only a numeric keypad. But it’s still pretty decent for one-handed operation. It does what I want out of a phone:

• Calls
• SMS
• Email (with hacked K-9 Mail version)
• Alarms and timer.
• Flashlight, on/off with one button, no going through gui.
• Browser. Just useful enough to look up a business, see its hours, and call.
• Hotspot for laptop for anything more serious.
• Waterproof (wash it under running water)
• Rugged
• 3 or more days battery life.
• FM radio

Addendum

I wrote this some time ago but hadn’t posted it. Very lately I’ve actually been playing with sailfish. I really like the interface, but it’s not very reliable. I’m considering trying out ubuntu touch. I don’t really want to buy another phone (nexus 5), and ubuntu touch is known to not be working fully on my secondary phone, but I should be able to help with that, especially as there is mention of not attaching to LXC containers *duck*. Just have to decide I’m willing to remove sailfish.