OCI based linux

Containers are most commonly distributed in two ways:

1. ‘Image based’: lxc and lxd distribute their container images as full images, a simple representation of root filesystem and some configuration info.
2. OCI: based on the original docker format, this has become an open standard for publishing not only container images, but any artifacts.

Our products are created, distributed, and installed as OCI. All services run as containers. Each container rootfs is re-created from its OCI image at every start. A physical machine’s rootfs is also shipped as an OCI image, and is recreated on every boot. A system representation therefore consists of a manifest specifying the OCI references for services to run. To make this secure,

1. Images must be verifiable. An fs extraction step, such as un-tarring, prevents us from verifying the result on next boot without re-extracting. Therefore we distribute OCI layers as squashfs instead of tarballs, and mount them using overlayfs.
2. Squashfs layers ship with their dmverity root hash in the image manifest.
3. The system manifest which lists the content-addressed OCI images is signed with a product key.
4. The certificate for a product’s manifest signing public key is stored with the system manifest. All product manifest signing certificates are signed by one manifest signing CA.
5. The manifest signing CA certificate is stored in initrd.
6. The initrd, ‘smooshed’ together with the kernel and kernel command-line into one kernel.efi, are signed with a kernel signing key.
7. The TPM keys for root filesystems and machine-identifying unique key are only unlocked for the pcr7 resulting from (our shim and) a kernel signed with the right kernel signing key certificate.

In this way, we can ship a single ‘kernel.efi’ for all TPM-enabled hardware and VM products. To protect different groups’ products from each other, products are provisioned with a product ID, which must match product ID in the product manifest signing certificate. Each machine is also provisioned with a unique keypair, supporting secure cluster bringup and remote attestation.

This allows us to use OCI as the source for (verifiably) securely installed and booted products. We can install the OS on a host in the traditional way, or we can pxe-boot specifying on the kernel command-line an OCI URL to a layer containing the manifest to boot into.

We hope to present the full solution (with source) at FOSDEM 2023.

# References
1. For more details on the OCI specification, see https://github.com/opencontainers/image-spec/blob/main/spec.md.
2. The very code for generating and mounting squashfs based OCI images is at https://github.com/project-stacker/stacker and https://github.com/project-stacker/stacker/tree/master/atomfs.
3. The in-development replacement for atomfs is puzzlefs, at https://github.com/anuvu/puzzlefs and https://github.com/anuvu/puzzlefs/blob/master/doc/index.md.
4. The TPM-based unattended encrypted filesystem solution was presented in full at LSS 2021: ‘Securing TPM secrets in the datacenter’: https://www.youtube.com/watch?v=wfJDmfPP1OA.

Advertisement
Posted in Uncategorized | Tagged , , | Leave a comment

Openconnect (anyconnect) on Ubuntu Jammy

Sorry, I should have posted this weeks ago to save others some time.

If you are running openconnect-sso to connect to a Cisco anyconnect VPN, then when you upgrade to Ubuntu Jammy, openssl 3.0 may stop openconnect from working. The easiest way to work around this is to use a custom configuration file as follows:


cat > $HOME/ssl.cnf
openssl_conf = openssl_init

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Options = UnsafeLegacyRenegotiation
EOF

Then use this configuration file (only) when running openconnect:


OPENSSL_CONF=~/ssl.cnf openconnect-sso --server=your-server.whatever.com

Posted in Uncategorized | Leave a comment

sxmo on pinephone

If you are looking for a new phone that either respects your privacy, leaves you in control, or just has a different form factor from the now ubiquitous 6″ slab, there are quite a few projects in various states of readiness

Freedom:

  • vollaphone
  • oneplus
  • pinephone
  • librem 5
  • fairphone

Different form factors:

Earlier this year I bought a pinephone, braveheart edition. I’ve tried several OSes on it. Just yesterday, I tried:

  • sailfish: looked great, but it would not recognize sim, and crashed when launching browser.
  • ubports (ubuntu touch): looked good, texting worked, but crashed when launching app store and would not ring on incoming calls.
  • mobian: nice set of default apps, but again would not ring on incoming calls.

So I’m back to running what I’ve had on it for a month or two – sxmo, the suckless mobile operating system. It’s an interesting, different take on> interacting with the phone, and I quite like it. More importantly, for now it’s the most reliable as a communication devvice. With it, I can

  • make and receive calls and texts.
  • send texts using vi :).
  • easily send/receive mail using mbsync, mutt, and msmtp.
  • easily customize using scripts – editing existing ones, and adding new ones to the menu system.
  • use a cozy, known setup (dwm, st, tmux, sshd)
  • change call and text ringtone based on the caller – few other phones I’ve had have done that, not one did it well.
  • have a good browsing experience.
  • use both wifi and 4G data. I’ve not hotspotted, but can see no reason why that will be a problem.

The most limiting thing about this phone is the battery. It drains very quickly, charges slowly, and if I leave the battery in while turned off, it continues to discharge until, after a day, it doesn’t want to turn back on. An external battery charger helps enormously with this. There is also an apparent hardware misfeature which will prevent the modem from waking the cpu during deep sleep – this will presumably be fixed in later hardware versions, remember mine is the braveheart .

Posted in Uncategorized | 2 Comments

New library: libsubid

User namespaces were designed from the start to meet a requirement that unprivileged users be able to make use of them. Eric accomplished this by introducing subuid and subgid delegations through shadow. These are defined by the /etc/subuid and /etc/subgid files, which only root can write to. The setuid-root programs newuidmap and newgidmap, which ship with shadow, respect the subids delegated in those two files.

Until recently, programs which wanted to query available mappings, like lxc-usernsexec, have each parsed these two files. Now, shadow ships a new library, libsubid, to facilitate more programatic querying of subids. The API looks like this:

struct subordinate_range **get_subuid_ranges(const char *owner);
struct subordinate_range **get_subgid_ranges(const char *owner);
void subid_free_ranges(struct subordinate_range **ranges);

int get_subuid_owners(uid_t uid, uid_t **owner);
int get_subgid_owners(gid_t gid, uid_t **owner);

/* range should be pre-allocated with owner and count filled in, start is
 * ignored, can be 0 */
bool grant_subuid_range(struct subordinate_range *range, bool reuse);
bool grant_subgid_range(struct subordinate_range *range, bool reuse);

bool free_subuid_range(struct subordinate_range *range);
bool free_subgid_range(struct subordinate_range *range);

The next step, which I’ve not yet begun, will be to hook these general queries into NSS. You can follow the work in this github issue </p.

Posted in Uncategorized | Tagged , | Leave a comment

Customizing ubuntu phone: terminal top keyboard row

This is one of those “for my own reference on my next (pine) phone” posts, but anyone using ubuntu phone (ubports.com) may find it useful.

I use mutt (in a libertine ‘container’) on the ubuntu phone for sending email. The terminal keyboard is not bad, but one annoying thing I’ve found is that the auxilliary keyboard rows were not optimal for use in vi. The top row, for those who haven’t seen it, is a single row of extra buttons. There are several top rows to choose from, i.e. Fn keys, control keys, scroll keys, and there’s even one where buttons type out full commands like ‘clear’, ‘rm’ and ‘chmod’.

The main buttons I use are the up arrow, tab, and escape. But escape is in the ‘fn’ row, while tab and up arrow are in the scroll list. So I kept having to switch between different rows. To switch rows, you hold down a button until a popup appears allowing you to choose. This is suboptimal.

To fix this, I went into /opt/click.ubuntu.com/com.ubuntu.terminal/0.9.4/qml/KeyboardRows/Layouts and edited ScrollKeys.json. I removed the _key suffix for all the labels, which just take up space so that fewer buttons show up in one line. I copied the escape key entry from FunctionKeys.json as the first entry in ScrollKeys.json. Then I moved all other entries which preceded the tab key to the end of the file (adjusting the trailing ‘,’ as needed). Finally, I copied ScrollKeys.json to AScrollkeys.json, to make this the first keyboard row whenever I fire up the terminal. (The file ~/.config/ubuntu-terminal-app/ubuntu-terminal-app.conf supposedly orders these, but it is re-written every time the terminal starts!)

Perhaps I should add a row for ‘|’, ‘!’, and a few others which I’m always going to the second number screen for. But for now, this should speed things up.

Posted in Uncategorized | Tagged , , | Leave a comment

First experience with Ubuntu Touch

For the past few weeks I’ve been using a nexus 4 running ubuntu touch as, mostly, my daily driver. I’ve enjoyed it quite a bit. In part that’s just the awesome size of the nexus 4. In part, it’s the ubuntu touch interface itself. If you haven’t tried it, you really should. (Sailfish ambiances are so much prettier, but ubuntu touch is much nicer to use – the quick switch to switch between two apps, for instance. Would that I could have both.). And in part it’s just the fact that it really feels like – is – a regular ubuntu system.

There have been a few problems. The biggest has been to do with email. I need a phone to do only a few things well – texts, calls, and imap email. The only imap mailer available by default, dekko2, looks very nice and is promising, but was simply not yet reliable for me. It would simply stop getting updates for hours, with no warning, for instance. So I’ve taken to using an ubuntu-push notification system for email notifications, and mutt and offlineimap in a libertine container for reading and sending. The notification system is based on https://forums.ubports.com/topic/3126/facebook-messenger-push-notifications/2 and the python mailbox library. It runs on my mail server, checks for new mail, and, if there is any, sends a push notification to my phone. The code I’m using is here on launchpad and here on github. It can certainly stand to be made a bit smarter (the seenmsgs list should be pruned, for instance, and maildir and mh folder support should be trivial to add for those cool cats who use those). Using this service instead of having the phone try to check for emails not only ends up being very reliable, but also saves a lot of battery life.

All in all this could definately work as my permanent new phone! Now if I could just get my hands on a pinephone or librem 5. The nexus 4 hardware is great, but it would be awesome being able to run an uptodate, upstream kernel. More than that – now that my experiment has succeeded, I probably need to stop, because running the ancient kernel simply is not as safe as I’d like. But I digress.

A huge thanks to Mark and the original touch team for creating it, and to the ubports team for keeping it going.

Nice job, everyone!

Posted in Uncategorized | 3 Comments

Running your own mail server

Motivation

Not too long ago there was some hubub around https://myaccount.google.com/purchases. In brief, if you use google mail, it tracks your purchases through receipts received in email. Now, some people see this as no big deal or even a feature. Others see it as a privacy invasion, and are reminded that all their data can be mined by the email provider and possibly third parties. Of those, some advoate getting a paid email provider. Agreed, that provides less incentive to monetize your data… but only a bit. Eventually, any company, however good its initial intentions, goes through leadership changes, is bought out, or goes bankrupt. At that point, your data is one of the assets being bargained with.

The other alternative, of course, is to run your own mail server. I won’t lie – this is not for everyone. But it’s not as bad as some make out. I recently reinstalled mine, so I wrote down the steps I took, and will leave them here. I’ve been holding onto this for at least 6 months hoping to eventually run through them again to work out some of the finer details. That hasn’t happened yet, so I’ll just post what I have now as a start.

Running your own mail server is not free. In particular, you’ll need to pay for a domain name ($10-15/year), and some place to run the mail server. If you have an always-on machine at home, and stable IP address, then you can run it there. You can pay for a tiny cloud instance on amazon/rackspace/digitalocean/etc. There are cheaper options (including “one year free” amazon micro instances), but a small digitalocean instance will be $5/month. Personally, I keep a large server online for running many VMs and containers, and run the mail server there.

You will also need a certificate. That’s now easy and free with letsencrypt.

There are also some non-monetary costs. You may get a bit more spam. And once in awhile, you may run into a case where your mail server is being rejected by another.

On the other hand, the server is entirely yours. You can create as many accounts for individual purposes as you like. You can point multiple domain names at it, so that you don’t give away your primary one for every little purchase you make. Ten, twenty years from now, you can still have all your friends’ and family emails in the same place, in the same format. This last one is too often overlooked, yet one of the best advantages of open source for all applications.

Setup

I picked up a new hosted server, and installed Ubuntu 18.04 on it. First thing I did was go to my dns provider and register a name for it, and set up the new mx record to point to it. The details of this vary a bit depending on your dns provider, so I won’t go into detail here (I’ll do a post if people ask for clarification). If you’re looking for a provider, I do recommend zoneedit.

I installed lxc and created a new container in which to run my mailserver:

apt-get -y install lxc1
lxc-create -t download -n mail -- -d ubuntu -r bionic -a amd64

I gave it a static ip address through dnsmasq:

echo "dhcp-host=mail,10.0.5.155" >> /etc/lxc/dnsmasq.conf
echo "LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf" >> /etc/default/lxc-net
sudo systemctl stop lxc-net
sudo systemctl start lxc-net

The point of the static ip address is to facilitate forwarding mail related ports into the container. I did this with a script started at boot by systemd:

cat > /usr/bin/container-ports-fwd << EOF
nic=enp0s31f6
iptables -t nat -A PREROUTING -p tcp -i ${nic} --dport 25 -j DNAT --to-destination 10.0.5.155:25
iptables -t nat -A PREROUTING -p tcp -i ${nic} --dport 465 -j DNAT --to-destination 10.0.5.155:465
iptables -t nat -A PREROUTING -p tcp -i ${nic} --dport 993 -j DNAT --to-destination 10.0.5.155:993
iptables -t nat -A PREROUTING -p tcp -i ${nic} --dport 587 -j DNAT --to-destination 10.0.5.155:587
EOF
chmod 755 /usr/bin/container-ports-fwd
cat > /etc/systemd/system/container-ports-forward.service << EOF
[Unit]
Description=Bring up port forwards for lxc
After=lxc-net.target
Before=lxc.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/container-ports-fwd

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable container-ports-fwd
systemctl start container-ports-fwd

I also installed and ran letsencrypt on the host:

sudo apt -y install letsencrypt
letsencrypt -d mail.example.org -m me@my.mail certonly

Next I started up the container and installed the basic mail tools:

sudo lxc-start -n mail
sudo lxc-attach -n mail apt -y install dovecot-imapd postfix mutt

New since my last mail server install is the removal of dovecot-postfix in favor of the mail-stack-delivery package:

sudo lxc-attach -n mail apt -y install mail-stack-delivery

After this I copied the letsencrypt keys into the container

lxc-attach -n mail -- mkdir -p /etc/letsencrypt/live/mail.example.org
cp /etc/letsencrypt/live/mail.example.org/* /var/lib/lxc/mail/rootfs//etc/letsencrypt/live/mail.example.org

and edited
/etc/postfix/main.cf and /etc/dovecot/conf.d/10-ssl.conf to point to those using these lines:

smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.org/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.org/privkey.pem

This is enough to be able to send and receive mail. Personally, I want to run this server in a uid-mapped container and from a luks-encrypted device. While you can setup the whole container that way from the start, for simplicity of examples, you could at this point copy over and uid-shift the container contents to a new device, and update the container configuration accordingly.

Some notes which I should elaborate on later:

    • SPF

    • postscreen setup
    • /etc/postfix/master.cf, i.e. uncomment smtps
    • /etc/dovecot/conf.d details
  • Posted in Uncategorized | 2 Comments

    My phone’s not dumb, it just looks it.

    Well actually it is a little.

    For my money, the height of the smartphone age was 2009-2011. That brought us the Nokia n900 and Nokia n9. Both brilliant for their own reasons. There were devices before that which I’d be happy to have back. But nothing since then. Sure, the Ubuntu Edge or Neo900 would have been great. But they never came to be.

    So my current phone is chosen to match my lifestyle: A rugged Sonim XP5 . It runs Android. Without the full store, but you can sideload apps. It has no touchscreen, and only a numeric keypad. But it’s still pretty decent for one-handed operation. It does what I want out of a phone:

    • Calls
    • SMS
    • Email (with hacked K-9 Mail version)
    • Alarms and timer.
    • Flashlight, on/off with one button, no going through gui.
    • Browser. Just useful enough to look up a business, see its hours, and call.
    • Hotspot for laptop for anything more serious.
    • Waterproof (wash it under running water)
    • Rugged
    • 3 or more days battery life.
    • FM radio

    Addendum

    I wrote this some time ago but hadn’t posted it. Very lately I’ve actually been playing with sailfish. I really like the interface, but it’s not very reliable. I’m considering trying out ubuntu touch. I don’t really want to buy another phone (nexus 5), and ubuntu touch is known to not be working fully on my secondary phone, but I should be able to help with that, especially as there is mention of not attaching to LXC containers *duck*. Just have to decide I’m willing to remove sailfish.

    Posted in Uncategorized | 14 Comments

    Unprivileged container builds using stacker

    One of the primary goals of user namespaces was to provide the ability for unprivileged users to have their own range of uids over which they would have privilege, with minimal need for setuid programs and no risk (barring bugs in the OS) of their privilege having effect on uids which are not “their own”.

    We’ve had user namespaces for awhile now. While there are some actions which cannot be done in a user namespace, such as mounting a loopback filesystem, there are many things, such as setting up a build environment with custom package installs, which used to be a challenge without privilege but are now simple.

    My friend Tycho wrote stacker stacker as a tool for building OCI images. A few of its features include:

    • Creates OCI images.
    • Can also be used for general software building.
    • Supports layer re-use between build stages to minimize redundant I/O and time.
    • Supports unprivileged use!

    To show how to use stacker for unprivileged builds, I created a little demo readme. You can see it in the README.md at my stacker-demo github repo.

    Posted in Uncategorized | Leave a comment

    Outdoors laptop (part 2)

    Some time ago I posted about wanting an outdoor laptop. The first option I listed was a panasonic toughbook. Recently (a year and a half later) I finally ordered one. I ordered from bobjohnson.com, because the people there are a class act who’ve been calmly answering my questions for a long time.

    Some highlights:

    * It has a transflective display. This means that it is emissive, but also uses reflected sunlight to boost brightness, up to 6000 nits. In comparison, my previous thinkpad was 350 nit (unusable sometimes even in shade), and my macbook was 500nit. With this laptop, I can leave the display on 25% brightness and move from a dark basement to hurt-my-skin bright sunlight.

    * It’s ‘fully rugged’, so using it in rain or dust storms should not be an issue. (I lost 4G ram in my thinkpad to dust).

    * It has a shoulder strap ($20 extra) screwed on solidly.

    * It has a touchscreen with stylus. (To use this under ubuntu 18.04 I had to install xserver-xorg-input-evdev and remove xserver-xorg-input-libinput. Note just installing evdev was not enough) I may look like a dweeb, but I prefer this to smudging my screen.

    The laptop I got is a CF-19 MK6. This is several years old and refurbished. The reason I went with this instead of a new toughbook (besides price) is because, as far as I can tell, only the CF-19 MK5 through MK8 have the transflective display. The replacement for the CF-19 (the CF-20) may have a better screen (i’ve not seen it), but it is not transflective and comes in at “only” 1000nit. Same with the slightly larger nonconvertable laptops.

    Mind you, there is (I trust) a reason these screens did not take off – the colors are kind of washed out, and it’s low resolution. But for reading kernel code by the pool without draining the battery in 1 hr, the only thing I can imagine being better is an eink screen.

    The CF-19 is compact: it’s a 10″ (convertible) netbook. This keyboard is more cramped than on my old s100 netbook. I do actually kind of like the keys – they have a good travel depth and a nice click. But it’s weird going back to a full-size keyboard.

    The first time I measured the battery life, it shut down when battery listed 36% remaining, after a mere got 3 hours. Panasonic had advertised 10 hours for this laptop. 3 was unacceptable, and I was about ready to send it back. But, reading the powertop output, I noticed that the sound card was listed as taking tons of battery power. So for my next run I did a powertop –auto-tune, and got over 4 hours battery life. Then I noticed bluetooth radio doing the same, so I did rmmod btusb. These are now all done on startup by systemd. The battery still stops at 35%, which takes getting used to, but it’s acceptable.

    4.5 hours is still limiting, so I picked up a second battery and an external charger. I can charge one battery while using the other, or take both batteries along for a longer trip.

    In summary – i may have found my outdoors laptop. I’d still prefer it be thinner, with a slightly larger and mechanical keyboard, and have 12 hour battery life, delivered on a unicorn…

    (Here is an attempt to show the screen in very bright sunlight. It’s hard to get a good photo, since the camera wants to play its own games) :

    Posted in Uncategorized | 9 Comments