SSH certificate transparency

Posted on 2026-01-07

(Note: This is more in-depth version of the lightning talk I gave at the 2025 transparency.dev summit)

How many entries does the authorized_keys file on your most long-lived remote machine have? How many of them are outdated?

At least for me, that number is larger than I would like it to be. None of those keys are compromised as far as I know, but it's still a mess. Likewise, when a new client device comes along, you either have to go through all servers/VMs and add the new key there or reuse an existing already-authorized key from another machine. So what can we do?

Act one: SSH certificates

SSH certificates offer a partial solution to this problem: Instead of manually distributing authorized_keys files, you can designate a "CA" key that can bless other keys by creating a certificate that authorizes them to access machines that trust this CA (check sshd(8) and sshd_config(5) for more details on how this works in OpenSSH).

However, while this solves the "authorize new keys" part of the problem, it doesn't solve the "revoke existing keys" part. While OpenSSH implements a CRL-style mechanism (they call it "Key Revocation List", or KRL), this just exchanges one file distribution problem (authorized_keys) for another (RevokedKeys). This has some parallels to the X.509 certificate revocation issue on the web – and we can use a similar mitigation strategy here: Since SSH certificates can have a validity time range, we can just issue short-lived certificates to our users, which cuts down the window of opportunity for an attacker. It does not eliminate it of course, but this is not worse than the authorized_keys situation (which is essentially equivalent to an infinite lifetime by default) and combined with the convenience SSH certificates offer for deploying new keys, this starts becoming a net positive.

I've been toying with the idea of deploying this on my infrastructure as well, but there have been two roadblocks. We'll get to the second one below, but the first simply is that there weren't any implementations of short-lived SSH certificates that worked well for me. The native OpenSSH tooling (via ssh-keygen) has no automation of any kind – it is simply a toolbox for generating certificates. There are more turnkey implementations of a CA (like the one by smallstep or OPKSSH), but they tie in with more "enterprise" infrastructure like OIDC SSO, which is a bit too much of a maintenance burden for me.

Instead, tkey-ssh-ca is much closer to what I want: It simply reuses the on-client SSH public key that will be included in the certificate for authentication towards the CA. The CA then has a list of allowed public keys and issues a new certificate for such a key on request. This might feel a little wrong at first, but if you think about it, this is pretty much equivalent to a centrally managed authorized_keys file.

tkey-ssh-ca is a bit bare-bones, so I built my own implementation of the same idea (a code dump can be found here, but I'm putting it up more for illustration than suggesting that anybody use it seriously at this point in time). Crucially, it relies on OpenSSH for all authentication (using authorized_keys with forced commands), so there is no risk of me messing up any of the dangerous network service bits. Also, it uses an SSH agent for all the cryptography. This made the implementation simpler and immediately allows access to the whole SSH ecosystem like using a FIDO token for the CA key.

I wanted to put this on a physically isolated machine for a little more security. While I want to build something more fancy in the future, a random SBC lying in the corner will do for now:

A Raspberry Pi in a plastic enclosure, with power and Ethernet feeding in and a YubiKey sticking out

The YubiKey is the basic FIDO version that I'm using to provide a sk-ssh-ed25519 hardware-backed key for the CA itself. I deliberately do not use resident keys of any sort so that just stealing the hardware token by itself is useless.

Then, to authorize or deauthorize a key, I simply can add/remove it from the config file:

[key."ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID4N9twVf6ObGCjb3CCDVS60BLS9lOPY8n16dNgXPpxj demo-key"]
groups = ["florolf"]

And then request a new certificate:

$ ssh ca@sshca.home.n621.de > cert.pub
PTY allocation request failed on channel 0
Successfully generated certificate for key "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID4N9twVf6ObGCjb3CCDVS60BLS9lOPY8n16dNgXPpxj demo-key"
Valid until: Wed Jan 7 01:19:03 2026 UTC
Connection to sshca.home.n621.de closed.
$ cat cert.pub
ssh-ed25519-cert-v01@openssh.com AAAAIHNzaC1lZDI1NTE5LWNlcnQtdjAxQG9wZW5zc2g…AAARY=
$

This works fine, but there is a final problem.

Act two: Certificate transparency

Alas, this is now a very good backdoor. While the CA will only sign short-lived certificates for allowed keys when working as intended, there is no cryptography (but only a thin layer of software) preventing it from silently signing arbitrary things when compromised, e.g. a certificate for the root principal (if you use TrustedUserCAKeys mechanism rather than putting the CA in an individual user's authorized_keys file on each target machine) or a certificate that never expires for an attacker-controlled key.

You can think up ways to mitigate all those issues, but in the end there is no way to fully close the gap. Delegating authority to the CA is the point after all. So while you can't fully fix the problem, it would at least be nice to know if you're screwed and can take countermeasures. Again, this mirrors what has been happening in the X.509 world and we can use the same solution: Put the certificates in a transparency log and make sure they only get accepted when they were in fact logged. Then, by monitoring the log, you can find out if a certificate you didn't expect has been issued and react accordingly.

There is a bunch of clever (but surprisingly down-to-earth) cryptography involved here to make this watertight as opposed to just writing the certificates to some list, which would be trivial to tamper with. There are some differences in how "modern" transparency logs work compared to the "classical" ones used for X.509 certificates, and as far as I know there is no concise overview available that I could link to. So if you're familiar with classic CT logs some of your intuitions how things work might be off a little bit. In that case or if you're unfamiliar with either, here are the core ideas:

There currently are a few options for where to get such a transparency log from, but here we will use Sigsum. I want to write some more about the current state of transparency logs and why I think they are a shaping up to be a very useful building block soon, so I'll keep things brief here:

Sigsum is a very neat design that distils the transparency log idea down to its essentials in a way that allows people to run one as a public service. This service effectively provides a separate transparency log for each user with zero setup effort other than generating a single Ed25519 key (or two plus a DNS record when the log enforces rate-limiting). Sigsum then allows you to publish the fact that something has been signed by that key in the log, generate offline-verifiable proofs of that fact and discover all the signatures made by that key through simple HTTP APIs. This is everything we need for our purposes.

SSH integration

Okay great, now we can log our certificates to a transparency log. But logging alone is not enough: Somebody who takes over our CA could simply not submit their malicious certificates to the log and we would be none the wiser. A necessary ingredient is that the party which accepts the certificates needs to verify that they have been correctly logged and reject them if they haven't. With Sigsum, we can get a cryptographic proof certifying this fact, which looks like this:

version=2
log=1643169b32bef33a3f54f8a353b87c475d19b6223cbb106390d10a29978e1cba
leaf=00bd6d01c249dc3c8442b538a54c578b3b881edaf9e6bfe37eecd854cd35e24a 80a58171439a064b280922257706d07e07173edcf86a80ac3a3e26f57aa7ea0a33c1a2f78f4e115746012861b4e429749795d72a2d77169caef806a4c7e89507

size=381320
root_hash=7d0f07eb4444eb311fb851c7d9ac94b9d69818b3d5144f1e1cf7768011307242
signature=2d10f2f696e2f78aee39a87a390716476446909eeadd0fb8258e4ddf12c0305633c66622ad654701fe88db1121a8763a4e9226979b7f1375c7e5c81ee69ec70f
cosignature=d960fcff859a34d677343e4789c6843e897c9ff195ea7140a6ef382566df3b65 1767544942 f45915ae6e267a489672c9901c378ee499d97b081d4e2eac47198da203839b85c9e10c0912736cc960ab30197705e713d89eb45a454046f9e3c3928f49a79c02
cosignature=e4a6a1e4657d8d7a187cc0c20ed51055d88c72f340d29534939aee32d86b4021 1767544942 f95ecce51eb095b217bce58adb1041d84d129b0d3d9f2b5a686b91e239efd2688557d63c7e35266e356975c3a7f136de6c58b96128aa7eae0f42f4629cf5b004
cosignature=1c997261f16e6e81d13f420900a2542a4b6a049c2d996324ee5d82a90ca3360c 1767544942 5e4543f00b1de02e156280492fad6ac104f73995e8d023cdf8fa45ab7151a77656a95629b0fc81a9cb97f92dcd323d9d6457a36b41a9a6d6ab49e7ccc67cf208
cosignature=42351ad474b29c04187fd0c8c7670656386f323f02e9a4ef0a0055ec061ecac8 1767544942 eaec855c72075d363661212c6c32d2e00af662b9450d02b94942766743826d38a988187817f0f6ab00eed4e860ca4b7139ca380c7a602590004dbf9c02bc1a0e
cosignature=49c4cd6124b7c572f3354d854d50b2a4b057a750f786cf03103c09de339c4ea3 1767544942 7e87fc2c004827e0a4972161f1ceac1f9f34a0661375ddf8018256f07f3eefc4f66a490ac9dc989d968a7d0311f74a020a8e721d4e9b2595662a77bb832a6e0f
cosignature=86b5414ae57f45c2953a074640bb5bedebad023925d4dc91a31de1350b710089 1767544942 84e4cbdebd4fe3d84f0b265fcacb0dccf534e940f90c2141624c4fd9a72cb4d9c665a35b5810a4a64e7845b88e23e63796d01b0257b396bf61ce7cc7d6a3780e
cosignature=70b861a010f25030de6ff6a5267e0b951e70c04b20ba4a3ce41e7fba7b9b7dfc 1767544942 edb2ab7e1a608da58fb68856427876d08497b945ba602454f3c646fad8ec77c7d19d3931a64175a459ab9e8f5366ea57ed7b3c3c470f3849e1dd2a443ec37101
cosignature=c1d2d6935c2fb43bef395792b1f3c1dfe4072d4c6cadd05e0cc90b28d7141ed3 1767544942 e603b57c52f8397bf2033365bb5e0f790cfd86a4792b52727b1c81412406d3761e1a826e91fc1a54ce093c800c3fb7451e85cd5e17279af9dc2b41e05d20ef08

leaf_index=381319
node_hash=0bca43e70e2ea3e853bcc3f334d838e8f756d20292f66c7a209989a873ccfa91
node_hash=8e8b03302ad455ff6893ccb72f581c018f69a74f41907349bd18b7b0a33bbdb3
node_hash=3822031e72577ba7679b75087e49b52f370b6d7bbf432d7abe8b50f437561997
node_hash=f77bc4db00e509149b6e2fc0028d7107dd415929dc9972f32fe758ce39bcc9a0
node_hash=3acb38f01c633d917b899ed4e522a49a02bf20d358f98ca530e3a3065591e7f2
node_hash=889de80c543a5ae8e35430988dc120ac7edde74b776f9082f814ea88190a601f
node_hash=199f812b9f3667dec31f964098e32652477a2f3d458019b6f8f4acc645cf0131
node_hash=084580f8f6324d4ae42dbcb779502ab9fab77e0c2b92519fe089be72e38d60ed
node_hash=9ddbece4939d621df53f31e2729d5fa7802fd82f3edfb784483d8b7fa9cf41e2
node_hash=e1c7a90c09949c263807e5970aef47f9a06164b759995ab814aff94aff9dcd00

The details are out of scope here, but the main ingredients are:

We now need to somehow include that proof in our certificate and then make sure to verify it in the SSH server. The first part is easy: Certificates can have extension fields and through namespacing we can make up arbitrary extensions of our own. In my case, I picked ssh-ct-proof-v1@n621.de and the payload is just the zlib-compressed version of the above ASCII format to make things a little less gigantic (I've been working on another way to represent and verify Sigsum proofs for embedded systems that is much more compact, but that came later than this project and also is still in progress as of right now).

This only leaves the verification part and normally that would be scary. Touching the OpenSSH authentication code path? Adding a custom parser and verification logic? Shipping custom-built sshd versions to all your machines? Nothing with which I could sleep well at night.

Luckily, there is a way out: The OPKSSH project has to solve a similar problem (verify an OIDC token during authentication) and they found a clever solution outlined in this blog post. In short, OpenSSH has the AuthorizedKeysCommand option which allows you to dynamically generate an authorized_keys file for each authentication attempt. Crucially, one of the things this command has access to is the raw public key or certificate supplied by the client.

So what we can now do is build a stub AuthorizedKeysCommand provider that checks the transparency log proof and outputs an authorized_keys file with a @cert-authority directive for our CA if that proof is valid and an empty file otherwise. Crucially, this bypasses none of the usual SSH authentication steps: Even if our code is buggy, the worst that can happen is that we accept properly signed certificates without a CT proof, but this would still require an attacker to compromise our CA on top of exploiting that bug.

I've built a proof of concept for this using the official sigsum-go libraries and have been running it in production for a little over a month now. So far it has been working great.

Closing the loop: Monitoring

There is one final missing component: Logging is pointless if nobody is checking the log. We need some kind of mechanism to alert us if the CA produces a new certificate so that we can check if this was a valid issuance we triggered or if something has gone awry. For this (and other applications), I've written a Sigsum monitoring tool.

One particularity about Sigsum is that it is designed as a public service and thus needs to worry about log poisoning – since anybody can submit data to the log and you can't remove anything from the log by design, it would take about half a second for somebody to submit something illegal to the log and you would have to take it down. Thus, Sigsum only ever publishes a hash (or strictly speaking a double hash, called the checksum in their terminology) of the actual content we want to submit to the log. A nice side-effect of this is that our certificates never end up out in the public (like with classic X.509 CT), so there is a privacy benefit to this.

To deal with this, sshca records all certificates it produces and provides an endpoint (again via SSH) for the monitor to retrieve a certificate via its checksum. A sigmon leaf_info hook can then go fetch that information and put it in the alert so we can (verifiably) know which actual certificate has been issued. If it fails to retrieve this information, that's a red flag.

Finally, the AuthorizedKeysCommand stub mentioned above supports disallowing certificates by their Sigsum checksum, so we can block misissued certificates solely based on the information available in the transparency log as an emergency measure.

Closing thoughts

As it stands this implementation relies on a bunch of hacks:

It would be nice to have something like this become a first-class OpenSSH feature at some point, but tying this to Sigsum feels like too strong a lock-in for something like the OpenSSH project. You could of course stand up your own SSH-flavored transparency log infrastructure just like they have done with certificates compared to the X.509 world, but this imposes significantly more infrastructure effort on people who want to make use of such a feature.

I'd be curious to hear how the OpenSSH folks feel about this.