Leveraging HashiCorp Vault's SSH Secrets Engine

Installing Vault is rather painless as it’s distributed by Hashicorp as a single binary compiled from Go. The latest binary is usually only a curl (or wget, if you’re into that sort of thing) away. 64-bit Linux is assumed:

It’s not a terrible idea to verify the checksum as well. In practice, if you don’t trust the TLS certificate for the site you’re grabbing the object from, then you shouldn’t trust the checksum file you downloaded from it either. Verifying the checksum will let us know if the file was somehow corrupted during the download process, though.

Now that we’ve got the archive containing the Vault binary, we’ll unzip it and place the binary some place convenient:

There’s not a lot that we need to do with Vault’s configuration file. Many of the important configuration items are stored in the encrypted backend.

Let’s add a minimal (i.e. non-production) configuration:

Start the Vault server:

Once Vault is up and running, it must be initialized and unsealed. Since Vault is running in the foreground, we’ll need to jump over to another terminal window.

Before initalizing and unsealing, we need to make sure that the Vault client can find the running server. By default, the Vault client expects to speak TLS on tcp port 8200 on 127.0.0.1. Vault is listening on 8200/tcp on all addresses, however we’ve disabled TLS since this setup is only for testing purposes. All we need to do is set the proper URL in the VAULT_ADDR environment variable.

Make sure this gets picked-up in new shells:

Now, we should be able to get the status of the server:

Next, we initialize Vault:

NOTE: In production, you’ll want to increase the key shares (number of keys that may unseal Vault) and the key threshold (number of keys required to unseal Vault).

Be sure to store the unseal key(s) some place safe. These are the keys to the kingdom and you’ll need them later. You should also save the initial root token to $HOME/.vault-token.

Vault is ready to be unsealed using the unseal key we just received:

At this point, Vault should be ready for (non-prod!) use.

Vault has some pretty robust auditing capabilities. Unfortunately, they’re not enabled by default. You’ll want to enable an audit device and potentially ship the logs to a log aggregator. Enabling auditing will allow you answer questions such as "When was the last time Bob requested a new certificate?".

Enabling the Syslog audit device is simple:

Having detailed logs is crucial to success when operating Vault. Do yourself a favor and get an audit device enabled before interacting with any of the Secrets Engines.

In order to make use of the SSH Secrets Engine, we must first enable it:

We then configure a Certificate Authority for signing the client certificates. We’ll redirect the public key output to a file which we’ll eventually need to distribute any hosts we wish to SSH to using client certificates:

We then configure the SSH daemon to use our new CA certificate:

Let’s also add a user called centos for later:

Check out Vault’s other Secrets Engines here.

Roles in Vault are created by writing data to special paths. Roles provide a fine-grained interface for constraining the details that go into a signed client certificate. For example, one could have a role that allows SSH access as the root user to non-prod IP ranges and another role to SSH as root to production.

NOTE: It’s very tempting to create dozens of highly-specific roles, but this is problematic for a number of reasons. To start, there’s the operational burden of managing and tracking many roles. For every role created, it’s conceivable that a complementary policy will also need to be created and managed. Also, clients are indirectly required to keep track of which roles (paths) they need to access in order to request a signed certificate. Defense-in-depth is a winning strategy when it comes to securing systems. Don’t fall into the trap of creating dozens of highly-customized roles where 2 or 3 generic roles and proper network access controls would be sufficient.

Let’s create a couple of roles. One role for SSH as a regular user and another role SSH as root:

This role will allow clients to request a signed certificate to SSH as centos,ec2-user or ubuntu only - it does not allow them to specify any other user such a root. This role does a couple of other nice things such as ensure that users can get a shell upon successful login ("permit-pty") and it sets a default principal (i.e. login user) when clients don’t specify any principals when requesting the certificate. The role also prevents users from specifying a key ID. This is important because the key ID is what gets logged by the SSH daemon during authentication attempts. Clients would have a field day with this - setting the key ID to all sorts of fun things. Using {{token_display_name}} will force the key ID to be the name of the Vault authentication token. This will simplify the process of correlating login attempts to clients.

By changing allowed_users and lowering ttl this same role would be perfectly suited a role for SSH’ing as root:

Now we write the roles to Vault:

Because we care about our users, we’ve made the process easy on them. When users have need to SSH as a regular user they hit the ssh-client/sign/regular path. When users have need (and are authorized) to SSH as root, they hit the ssh-client/sign/root path.

For a full list of role options, reference the API docs.

Policies allow us to define which users can request certificates from (i.e. write data to) which paths. Not all users should be able to request a certificate which would allow them to SSH as root. However, we assume all users should be able to SSH as a regular user.

First we a create a policy for regular users:

Next, we create a policy for admins - who will SSH as root:

In both cases, we allow users with the attached policy to create new data and update existing data at the respective paths.

When we’re satisfied with the policies, we can write them to Vault:

In order to test the client workflow for both the regular user path and the root user path, we’ll need a couple of users. We start by enabling the userpass authentication method:

This authentication method is a basic username and password authentication mechanism.

Next, we create some users in Vault:

Now, we’re ready to test the client-side workflow.

The very first step is to generate an RSA key pair if you don’t already have one:

Then, we authenticate and get a token from Vault.

Next, we request that Vault sign our SSH public key and return the signed certificate:

We can examine the resultant certificate with ssh-keygen:

Let’s try to SSH using our new certificate:

If everything goes according to plan, we should see a successful authentication attempt in the journal:

Note that the certificate ID is plain-withoutroot. plain is the authentication mechanism path and withoutroot is the user we authenticated with.

Now we can try getting a certificate to SSH as root by hitting the ssh-client/sign/root path:

Bummer. It failed. Why? Because we’re still using the token for the user withoutroot who doesn’t have any permissions to ssh-client/sign/root.

We’ll need to authenticate as withroot and try again:

Let’s try to get the certificate again:

Another failure! This time it has to do with the principals we’re passing. Recall that the root role we created earlier only has a single valid principal: root. Therefore, any attempt to pass a different principal will result in a failure.

We can either adjust valid_principals or remove it entirely since root is the default principal for the role:

No errors this time!

Let’s examine the certificate:

We now see that the key ID reflects the user we authenticated with and the "Principals" field is set to root.

When using certificate-based authentication, questions about how to revoke certificates naturally arise. Currently, Vault does not have any CRL or OCSP capabilities. Based on this issue, there also do not appear to be any plans to add such a feature any time soon.

As mentioned in the issue, Vault provides the ability to issue very short-lived certificates. This largely negates any need for a CRL. Using a CRL could quickly turn out to be an administrative nightmare when attempting to use it at scale with thousands of hosts who will need to get constant CRL updates.

Hopefully the utility of Vault for SSH certificate management is obvious at this point. The process turns out to be relatively painless for everyone involved. A whole new world of possibilities opens up when adding a reverse proxy in front of Vault. You could get fancy and abstract away many of the Vault paths into flat URLs such as https://ssh.example.com which passes requests to ssh-client/sign/regular on the backend.

In future posts, I hope to share some thoughts on other Vault use-cases and deployment practices. Until then, hopefully the information in this post gets you decent milage with Vault’s SSH Secrets Engine.