Seven tips for working with X.509 certificates in .NET

Octopus Deploy utilizes X.509 certificates to allow for secure communication between the central Octopus server, and the remote agents running the Tentacle service. Upon installation, both services generate a self-signed X509 certificate. An administrator then establishes a trust relationship between the two by exchanging the public key thumbprints of each service to the other.

This is a common security model in B2B applications, and it means both services are able to authenticate without exchanging a shared secret or password, or being on the same active directory domain.

But dealing with X.509 certificates on Windows is, well, a pain in the ass. It's the source of a lot of bug reports. In this post, I'm going to share what I've learned about dealing with them so far.

Tip 1: Understand the difference between certificates and PKCS #12/PFX files

In .NET, the X509Certificate2 object has properties for the PublicKey and PrivateKey. But that's largely for convenience. A certificate is something you are supposed to present to someone to prove something, and by design, it's only the public portion of the public/private key pair that is ever presented to anyone. When an X509 certificate is presented to someone, .NET of course strips out the private key. Having the private key property on the certificate object is a bit of a misrepresentation, especially since, as we'll see, there's a big difference in how the public and private key are dealt with.

On Windows a certificate typically has a .cer extension, and they don't contain a private key. You create them like this:

File.WriteAllBytes("Hello.cer", cert.Export(X509ContentType.Cert));

Sometimes it's handy to export the X.509 certificate (which is the public stuff) and the private key into a single file. On Windows we typically use the .PFX extension, which is a PKCS#12 file. In C# we do it like this:

File.WriteAllBytes("Hello.pfx", cert.Export(X509ContentType.Pkcs12, (string)null));

If you are planning to persist a certificate and a private key into a string to store somewhere (like we do), then you can use that Export call above, giving you both the certificate and private key.

Tip 2: Understand the certificate stores

Windows has an MMC snapin that allows you to store certificates. You might think that Windows has some special file on disk somewhere that this snapin manages. In fact, the certificates live in the registry and in various places on disk, and the certificate store just provides convenient access to them.

When you run MMC.exe and go to File->Add/Remove Snap-in..., you can select the Certificates snap-in. When you click Add, you can choose three different stores to manage:

These are the equivalent of the StoreLocation enum that you pass to the X509Store constructor. Each certificate in the store lives in the registry, and the private keys associated with the certificate live on disk.

For example, if I do this:

var store = new X509Store(StoreName.My, StoreLocation.CurrentUser);
store.Open(OpenFlags.ReadWrite);
store.Add(certificate);
store.Close();

StoreLocation.CurrentUser specifies that I want the "My user account" store. StoreName.My maps to the Personal folder in recent versions of Windows. The X509 certificate (not the private key, see the discussion above) is actually added to the registry. Certificates for the current user can go to:

HKEY_CURRENT_USER\SOFTWARE\Microsoft\SystemCertificates

Or to disk, at:

C:\Users\Paul\AppData\Roaming\Microsoft\SystemCertificates\My\Certificates

While certificates for the machine (StoreLocation.LocalMachine, or the "Computer account" option) go to:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates

What exactly is written there? A key exists for each store name (folder), and then under the Certificates sub key is a key with a long, random-looking name.

That name is actually the public thumbprint of the certificate. You can verify this by looking at the thumbprint properties from the snap-in.

The only value stored against this key is a blob containing the public portion of the X509 certificate:

There's an MSDN article with more information about these paths if you need more details.

Tip 3: Understand that private keys live somewhere else

As I mentioned, while in .NET you have an X509Certificate2 object containing both a private and public key, the "certificate" is only the public part. While the certificate is stored in the paths above, the private keys are stored elsewhere. They might be stored under the Keys subkey for the store, or, they might be stored on disk.

For example, if I do this:

var cert = new X509Certificate2(bytes, password, X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.Exportable);
var store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
store.Open(OpenFlags.ReadWrite);
store.Add(cert);
store.Close();

Then I'll end up with the private key stored in the registry. Since I'm specifying StoreLocation.LocalMachine, they go to:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\MY\Keys

However, if I did this:

var cert = new X509Certificate2(bytes, password, X509KeyStorageFlags.UserKeySet | X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.Exportable);
var store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
store.Open(OpenFlags.ReadWrite);
store.Add(cert);
store.Close();

Then I have a problem. Keep in mind that I'm adding the certificate to the same place; but I'm using the UserKeySet option instead of the MachineKeySet option. In this case, the key actually gets written to:

C:\Users\Paul\AppData\Roaming\Microsoft\SystemCertificates\My\Keys\62207B818FC553C92CC6D2C2F869603C190544FB 

Umm, that's no good. I'm importing a certificate for the whole machine to use, so the certificate goes to the registry. But the private key is being written to disk under my personal profile folder. If other users on the machine (including service accounts) don't have access to that file (which they won't by default) they'll be able to load the certificate, but not the private key.

That's not all. When the certificate is loaded, the private key is also written to a path that looks like:

C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\6cf6a27d290e81ccab98cbd34c112cb7_68b198b5-4c92-4b3e-9d30-8e2a81ccb3d7

Or when importing a user key:

C:\Users\Paul\AppData\Roaming\Microsoft\Crypto\RSA\S-1-5-21-992800734-1677258167-2839820197-1001\31c8414d419a75bb6417bc744bf81592_68b198b5-4c92-4b3e-9d30-8e2a81ccb3d7

So again, there's a chance that other accounts don't have access to this file. That leads to a common exception:

System.Security.Cryptography.CryptographicException: Keyset does not exist
 at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
 at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)
 at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
 at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()

The stupid thing about this exception is that you'll know you have a private key. Certificate.HasPrivateKey returns true. You might have just loaded the certificate from a blob with the key. And it might even work under other user accounts or when running interactively rather than as a service. But the cause will probably be because you don't have permissions to that key file.

Here are some examples of times I've seen this:

• When I forgot to specify PersistKeySet for a certificate that I planned to import once and use many times. I figured the key would be imported. In reality, the file on disk just gets linked to. If the key isn't persisted, it can't be used.
• When I created the certificate using UserKeySet and then tried to use it from another account
• When I created the certificate using MachineKeySet, but my user account didn't have access to the default paths above. In one case, the Local System account didn't even have access. That prevented the user from being able to use the key.

The best way to diagnose these issues is to run Procmon from SysInternals and to monitor the disk and registry access that happens when the key is imported and accessed.

Tip 4: Understand the key storage flags

As you might have gathered from above, getting the key storage flags right is crucial. And there's no one sized fits all.

• X509KeyStorageFlags.Exportable - I like to always specify this because it's nice for users to be able to back up the private key
• X509KeyStorageFlags.MachineKeySet - the key is written to a folder owned by the machine. Note that your user account may or may not have access to this location
• X509KeyStorageFlags.UserKeySet - the key is written to a folder owned by you. This is more likely to work the first time, but other users will have trouble accessing the key. Also, beware of temporary profiles, which I'll discuss later.

The note on X509KeyStorageFlags.MachineKeySet is important. Sometimes, you can create a certificate from a blob in memory using the X509KeyStorageFlags.MachineKeySet option. But when you try to access the private key, you'll get the "keyset does not exist" error above. That's because the file couldn't be written or read, but you won't actually see an error message about this.

Tip 5: Don't load direct from a byte array

We used to do this in Octopus:

var certificate = new X509Certificate2(bytes);

It turns out that this writes a temporary file to the temp directory that on some versions of Windows doesn't get cleaned up.

That's a big problem because the file is created using GetTempFile. Once you have more than 65,000+ files, the process will stall as it endlessly tries to find a file name that hasn't been taken. End result: hang.

To be safe, create your own file somewhere, and make sure you delete it when done. Here's how I do it:

var file = Path.Combine(Path.GetTempPath(), "Octo-" + Guid.NewGuid());
try
{
    File.WriteAllBytes(file, bytes);
    return new X509Certificate2(file, /* ...options... */);
}
finally
{
    File.Delete(file);
}

Tip 6: Temporary profiles

Sometimes you'll get this error:

The profile for the user is a temporary profile

A user typically has a profile folder like C:\Users\Paul. When you load a key using the UserKeySet option, the key will be written underneath that profile.

But sometimes, a process might be running under an account with a profile path set to C:\Windows\Temp. Since that folder isn't really meant to be a profile folder, the Windows cryptography API will prevent you from trying to write anything.

This commonly happens when you are running under an IIS application pool, and the Load Profile option is turned off on the application pool.

However it can also happen just sometimes, randomly. Maybe there was a problem with the registry that prevented a profile directory being created. Maybe someone got a little overzealous with group policy. I've had all kinds of bug reports about this. One option is to try stopping any services that run under that account (including application pools) and then logging in interactively to the computer as the user to force a profile to be created. Then log out, and restart the services.

Tip 7: Know the tools to use

There are two tools that will help you to understand what's going on with certificate issues.

The first is SysInternals Process Monitor, which will show you the file IO and registry access that's happening when you try and use your certificates. This is a good way to see where the certificates and keys are being read from and written to.

The other useful tool is a .NET sample called FindPrivateKey.exe which does what it says on the tin. We're actually going to embed some of this code into Octopus vNext to help provide better log errors when we have certificate problems.

Conclusion

The cryptography capabilities in Windows were obviously designed by someone way smarter than me. But I can't help but feel like they were also designed for someone way smarter than me. There are plenty of ways that permissions, group policies, and other issues can creep in to really mess with your use of X.509 certificates in .NET. I wish I'd known of all these pitfalls when I first started using them in Octopus, and hopefully this post will be useful to you. Happy cryptography!