Secrets Management: Sealed Secrets

In Kubernetes, a Secret is a resource containing base64 encoded data, typically used for storing credentials and other sensitive data. Since we are using GitOps we would preferably be able to put secrets inside our GitOps repository. But doing so in plain text would be somewhat reckless. Sealed Secrets together with the kubeseal CLI tool allows encrypting secrets via a controller running inside the Kubernetes cluster so that we can then store the encrypted SealedSecret resources directly in git.

We have then taken this a step further by wrapping kubeseal with our k CLI tool to make secrets management about as easy as it ever was on heroku.

Github: https://github.com/bitnami-labs/sealed-secrets
Documentation: https://sealed-secrets.netlify.app

Architecture Decision Record

Pros

• Works well with GitOps
• Simple and easy to comprehend
• No dependency on third party services

Cons

• Can't share secrets across clusters
• When changing a single piece of a secret the entire secret payload changes due to the encryption which feels a bit unintuitive and bloaty
• Very time consuming to work with manually (solved by additional tooling in k)

Alternatives Considered

We talked a bit about looking at Hashicorp Vault but we felt happy with a simple GitOps based solution and didn't want to depend on a big third party system if we could help it.

Application ENV variables

Secrets is the primary mechanism we use to configure applications via ENV variables. This works by having Secret resources containing ENV style key/value pairs and then referencing them in the envFrom array of the application's values.yaml file.

Read more about how this works in the Kubernetes Distribute Credentials Securely Using Secrets documentation.

In the GitOps repository, all secrets are stored as SealedSecret resources in the applications/shared-secrets directory. Each SealedSecret will get reconciled into a Secret resource with the same name via the Sealed Secrets controller.

You can also configure ENV variables directly via the env arrays in the application's values.yaml file. This is particularily useful for ENV variables referencing values in secrets created by our operators as well as using Dependent Environment Variables to interpolate values from one ENV variable to another.

Managing secrets and configuration using k

Use k config <application-name> to list all ENV variables configured for an application, including annotations on where they're coming from.

Use k config:get <application-name> <env-variable> to get the value of a specific ENV variable. Useful when you want to get a value programmatically in a script.

You can list all secrets in the cluster by running k secrets. This will list each secret together with the applications that are currently inheriting it.

To create a new secret run k secrets:create <secret-name>. This will open up the editor you have configured via the EDITOR ENV variable to allow you to edit the secret payload. When you save and exit the editor the secret will be committed and pushed into the GitOps repository.

To edit an existing secret use k secrets:edit <secret-name>.

To delete a secret remove the secret from applications/shared-secrets in your GitOps repository and commit and push the change.

Migrating Heroku configuration

To make it easier to migrate existing configuration from a Heroku application to a Secret we have created the k env-to-secrets command. Run it from within your GitOps repository and follow the instructions given in the command output.