Phone Number +1-202-802-9399 (US)
The Lockdown

Thycotic’s Cyber Security Blog

DevOps Secrets Vault for Speed and Security

mm

Written by Billy VanCannon

December 31st, 2019

Most advances in DevOps today are being leveraged for speed. Continuous integration and continuous delivery (CI/CD) is driven by the need to meet internal demands, meet customer demands, and adapt to the competition. However, as the headlines constantly remind us, security can’t be overlooked in our rush to deploy.

No matter where your organization is on the spectrum from monolithic code to microservices, or annual to continuous releases, security is often considered an impediment to progress. Of course, the ideal situation is for tools to deliver speed and security.

Why native security is not sufficient

Starting with applications, there’s often a need for one application to access another application or database either within or outside the enterprise. This is how secrets (passwords, keys) end up hardcoded, or barely better, in configuration files, into the applications themselves. We’ve seen many instances of hardcoded credentials posted publicly in Github, and we’re sure this happens in internal repositories as well. DevOps tools often provide their own method of secrets management. Kubernetes provides a method to store secrets, which I’ll discuss later. Jenkins, Chef, and Puppet all have some type of vault. So why not use them?

Secrets rotation is far easier from one vault location as opposed to coordinating several locations in a variety of different tools

A single centralized vault provides efficiency and security. Many of your applications and DevOps tools need access to the same assets, so having one vault enables the creation of a unified policy for access to those assets. Secrets rotation is far easier from one vault location as opposed to coordinating several locations in a variety of different tools. A centralized vault also enables simplified auditing and alerting from that single source rather than having to pull logs from various tools.

It’s important to note that none of this is very easy, if possible at all, with hardcoded credentials in applications or secrets spread over several DevOps tools. For both applications and DevOps tools, when secrets are needed, an API call to the centralized vault is ideal.

Kubernetes security with DevOps Secrets Vault

As mentioned, Kubernetes provides a mechanism for the applications in pods to access secrets. They are managed in the etcd (pronounced et-cee-dee) distributed database along with all the other cluster configuration information.

Beyond the non-centralized vault issues mentioned above, there are two known issues with this method. The first is that any secrets stored in etcd are available to any node in the cluster. So an intrusion into any node in the cluster allows access to all the other nodes via lateral movement. The other issue is the secrets are stored in plain text (base64 encoded) in the etcd database.

The DevOps Secrets Vault Kubernetes Plug-in uses a different approach. It consists of a Broker in its own pod and a Client that acts as a sidecar within the pod of any application that needs a secret. Let’s look at how this technically works. (See diagram)

Kubernetes security with DevOps Secrets Vault
Click to Enlarge

The secret volume is only available inside the Pod and is already built into Kubernetes. It’s used by the DevOps Secrets Vault plug-in and secrets are updated in the Client rather than the etcd database. To enable the automated connection to DevOps Secrets Vault, the client build file sets an annotation to “dsv.” The Broker then watches for any Pod that has the client with the “dsv” annotation and registers it.

The Broker and Client then work together to keep any secret(s) updated in the secrets volume by connecting to the DevOps Secrets Vault API. The secrets are cached in the Broker for as long as you set them to allow for secrets rotation. If for security reasons you didn’t want to cache secrets, you could set the policy to 0 seconds so it must connect to the API every time the Client requests the secret.

The Broker will keep the secrets updated as long as the Pod remains active. If the Pod’s not active, then the Broker will delete the secrets. The Broker will also constantly watch for new pods and automatically connect them to DevOps Secrets Vault, which is especially helpful when you have autoscaling in your Kubernetes environment.

The plug-in also allows for the communication between the Broker and Client and the Broker and DevOps Secrets Vault to be TLS encrypted. In this case, it makes sense for the TLS keys to be managed by etcd because this is a Cluster-level function. Alternatively, a service mesh like Linkerd or Istio could be used to set up and manage the TLS communications between Pods.

Streamlining secrets management for DevOps

DevOps Secrets Vault is an API-as-a-Service, which makes getting up and running easy. No installation of the vault or database is required and Thycotic even handles all the updates. A command-line interface (CLI) is provided for Windows, Mac, and Linux. In addition to the Kubernetes plug-in, we have a Jenkins plug-in now, will release Terraform, Ansible, Chef and Puppet plug-ins in January 2020, and will be adding more DevOps tool plug-ins over time.

To help applications go to DevOps Secrets Vault directly, we have a Java SDK now and will release Go, Python, and Ruby in January 2020. The CLI, plug-ins, and SDKs are open source, so you have the freedom to add your own customizations. Sign up for the free version of DevOps Secrets Vault get full capabilities to automatically create, archive, and retrieve up to 250 passwords and other secrets.

 

Like this post?

Get our top blog posts delivered to your inbox once a month.

SHARE THIS


The following two tabs change content below.
mm

Billy VanCannon

Billy VanCannon has over 11 years of experience as an RF HW engineer designing radios for first responders and the military. He joined the dark side (business team) and worked with federal government customers. They constantly had questions about the security of the IT side of the networks. He taught himself networking and security and that led to him running the SoC business at Motorola. He then went to Trustwave, a cyber security company in Chicago, where he ran their Certificate Authority (SSL/TLS certs) and their PCI SaaS (helping 3 million merchants with PCI compliance). At Thycotic, Billy is leading new cloud-based initiatives. Billy attended Iowa State for a BSEE, received his MBA at Kellogg (Northwestern) and is a CISSP.