Engineering Blog

Keeping Secrets with Chef

by Community Manager on ‎02-08-2017 10:30 AM (4,700 Views)

Written by Karen Bruner 6/26/2014

Systems and Architecture



Chef @Ooyala


Here at Ooyala, we are managing more and more of our server and application configurations with Chef. Chef is a platform which allows users to manage their servers programmatically, thus providing great power and flexibility in customizing configurations on the fly, depending on various attributes of each target server.


Far beyond simply writing configuration files, Chef can easily configure the operating system, install and configure applications, start and stop them in given situations, and more, using "recipes" or specialized scripts written in the Chef DSL, an extension of Ruby. The recipes are organized into "cookbooks," collections of recipes generally related to a single service. When a client server contacts its chef server, the server authenticates via the client's key and sends the client its run list, the recipes it needs to execute. Each client's specific attributes then determine how each recipe behaves on that client.


Chef's power and flexibility allows all Ooyala systems, both physical and virtual, across multiple OS platforms, application roles, development environments, and geographical regions to be managed using the same set of cookbooks. Configuration becomes increasingly automated as we write or modify cookbooks to manage our applications.



Delivering Sensitive Data


An important security issue arises when deciding how to deliver data and files that need to be kept hidden, such as SSL certificate keys and database passwords, to the servers that need them. Choices include having a human manually copy them from a secure location using scp, keeping them unencrypted in the deployment repository, or using a key management service that can also handle other data. Chef can serve as such a system using encrypted data bags.


Chef servers can store pieces of data in "data bags" as key-value pairs. Chef also supports encryption for the data, using the same RSA asymmetric keypairs that it uses to authenticate clients over the API. The server stores the clients' public keys, and they can be used to encrypt data bag contents, which can then only be decrypted by the appropriate private key.


Chef-vault simplified the generation of creating encrypted data bags with a knife plugin. "Knife" is the CLI for interacting directly with a chef server, allowing users to view or modify clients' run lists, view or create data bags, manage cookbooks on the server, and more. Its functionality can be easily extended by creating plugins, classes in Ruby that inherit from the Chef::Knife class, and installing them on the user's host as a Ruby gem.


To create an encrypted data bag with a mysql password for all chef clients tagged "mysql-client" on the server (Chef servers generate SOLR indices of their clients' attributes which can be queried in multiple ways):


$ knife vault create secret_vault secret_example '{"user": "mysql", "password": "CHANGEME"}' \
> --admins admin --search tags:mysql-client


This will create a data bag named secret_vault (if it doesn't exist) and an item in that data bag named secret_example, which has the encrypted data and looks like this:


$ knife data bag show secret_vault secret_example
id:       secret_example
  cipher:         aes-256-cbc
  encrypted_data: LnP6dowSsEQlivDizN0kwq4Ro8m0ydxV14lrzga/8S0=

  iv:             WA7iGztKLlelyfJR6U3RtQ==

  version:        1
  cipher:         aes-256-cbc
  encrypted_data: 08pg5mJ0T+cOGAvAIjy/uHhQuICgs0+ct5FRqHrb4xw=

  iv:             a+6Rdu7bfpfOHYd+BgoGIg==

  version:        1


Chef-vault also created a data bag item named secret_example_keys to store the clients, the data bag item's 'secret' (encrypted separately with each client's key), and other metadata:


$ knife data bag show secret_vault secret_example_keys
admin:         KT/uD/R7Ut4V8BlVfisQojJeraqNBEg/wY4BLCysa0e7KewOMd4OmnD9lOgo

admins:        admin
clients:       sample_server
id:            secret_example_keys
sample_server: XgVw6gezcP4WOEOettj3hmAv/ZbIR7bZsXDyfn8HdmFwpRv093PBB/Rama9K

search_query:  tags:mysql-client


Now, when sample_server runs a chef recipe that needs to write the file where sample_server keeps its mysql client login information, it can:


  1. retrieve the encrypted data bag
  2. retrieve that data bag's encryption key from the item's associated keys (where it's encrypted with each client's public key)
  3. decrypt that data bag's key with the client's private key
  4. and then decrypt the actual data with that secret key


All those steps are handled transparently via Opscode's chef-vault cookbook, so the final recipe code to decrypt and write the data might look something like this:


# Write mysql client login information

include_recipe 'chef-vault'

vault = 'secret_vault'
item = 'secret_example'

  mysql_login = chef_vault_item(vault, item)
rescue ChefVault::Exceptions::KeysNotFound => e
  log e.message
  log 'No mysql client data found.  Check if data bag exists.'

# Write the file
template "#{node['mysql-client']['conf_dir']}/mysql-login" do
  source 'mysql-login.erb'
  mode 0600
  owner node['mysql-client']['user']
  group node['mysql-client']['group']
    :mysql_user => mysql_login['user'],
    :mysql_password => mysql_login['password'],
  action :create


The cookbook would have a Ruby template file called "mysql-login.erb" with holders for the mysql_user and mysql_password variables, maybe like this:


username="<%= @mysql_user %>"
password="<%= @mysql_password %>"


Now whatever program on sample_server that needs to connect to the mysql server can read this file and use the login credentials.



Automation Limitation


For environments like Ooyala's where we create new application server instances in the cloud on a regular basis, the chef-vault model has a limitation. It can only match client nodes that exist or have attributes that match the search string at the time of the vault item creation. After that, the vault item remains static unless it is updated with knife vault update. That's simply not a feasible method for secrets management in a dynamic, automated environment.


As an initial measure, Ooyala has a cookbook recipe for our chef servers that finds and regenerates the client key list for all vault items. This required monkey patching the chef-vault class files and temporarily overriding the chef-server's client key to use the 'admin' user's key, which was easier to have added to the client list via the --admins knife vault switch. This method works, but it's still less-than-optimal. There is now a fork of the chef-vault repo on GitHub, which adds multiple methods and knife plugin options to help allow automation for refreshing the client keys in vault items. (It also adds other missed functionality, such as the ability to specify multiple search queries.)


Our next avenue for extending Chef's ability to store and disseminate secrets to their intended targets may be an API service that our developers can use to enter the data needed to generate the secrets. A backend Jenkins job could then run "knife vault create" with the parameters against all our chef servers.


Ooyala's repository of cookbooks is still growing and evolving, especially as we find easier and smarter methods to manage age-old tasks. Using chef-vault to store sensitive data that chef can deliver to the clients that need it (and only those clients) is one of those steps.