Engineering Blog

Mixing up Vagrant in Test-Kitchen

by Community Manager on ‎02-08-2017 10:43 AM (2,605 Views)

Written by Karen Bruner 3/31/2015

Systems and Architecture



Mixing up Vagrant in Test-Kitchen


The Problem


We had a use case in test-kitchen where one of our end-to-end cookbook tests installed so many packages and support tools, it was filling up the 10Gb root disk of our custom VirtualBox images. Rather than make larger images, since most groups were not having the same issue, it made more sense to add more disk space when needed to the Vagrant/VirtualBox instances.

Potential use cases other than simply needing more disk space include testing recipes that create RAID devices, logical volumes, filesystem creation and manipulation, and so on.



Considering Solutions


There were two obvious options here: grow the image or add a disk. Growing the existing image would have required resizing the root filesystem or creating and formatting a new filesystem on the root 'disk' within the test-kitchen provision. While this was possible, it requires a recipe that is never needed in actual production systems, and it could be potentially time-consuming.

Adding a second virtual disk made more sense, in large part because we already have an internal Chef cookbook that finds additional, unformatted disks on a host and creates, formats, and mounts them as a RAID device. By prepending that recipe to our test-kitchen suite's run_list, we would be using existing code and, if anything, mimicking the production environment even more.


However, probably the most important consideration was to implement the solution in a way that disrupted our established Chef cookbook development and test workflow as little as possible. Ideally, after copying over a few generic scripts or configs into the cookbook, an engineer would only need to add an attribute to a suite in their cookbook's .kitchen.yml file and everything else would be automatic.



Making It Work


The final solution, which is, admittedly, just one way of getting to the same end goal, required a bit more than just that magic attribute, but not by much.


By passing the kitchen-vagrant driver a supplementary Vagrantfile using the vagrantfiles option, it was quite easy to create an unformatted virtual disk and add it to the instance. The Vagrantfile would look something like this:


disk = '/my/super/special/instance/disk.vdi'
Vagrant.configure('2') do |c|
  c.vm.provider :virtualbox do |p|
    p.customize ['createhd', '--filename', disk, '--size', '10240']
    p.customize ['storageattach', :id,
                 '--storagectl', 'IDE Controller',
                 '--port', '1',
                 '--device', '1',
                 '--type', 'hdd',
                 '--medium', disk]


One consideration was to make sure only suites needing an additional disk got it, and that the additional disk would be unique for each instance, to make parallel convergence safe. Therefore adding a one-size-fits-all Vagrantfile via the kitchen-vagrant vagrantfiles option wasn't viable. (While Vagrantfiles are interpreted ruby, nothing in the kitchen-vagrant execution was exposing, say, the instance name to the vagrant run, making using the same file for every instance and setting paths dynamically impractical.)


How to get a Vagrantfile hardcoded with the necessary information in the right place? kitchen-vagrant also supports a pre_create_command hook, but it also exposes very little information, other than the Vagrant path of the current instance. As of kitchen-vagrant-0.15.0, the hook wasn't passing the vagrant root properly (although it is fixed in 0.16), and even after patching, it still posed a few problems, as the suite attributes were not available to the script, preventing the decision of whether or not to create the additional disk from being made intelligently.


As it looked like the best place to do the pre-processing was outside test-kitchen execution, rake became the obvious solution. We already encourage use of Rakefiles in our cookbooks for testing, so pre-processing would just require adding a few lines. Yes, this would require an additional setup step, but it did have the benefit of not requiring making a local fork of kitchen-vagrant to allow hackiness that didn't necessarily make sense for the entire kitchen community.





A few tasks got added to the cookbook Rakefile:


# Integration tests.
namespace :integration do
  desc 'Run test-kitchen pre-processing scripts'
  task :pre_cmds do
    cmd ='./scripts/write_vagrantfile.rb')

# Clean up
namespace :cleanup do
  desc 'Destroy test-kitchen instances'
  task :kitchen_destroy do
    destroy = do |obj|
      def obj.destroy()
        config.instances.each do |instance|

  desc 'Remove vagrant disks'
  task :rm_vdi do

  desc 'Remove Vagrantfiles/ dir'
  task :rm_vagrantfiles do

  desc 'Remove .kitchen.local.yml'
  task :rm_kitchen_local do
    ::File.unlink('.kitchen.local.yml') if ::File.exist?('.kitchen.local.yml')

desc 'Run full integration'
task integration: %w(integration:pre_cmds integration:vagrant)

desc 'Generate the setup'
task setup: %w(integration:pre_cmds)

desc 'Clean up generated files'
task cleanup: %w(cleanup:kitchen_destroy cleanup:rm_vdi
                 cleanup:rm_kitchen_local cleanup:rm_vagrantfiles)


The write_vagrantfile.rb script is admittedly a bit of a hack; it reads and parses the .kitchen.yml and then adds some glue and writes a .kitchen.local.yml. Because some of the information it needs comes from private methods in the kitchen-vagrant classes, some of those method definitions were ganked and pasted into the script.



write_vagrantfile.rb steps:


  1. Read the .kitchen.yml file and create the associated Kitchen:: objects.
  2. Compile a list of the instances (suites * platforms) and identify those, via attribute, that need an additional disk.
  3. Create custom Vagrantfiles for the Vagrant run for those instances.
  4. Generate the .kitchen.local.yml file to add the vagrantfiles driver option with the appropriate Vagrantfile path for each suite.




The solution is perhaps not the most elegant in terms of pretending like we aren't looking under Kitchen's hood or being 100% integrated into kitchen execution, but it's reasonably lightweight and flexible, and most importantly, it fills a need we had.

Example code for this is available on Ooyala's github.