Just Enough Developed Infrastructure

Behavioral testing with Vagrant - Take 2

(2011-12-15) - Comments

A big thanks to Atlassian for allowing me to post this series!!

Running tests from within the VM

After I covered Puppet Unit Testing, the logical step is writing about Behavioral testing.

While writing this , I can up with a good example of why BDD needs to complement your Unit tests: I have installed the Apache Puppet Module, and all provision ran ok. I wasn't until I tested the webpage with lynx http://localhost that I understood I needed to create a default website. This is of course a trivial example, but I shows you that BDD can help you in testing logical errors.

When this topic arises, most people are familiar with Cucumber Nagios. It contains a series of Cucumber steps that allow you to test http request, amqp, dns, ssh, command.

From what I found, most people would execute these test on the VMs directly. This requires you to install cucumber and all of it's dependent gems in the VM. Gareth RushGrove wrote a great blogpost on packaging cucumber-nagios with fpm

Running tests from outside the VM - Take 1

In some situations, the required gems, libraries might lead to conflicts or introduce dependencies you would rather not have on your production machine. And they would become another point to maintenance in your production machines.

So in a previous blogpost Vagrant Testing,Testing One Two , I already described using modified Cucumber-Nagios steps that interact with Vagrant over ssh.

Running tests from outside the VM - Take 2

But I had a problem with the previous approach. Depending on the situation I would need to run the same tests via different connection methods: vagrant uses ssh, ec2 via fog, openvz via vzctl etc...

So I came up with a new flexible approach: use a configurable command to connect to a vm and have it execute the same steps.

With a little Aruba help

While Cucumber-Nagios slowly moves into Cuken, the SSH steps are getting converted Aruba steps for local exection. And in combination to the ssh-forever steps for ssh interaction.

The Aruba gem is a set of CLI Steps for Cucumber. You can use it to interactively interact with a process or just do a run. Example steps could look like:

Given I run "ssh localhost -p 2222" interactively
And I type "apache2ctl configtest"
And the exit status should be 0

Making it connection neutral

As you can see in the previous step, there is still the connection in the Feature. Not great if we want to run it local. I rephrased it to:

Feature: apache check

  Scenario: see if the apache header is served
    Given I execute `lynx http://localhost --dump` on a running system
    Then the output should match /It works/
    Then the exit status should be 0

  Scenario: check if the apache config is valid
    Given I execute `apache2ctl configtest` on a running system
    Then the exit status should be 0

Writing the logic

Here is the logic to make this work (put it in features/support/step_definitions/remote_system_connect_steps.rb . It uses two environment variables:

SYSTEM_EXECUTE: the command to execute just one command
SYSTEM_CONNECT: the command to connect to the system

Example for vagrant would be:

SYSTEM_EXECUTE: "vagrant ssh_config | ssh -q -F /dev/stdin default"
SYSTEM_CONNECT: "vagrant ssh"

This can be also your favorite knife ssh, vzctl 33 enter, mc-ssh somehost

When /^I execute `([^`]*)` on a running system$/ do |cmd|
  @execute_command=ENV['SYSTEM_EXECUTE']
  @connect_failed=false
  unless @execute_command.nil?
    steps %Q{ When I run `#{@execute_command} "#{cmd}"` }
  else
    @execute_failed=true
    raise "No SYSTEM_EXECUTE environment variable specified"
  end
end

When /^I connect to a running system interactively$/ do
  @connect_command=ENV['SYSTEM_CONNECT']
  @connect_failed=false
  unless @connect_command.nil?
    steps %Q{
        When I run `#{@connect_command}` interactively
    }
  else
    @connect_failed=true
    raise "No SYSTEM_COMMAND environment variable specified"
  end
end

When /^I disconnect$/ do
  steps %Q{ When I type "exit $?" }
end

Monkey Patching Aruba

By default, Aruba uses shellwords to parse the commandlines you pass, it seems to have an issue with "|" symbols. This is the patch I came up with: (in features/support/env.rb)

require 'aruba/cucumber'
require 'shellwords'

# Here we monkey patch Aruba to work with pipe commands
module Aruba
  class Process
    include Shellwords

    def initialize(cmd, exit_timeout, io_wait)
      @exit_timeout = exit_timeout
      @io_wait = io_wait

      @out = Tempfile.new("aruba-out")
      @err = Tempfile.new("aruba-err")
      @process = ChildProcess.build(cmd)
      @process.io.stdout = @out
      @process.io.stderr = @err
      @process.duplex = true
    end
  end
end

After this a regular cucumber run, should work (Note: use a recent cucumber version 1.1.x)

Automating it with Rake

The last part is automating this for Vagrant. For this we create a little rake task:

require "cucumber/rake/task"
task :default => ["validate"]

# Usage rake validate
# - single vm: rake validate
# - multi vm: rake validate vm=logger
Cucumber::Rake::Task.new(:validate) do |task|
    # VM needs to be running already
    vm_name=ENV['vm'] || ""
    ssh_name=ENV['vm'] || "default"
    ENV['SYSTEM_CONNECT']="vagrant ssh #{vm_name}"
    ENV['SYSTEM_EXECUTE']="vagrant ssh_config #{vm_name}| ssh -q -F /dev/stdin #{ssh_name}"
    task.cucumber_opts = ["-s","-c", "features" ]
end

Final words

The solution allows you to reuse the command execution steps, for running them locally, over ssh, or some other connection command.

  • This only works for commands that run over ssh, but I think it is already powerfull to do this. If would require amqp testing, you could probably find a command check as well.
  • Shell escaping is not 100% correct, this needs more work to work with the special characters or quotes inside quotes.
  • When testing, I sometimes miss the context of how a server is created (f.i. the params passed to the puppet manifest or the facts), maybe I could this in a puppet manifests. Not sure on this
  • If there is an interest, I could turn this into a vagrant plugin, to make it really easy.

All code can be found at the demo project: https://github.com/jedi4ever/vagrant-guard-demo

Test Driven Infrastructure with Vagrant, Puppet and Guard

(2011-12-13) - Comments

This is a repost of my SysAdvent blogpost. It's merely here for archival purposes, or for people who read my blog but didn't see the sysadvent blogpost.

Why

Lots has been written about Vagrant. It simply is a great tool: people use it as a sandbox environment to develop their Chef recipes or Puppet manifests in a safe environment.

The workflow usually looks like this:

  • you create a vagrant vm
  • share some puppet/chef files via a shared directory
  • edit some files locally
  • run a vagrant provision to see if this works
  • and if you are happy with it, commit it to your favorite version control repository

Specifically for puppet, thanks to the great work by Nikolay Sturm and Tim Sharpe, we can now also complement this with tests written in rspec-puppet and cucumber-puppet. You can find more info at Puppet unit testing like a pro.

So we got code, and we got tests, what else are we missing? Automation of this process: it's funny if you think of it that we automate the hell out of server installations, but haven't automated the previous described process.

The need to run vagrant provision or rake rspec actually breaks my development flow: I have to leave my editor to run a shell command and then come back to it depending on the output.

Would it not be great if we could automate this whole cycle? And have it run tests and provision whenever files change?

How

The first tool I came across is autotest: it allows one to automatically re-execute tests depending on filesystem changes. Downside is that it could either run cucumber tests or rspec tests.

So enter Guard; it describes itself as a command line tool to easily handle events on file system modifications (FSEvent / Inotify / Polling support). Just what we wanted!

Installing Guard is pretty easy, you require the following gems in your Gemfile

gem 'guard'
gem 'rb-inotify', :require => false
gem 'rb-fsevent', :require => false
gem 'rb-fchange', :require => false
gem 'growl', :require => false
gem 'libnotify', :require => false

As you can tell by the names, it uses different strategies to detect changes in your directories. It uses growl (if correctly setup) on Mac OS X and libnotify on Linux to notify you if your tests pass or fail. Once installed you get a command guard.

Guard uses a configuration file Guardfile, which can be created by guard init. In this file you define different guards based on different helpers: for example there is guard-rspec, guard-cucumber and many more. There is even a guard-puppet(which we will not use because it works only for local provisioning)

To install one of these helpers you just include it in your Gemfile. We are using only two here:

gem 'guard-rspec'
gem 'guard-cucumber'

Each of these helpers has a similar way of configuring themselves inside a Guardfile. A vanilla guard for a ruby gem with rspec testing would look like this:

guard 'rspec' do
  watch(%r{^spec/.+_spec\.rb$})
  watch(%r{^lib/(.+)\.rb$})     { |m| "spec/lib/#{m[1]}_spec.rb" }
  watch('spec/spec_helper.rb')  { "spec" }
end

Whenever a file that matches a watch expression changes, it would run an rspec test. By default if no block is supplied, the file itself is run. You can alter the path in a block as in the example.

Once you have a Guardfile you simply run guard (or bundle exec guard) to have it watch changes. Simple hu?

What

Vagrant setup

Enter our sample puppet/vagrant project. You can find the full source at http://github.com/jedi4ever/vagrant-guard-demo It's a typical vagrant project with the following tree structure:(only 3 levels shown)

├── Gemfile
├── Gemfile.lock
├── Guardfile
├── README.markdown
├── Vagrantfile
├── definitions # Veewee definitions
│   └── lucid64
│       ├── definition.rb
│       ├── postinstall.sh
│       └── preseed.cfg
├── iso # Veewee iso
│   └── ubuntu-10.04.3-server-amd64.iso
└── vendor
    └── ruby
        └── 1.8

Puppet setup

The project follows Jordan Sissel's idea of puppet nodeless configuration. To specify the classes to apply to a host, we use a fact called: server_role. We read this from a file data/etc/server_tags via a custom fact (inspired by self-classifying puppet node).

This allows us to only require one file, site.pp. And we don't have to fiddle with our hostname to get the correct role. Also if we want to test multiple roles on this one test machine, just add another role to the data/etc/server_tags file.

├── data
│   └── etc
│       └── server_tags

$ cat data/etc/server_tags
role:webserver=true

The puppet modules and manifests can be found in puppet-repo. It has class role::webserver which includes class apache.

puppet-repo
├── features # This is where the cucucumber-puppet catalog policy feature lives
│   ├── catalog_policy.feature
│   ├── steps
│   │   ├── catalog_policy.rb
│   └── support
│       ├── hooks.rb
│       └── world.rb
├── manifests
│   └── site.pp #No nodes required
└── modules
    ├── apache
    |    <module content>
    ├── role
    │   ├── manifests
    │   │   └── webserver.pp # Corresponds with the role specified
    │   └── rspec
    │       ├── classes
    │       └── spec_helper.rb
    └── truth # Logic of puppet nodeless configuration
        ├── lib
        │   ├── facter
        │   └── puppet
        └── manifests
            └── enforcer.pp

Puppet - Vagrant setup

These are the settings we use in our Vagrant file to make puppet work:

config.vm.share_folder "v-data", "/data", File.join(File.dirname(__FILE__), "data")
# Enable provisioning with Puppet stand alone.  Puppet manifests
# are contained in a directory path relative to this Vagrantfile.
config.vm.provision :puppet, :options => "--verbose"  do |puppet|
  puppet.module_path = ["puppet-repo/modules"]
  puppet.manifests_path = "puppet-repo/manifests"
  puppet.manifest_file  = "site.pp"
end

Puppet tests setup

The cucumber-puppet tests will check if the catalog compiles for role role::webserver

Feature: Catalog policy
  In order to ensure basic correctness
  I want all catalogs to obey my policy

  Scenario Outline: Generic policy for all server roles
    Given a node with role "<server_role>"
    When I compile its catalog
    Then compilation should succeed
    And all resource dependencies should resolve

    Examples:
      | server_role |
      | role::webserver |

The rspec-puppet tests will check if the package http gets installed

require "#{File.join(File.dirname(__FILE__),'..','spec_helper')}"
describe 'role::webserver', :type => :class do
  let(:facts) {{:server_tags => 'role:webserver=true',
      :operatingsystem => 'Ubuntu'}}
  it { should include_class('apache') }
  it { should contain_package('httpd').with_ensure('present') }
end

Guard setup

To make Guard work with a setup like our puppet-repo directory we need to change some things. This has mostly to do with conventions used in development projects where Guard is normally used.

Fixing Guard-Cucumber to read from puppetrepo/features

The first problem is that the Guard-Cucumber gem standard reads it's features from features directory. This is actually hardcoded in the gem. But nothing a little monkey patching can't solve:

require 'guard/cucumber'

# Inline extending the ::Guard::Cucumber
# Because by default it only looks in the ['features'] directory
# We have it in ['puppet-repo/features']
module ::Guard
  class ExtendedCucumber < ::Guard::Cucumber
    def run_all
      passed = Runner.run(['puppet-repo/features'], options.merge(options[:run_all] || { }).merge(:message => 'Running all features'))

      if passed
        @failed_paths = []
      else
        @failed_paths = read_failed_features if @options[:keep_failed]
      end

      @last_failed = !passed

      throw :task_has_failed unless passed
    end
  end
end

# Monkey patching the Inspector class
# By default it checks if it starts with /feature/
# We tell it that whatever we pass is valid
module ::Guard
  class Cucumber
    module Inspector
      class << self
        def cucumber_folder?(path)
          return true
        end
      end
    end
  end
end

Orchestration of guard runs

The second problem was to have Guard only execute the Vagrant provision when BOTH the cucumber and rspec tests would be OK. Inspired by the comments of Netzpirat, I got it working so that the block vagrant provision would only execute on both tests being complete.

# This block simply calls vagrant provision via a shell
# And shows the output
def vagrant_provision
  IO.popen("vagrant provision") do |output|
    while line = output.gets do
      puts line
    end
  end
end

# So determine if all tests (both rspec and cucumber have been passed)
# This is used to only invoke the vagrant_provision if all tests show green
def all_tests_pass
  cucumber_guard = ::Guard.guards({ :name => 'extendedcucumber', :group => 'tests'}).first
  cucumber_passed = cucumber_guard.instance_variable_get("@failed_paths").empty?
  rspec_guard = ::Guard.guards({ :name => 'rspec', :group => 'tests'}).first
  rspec_passed = rspec_guard.instance_variable_get("@failed_paths").empty?
  return rspec_passed && cucumber_passed
end

Guard matchers

With all the correct guards and logic setup, it's time to specify the correct options to our Guards.

group :tests do

  # Run rspec-puppet tests
  # --format documentation : for better output
  # :spec_paths to pass the correct path to look for features
  guard :rspec, :version => 2, :cli => "--color --format documentation", :spec_paths => ["puppet-repo"]  do
    # Match any .pp file (but be carefull not to include any dot-temporary files)
    watch(%r{^puppet-repo/.*/[^.]*\.pp$}) { "puppet-repo" }
    # Match any .rb file (but be carefull not to include any dot-temporary files)
    watch(%r{^puppet-repo/.*/[^.]*\.rb$}) { "puppet-repo" }
    # Match any _rspec.rb file (but be carefull not to include any dot-temporary files)
    watch(%r{^puppet-repo/.*/[^.]*_rspec.rb})
  end

  # Run cucumber puppet tests
  # This uses our extended cucumber guard, as by default it only looks in the features directory
  # --strict        : because otherwise cucumber would exit with 0 when there are pending steps
  # --format pretty : to get readable output, default is null output
  guard :extendedcucumber, :cli => "--require puppet-repo/features --strict --format pretty" do

    # Match any .pp file (but be carefull not to include any dot-temporary files)
    watch(%r{^puppet-repo/[^.]*\.pp$}) { "puppet-repo/features" }

    # Match any .rb file (but be carefull not to include any dot-temporary files)
    watch(%r{^puppet-repo/[^.]*\.rb$}) { "puppet-repo/features" }

    # Feature files are monitored as well
    watch(%r{^puppet-repo/features/[^.]*.feature})

    # This is only invoked on changes, not at initial startup
    callback(:start_end) do
      vagrant_provision if all_tests_pass
    end
    callback(:run_on_change_end) do
      vagrant_provision if all_tests_pass
    end
  end

end

The full Guardfile is on github

Run it

From within the top directory of the project type

$ guard

Now open a second terminal and change some of the files and watch the magic happen.

Final remarks

The setup described is an idea I only recently started exploring. I'll probably enhance this in the future or may experience other problems.

For the demo project, I only call vagrant provision, but this can of course be extended easily. Some ideas:

  1. Inspired by Oliver Hookins - How we use Vagrant as a throwaway testing environment:
  2. use sahara to create a snapshot just before the provisioning
  3. have it start from a clean machine when all tests pass
  4. Turn this into a guard-vagrant gem, to monitor files and tests

Devops from a sysadmin perspective

(2011-12-07) - Comments

This year LISA (Large Installation System Administration) 2011 Conference has a theme on "devops".

The LISA crowd has been practicing automation for a long time, and many of them just look at devops as something they have always been doing.

So they have asked me to write an article for Usenix ;Login magazine to explain devops from a sysadmin perspective. As the article requires a subscription ,I'm re-posting it here for others to enjoy :)

Introduction

While there is not one true definition of devops (similar to cloud), four of it's key-points resolve around Culture, Automation, Measurement and Sharing (CAMS). In this article we will show how this affects the traditional thinking of the sysadmin.

As a sysadmin you are probably familiar with the Automation and Measurement part: it has been good and professional practice to script/automate work to make things faster and repeatable. Gathering metrics and doing monitoring is an integral part of the job to make sure things are running smoothly.

The pain

For many years, operations (of which the sysadmin is usually part) has been seen as an endpoint in the software delivery process: developers code new functionality during a project in isolation from operations and once the software is considered finished, it is presented to the operations departement to run it.

During deployment a lot of issues tend to surface: some typical examples are the development and test environment not being representative to the production environment, or that not enough thought has been given to backup and restore strategies. Often it is too late in the project to change much of the architecture and structure of the code and it gives way to many fixes and ad-hoc solutions. This friction has created a disrespect between the two groups: developers feel that operations knows nothing about software, and operation feel that developers know nothing about running servers. Management tends to keep those two groups in isolation from each other, keeping the interaction at the minimum required. The result is a 'wall of confusion'

Culture of collaboration

Historically two drivers have fuelled devops: the first one was Agile Development which led in many companies to many more deployment than operations was used to. The second one was Cloud and large scale web operations , where the scale required a much closer collaboration between development and operations.

When things really go wrong, organizations often create a multi-disciplined task force to tackle production problems. Truth is that in today's IT, environments have become so complex that they can't be understood by one person or even one group. Therefore instead of separating developers and operations as we used to do, we need to bring them together more closely: we need more practice, and the motto should be "if it's hard do it more often".

Devops recognizes that software only provides value if it's running production and running a server without software does not provide value either. Development and operations are both working to serve the customer not for running their own department.

Although many sysadmins have been collaborating with other departments, it has never been seen as a strategic advantage. The cultural part of devops, seeks to promote this constant collaboration across silos, in order to better meet the business demands. It goes for 'friction-less' IT and promotes the cross-departmental/cross-disciplinary approach.

A good place to get started with collaboration are places where the discussion often escalates: deployment, packaging, testing, monitoring, building environments. These places can be seen as boundary objects: places where every silo has it's own understanding of. These are exactly the places where technical debt accumulates so they should contain real pain issues.

Culture of sharing

Silo's exist in many forms in the organization, not only between developers and operations. In some organizations there are even silos inside of operations: network, security, storage, servers avoid collaboration and each work in their own world. This has been referred to as the Ops-Ops problem. So in geek-speak devops is actually a wildcard for devops* collaboration.

Devops doesn't mean all sysadmins need to know how to code software now, or all developers need to know how to install a server. By collaborating constantly, both groups can learn from each other, but can also rely on each other to do the work. A similar approach has been promoted by Agile between developers and testers. Devops can be seen as the extend of bringing system administrators into the Agile equation.

Starting the conversation sometimes takes courage but think about the benefits: you get to learn the application as it grows, and you can actively shape it by providing your input during the process. A sysadmin has a lot to offer to the developers: f.i. you have the knowledge of how production looks like, therefore you can build representative environment in test/dev. You can be involved in loadtesting, failover testing. Or you can setup a monitoring system that developmers can use to see what's wrong. Give access to production logs so developers can understand real world usage.

A great way to share information and knowledge is by pairing together with developer or collegues: while you are deploying code he comments on what the impact is on the code and allows you to directly ask questions. This interaction is of great value to understand both worlds better.

Revisiting Automation

Like specified in the Agile Manifesto, devops values "Individuals and interactions over processes and tools". The great thing about tools is that they are concrete and can have a direct benefit as opposed to culture. It was hard to grasp the impact of Virtualization and Cloud unless you started doing it. Tools can shape the way we work and consequently change our behavior.

A good example is Configuration Management and Infrastructure as code. A lot of people rave about it's flexibility and power for the automation. If you look beyond the effect of saving time, you will find that it also has a great sharing aspects: It has created a 'shared' language that allows you can know exchange the way you manage systems with collegues and even outside your company by publishing recipes/cookbooks on github. Because we know use concepts as version control and testing we have a common problemspace with developers. And most importantly the automation is freeing us from the trivial stuff and allows us to discuss and focus on the stuff that really matters.

Revisiting Metrics

Measuring the effects of collaboration can't be done by measuring the number of interactions, after all more interaction doesn't mean a better party. It's similar to a black hole , you have to look at the objects nearby. So how do you see that things are improving? As an engineer you collect metrics about number of incidents, failed deploys, number of succesful deploys, number of tickets. Instead of keeping these information in their own silo, you radiate this to the other parts of the company so they could learn from them. Celebrate successes and failure and learn from them. Doing post-mortems with all parties involved and improve on it. Again this changes the focus of metrics and monitoring from only fast fixing to feedback to the whole organization. Aim to optimize the whole instead of only your own part.

The secret sauce

Several of the 'new' companies have been front-runners in these practices. Google with their two-pizza team approach, Flick with their 10 deploys a day where front runners in the field, but also more traditional companies like National Instruments are seeing the value from this culture of collaboration. They see collaboration as the 'secret sauce', that will set them apart from their competition. Why? Because it recognizes the individual not as a resource but as resourceful to tackle the challenges that exist in this complex world called IT.

Links index:

  1. Patrick Debois's Devopsdays Melbourne Keynote
  2. John Willis, What devops means to me
  3. Damon Edwards, what is devops
  4. Israel Gat, boundary objects in devops
  5. Agile Manifesto
  6. Ernest Mueller, Originality and Operations
  7. Cliff Stoll, The Cuckoos Egg
  8. Andrew Shaefer, Israel Gat, Patrick Debois Velocity Conference 2011 Devops Metrics"
  9. Amazon Architecture
  10. John Allspaw, 10 deploys per day - dev and ops cooperation at flickr
  11. Jesse Robbins, Operations is a competitive advantage

Puppet unit testing like a pro

(2011-12-05) - Comments

A big thanks to Atlassian for allowing me to post this series!!

In our previous blogpost on Puppet Versioning, we described the most basic check to see if a puppet manifest was valid. We used the parseonly function to see if it would compile.

Until know this means we have only have if the compiler is happy, not that it performs the function it needs to do. In 2009 after the first devopsdays I wrote a collection of Test Driven Infrastructure Links . This was obviously inspired by Lindsay Holmwood's talk on cucumber-nagios.

On the Opscode chef front, Stephen Nelson-Smith wrote a great book Test-driven Infrastructure with Chef on how to do this. Also see the cuken project where re-usable cucumber steps are grouped.

Because we are using Puppet here at Atlassian, I was out to understand the current state of puppet testing. A lot can already be found at http://puppetlabs.com/blog/testing-modules-in-the-puppet-forge/

Note that I've purposely named this blog 'Puppet unit testing', as the tests I'm describing now, don't run against an actual system. Therefore it's hard to test the actual behavior.

Tip 1: cucumber-puppet

Inspired by Lindsay Holmwood's talk on cucumber-nagios and Ohad Levy's manitest Nikolay Sturm created cucumber-puppet

In his post on Thoughts on testing puppet manifests he explains that the idea of writing tests is NOT about duplicating the code, and he identified the most common problems he was facing are:

  • catalog does not compile: syntax errors, missing template files, ..
  • catalog does compile, but cannot be applied: unreachable or non-existent resources, missing file resources in repo
  • catalog does applies, but is faulty: faulty files, due to empty manifests variables or wrong values, missing dependencies (wrong order ...), files are installed without ensuring a directory ...

An important advice is:

Resource specifications can be useful for documentation purposes or refactorings. However, there is a risk of reimplementing your Puppet manifest, so be wary.

$ cd puppet-mymodule
$ gem install cucumber-puppet

Write features per module, this is the structure we are aiming at:

module
  +-- manifests
  +-- lib
  +-- features
       +-- support
       |     +-- hooks.rb
       |     +-- world.rb
       +-- catalog
       +-- feature..

Generate a cucumber-puppet world:

$ cucumber-puppet-gen world
Generating with world generator:
     [ADDED]  features/support/hooks.rb
     [ADDED]  features/support/world.rb
     [ADDED]  features/steps

# Adjust the paths to your modules and manifests
$ cat features/support/hooks.rb
Before do
  # adjust local configuration like this
  # @puppetcfg['confdir']  = File.join(File.dirname(__FILE__), '..', '..')
  # @puppetcfg['manifest'] = File.join(@puppetcfg['confdir'], 'manifests', 'site.pp')
  # @puppetcfg['modulepath']  = "/srv/puppet/modules:/srv/puppet/site-modules"

  # adjust facts like this
  @facts['architecture'] = "i386"
end

# Nothing exciting here
$ cat features/support/world.rb

require 'cucumber-puppet/puppet'
require 'cucumber-puppet/steps'

World do
  CucumberPuppet.new
end

Generating a policy feature:

$ cucumber-puppet-gen policy
Generating with policy generator:
     [ADDED]  features/catalog

# Notice the <hostname>.example.com.yaml
# These files contain the facts to test your catalog against
# 
$ cat features/catalog/policy.feature 
Feature: General policy for all catalogs
  In order to ensure applicability of a host's catalog
  As a manifest developer
  I want all catalogs to obey some general rules

  Scenario Outline: Compile and verify catalog
    Given a node specified by "features/yaml/<hostname>.example.com.yaml"
    When I compile its catalog
    Then compilation should succeed
    And all resource dependencies should resolve

    Examples:
      | hostname  |
      | localhost |

To do an actual run:

$ cucumber-puppet features/catalog/policy.feature 
Feature: General policy for all catalogs
  In order to ensure applicability of a host's catalog
  As a manifest developer
  I want all catalogs to obey some general rules

  Scenario Outline: Compile and verify catalog                            # features/catalog/policy.feature:6
    Given a node specified by "features/yaml/<hostname>.example.com.yaml" # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:1
    When I compile its catalog                                            # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:14
    Then compilation should succeed                                       # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:48
    And all resource dependencies should resolve                          # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:28

    Examples: 
      | hostname  |
      | localhost |
      Cannot find node facts features/yaml/localhost.example.com.yaml. (RuntimeError)
      features/catalog/policy.feature:7:in `Given a node specified by "features/yaml/<hostname>.example.com.yaml"'

Failing Scenarios:
cucumber features/catalog/policy.feature:6 # Scenario: Compile and verify catalog

1 scenario (1 failed)
4 steps (1 failed, 3 skipped)
0m0.006s

List of commands:

Generators for cucumber-puppet

Available generators
    feature                          Generate a cucumber feature
    policy                           Generate a catalog policy
    testcase                         Generate a test case for the test suite
    testsuite                        Generate a test suite for puppet features
    world                            Generate cucumber step and support files

General options:
    -p, --pretend                    Run, but do not make any changes.
    -f, --force                      Overwrite files that already exist.
    -s, --skip                       Skip files that already exist.
    -d, --delete                     Delete files that have previously been generated with this generator.
        --no-color                   Don't colorize the output
    -h, --help                       Show this message
        --debug                      Do not catch errors

He has also added support for testing exported-resources.

And for a more practical explanation, see how Oliver Hookins describes the way Nokia uses cucumber-puppet

Scenario: Proxy host and port have sensible defaults
  Given a node of class "mymodule::myapp"
  And we have loaded "test" settings
  And we have unset the fact "proxy_host"
  And we have unset the fact "proxy_port"
  When I compile the catalog
  Then there should be a file "/etc/myapp/config.properties"
  And the file should contain "proxy.port=-1"
  And the file should contain /proxy\.host=$/

----

Then /^the file should contain "(.*)"$/ do |text|
  fail "File parameter 'content' was not specified" if @resource["content"].nil?
  fail "Text content [#{text}] was not found" unless @resource["content"].include?(text)
end

Then /^the file should contain \/([^\"].*)\/$/ do |regex|
  fail "File parameter 'content' was not specified" if @resource["content"].nil?
  fail "Text regex [/#{regex}/] did not match" unless @resource["content"] =~ /#{regex}/
end

Tip 2: rspec-puppet

While the idea on using specs and puppet is not new (https://github.com/jes5199/puppet_spec), the new tool on the block is rspec-puppet brought to us by Tim Sharpe. The same person who gave us vim-puppet and puppet-lint

Like the cucumber-puppet structure, the idea is to have specs directory close to your module:

module
  +-- manifests
  +-- lib
  +-- spec
       +-- spec_helper.rb
       +-- classes
       |     +-- <class_name>_spec.rb
       +-- defines
       |     +-- <define_name>_spec.rb
       +-- functions
             +-- <function_name>_spec.rb

I found it useful to change the default spec_helper.rb as the default

require 'rspec-puppet'

RSpec.configure do |c|
   c.module_path = File.expand_path(File.join(File.dirname(__FILE__), '..', '..'))
   c.manifest_dir = File.expand_path(File.join(File.dirname(__FILE__), '..', '..','..','manifests'))
end


desc "Run specs check on puppet manifests"
RSpec::Core::RakeTask.new(:spec) do |t|
   t.pattern = './demo-puppet/modules/**/*_spec.rb' # don't need this, it's default
   t.verbose = true
   t.rspec_opts = "--format documentation --color"
    # Put spec opts in a file named .rspec in root
  end

Here is a quick example for checking if the class apache installs a package httpd when on a Debian system

require "#{File.join(File.dirname(__FILE__),'..','spec_helper')}"

describe 'apache', :type => :class do
  let (:title { 'basic' })
  let(:params) { { } }
  let(:facts) { {:operatingsystem => 'Debian', :kernel => 'Linux'} }

  it { should contain_package('httpd').with_ensure('installed') }
end

A more detailed description can be found at

For more generic information on rspec:

Conclusion cucumber-puppet vs rspec-puppet

I think you can write your tests in both to do the same. Currently they both support 2.6 and 2.7

I found the rspec-puppet a bit simpler to juggle with providing params like :name or :facts. The yaml file didn't feel to flexible to me. Also cucumber seems to install more dependent gems, that might inflict with other projects.

But as Nikolay already said:

"don't duplicate your manifests in your tests" Focus on the catalog problems he described earlier and test your logic. Don't test if puppet is doing it's job, test that your logic it's doing it's job.

This is why I called them unit-tests, they don't test the real functionality. (That's for the next blogpost)

Tip 3: puppet-lint

To check you files against programming style you can use https://github.com/rodjek/puppet-lint. It will check for Rules on Spacing, Identation & Whitespace , Quoting, Resources, Conditionals, Classes

An easy way to integrate it in your Rakefile is:

require 'puppet-lint'

desc "Run lint check on puppet manifests"
task :lint do
linter =  PuppetLint.new
  Dir.glob('./demo-puppet/modules//**/*.pp').each do |puppet_file|
    puts "Evaluating #{puppet_file}"
    linter.file = puppet_file
    linter.run
  end
  fail if linter.errors?
 end

Now you can simply run:

$ rake lint

Tip 4: go wild and build your own test/catalog logic

After having a look at the rspec-puppet logic, I looked deeper in the way to walk trough the catalog object. This is pretty much work in progress, but the idea is find a way to look at changes in the catalog.

The following is a list of useful examples on understanding on how to work with puppet in ruby code:

The first list of links are some fun tools written by Dean Wilson of www.puppetcookbook.com fame:

R.I. Pienaar of Mcollective Fame shows a way to create diff on a catalog. this can be useful to understand what tests to run in between changes:

This final gist shows how to walk through the catalog and check the classes and resources available:

https://gist.github.com/1430062#file_puppet_demo.rb

Puppet versioning like a pro

(2011-12-05) - Comments

A big thanks to Atlassian for allowing me to post this series!!

There is NO reason, not to use a version control system while developing puppet manifest/modules. Stating that should be an open door. It allows you to go back in time, share things more easily and track your changes. There is a lot of information out there on how to work with git or any other system. But here a few tips that might help you developing modules:

Tip 1 : give each module it's own repo and use a superproject to join

In a lot of blogposts and even in the excellent Pro Puppet book I see people checking in their entire environment directory into version control.

I'm all for version control but if you manage your modules dir as one flat repository, you loose the way to easily update and share modules from the forge. In essence you are doing a copy that starts living it's own life.

The idea goes like this:

/etc/puppet/environments/development/modules (super-project repo)
  -- puppet-apache (sub-project repo)
  -- puppet-mysql (sub-project repo)
  ...

The super-project repo will contain links to the submodules it uses. This allows the reuse of the sub-project repos in different super-projects. F.i. puppet-modules-team1, puppet-modules-team2 could be superprojects and use different sub-modules.

Using Git

Git has the concept of submodules that allows you to link a parent repository with subprojects. Further detailed documentation can be found at http://book.git-scm.com/5_submodules.html

Using this approach with puppet is nicely described at https://we.riseup.net/riseup+tech/puppet-git-submodules, with some tips at https://labs.riseup.net/code/documents/7

I was always scared away of using submodules, because things like checking in the superproject first and forgetting the submodule, makes part of it unusable. Let alone adding the submodule directoy with a 'slash' has bitten me a few times.

It's a good approach but it requires being awake :)

Using Mercurial

At Atlassian we heavily use Mercurial version control. So I learned the power of Mercurial .

To create a development repo with a specific apache submodule

$ cd /etc/puppet/environments/
$ hg init development
$ cd development/
$ mkdir modules
$ echo 'modules/apache = [git]git://github.com/puppet-modules/puppet-apache.git' > .hgsub
$ hg add .hgsub
$ git clone git://github.com/puppet-modules/puppet-apache.git modules/apache
$ hg ci -mapache

Now if we are checking this out to our test environment, we'll see that it automatically checks out the submodules.

$ cd /etc/puppet/environments
$ hg clone development/ test
updating to branch default
resolving manifests
getting .hgsub
getting .hgsubstate
cloning subrepo modules/apache from git://github.com/puppet-modules/puppet-apache.git
remote: Counting objects: 177, done.
remote: Compressing objects: 100% (94/94), done.
remote: Total 177 (delta 59), reused 168 (delta 52)
Receiving objects: 100% (177/177), 22.97 KiB, done.
Resolving deltas: 100% (59/59), done.
2 files updated, 0 files merged, 0 files removed, 0 files unresolved

I found this workflow:

  • less scary: no way to wrongly add directories
  • it's handy that you can add both svn, git and hg subrepositories
  • submodules are checked out by default (no git submodules init, update)

Note:

Tip 2: Think about your git workflow

In many of the examples you'll see people just commit to the 'master' branch. This works of course but if you are working on different modules/features, it's best to think about your git workflow. There are a lot of blogpost describing how to work in feature branches. Still I found it hard to remember the commands for branching, deleting old branches and so on.

Thanks to Tim Dysinger I learned about Gitflow.

It's basicly git helpers that take away a lot of the pain for working on hotfixes, features, releases. The usage is pretty easy:

# Create the master/developer/release/hotfix branches
$ git flow init

# Start working on a feature (branched from develop)
$ git flow feature start feature1
... do some work
$ git add ...somework...
$ git commit -m "somework feature1"

# This will merge feature1 back to develop
$ git flow feature finish feature1

# Now lets start a release
$ git flow release start release1
... do some work
$ git add ...somework...
$ git commit -m "release1"

# This will merge release1 into master
$ git flow release finish release1

The whole idea is described in details at http://nvie.com/posts/a-successful-git-branching-model/ . And the following video will show you how it works.

More information can also be found at:

Tip 3: Use pre/post-commit hooks

Even with the awesome editor support we previously described, it's still easy to miss a semi-column, or have an incorrect puppet syntax.

Using pre-commits

It's good practice to verify the syntax of your puppet manifests before comitting them to version control. This is well described on the puppetlabs version control page.

In essence before you commit, it will execute 

puppet --parseonly
to check if the syntax is correct.

Using post-commits

A lesser used technique is to run the same check on post-commit:

Instead of running your master repository out of /etc/puppet/environment [PROD], you checkin to an intermediate repository [CHECK].

developer -> LOCAL REPO (pre-commit) -> push -> CHECK REPO -> (post-commit) -> PROD REPO

In the post commit you can:

  • also verify the syntax in case someone didn't check before commiting.
  • if all successfull push the repo to the PROD directories

This helps overcome the problem that your puppetmaster does a run while the repo is an incorrect state.

More details are described at Git workflow and Puppet Environments

More to come

The next post will be about different ways to test your puppet manifests

Puppet editing like a pro

(2011-12-05) - Comments

I've spent some time recently on setting up my environment to work more productively on writing puppet manifests. This blogpost highlights some of the findings to get me more productive on editing puppet files and modules. Some older information can be found at Editor Tips on the puppetlabs website.

Tip 1: Syntax highlighting,snippet completion

Puppet syntax is very specific, it's important to get clues about missing curly braces, semi-colums, etc .. as fast as possible. There is support for this in the most common editors:

Using Textmate:

@Masterzen has created a textmate bundle for use with puppet. You can find it at https://github.com/masterzen/puppet-textmate-bundle.

Michael Halligan describes how to install it from the commandline

mkdir -p /Library/Application\ Support/TextMate/Bundles
cd /Library/Application\ Support/TextMate/Bundles
git clone git://gitorious.org/git-tmbundle/mainline.git Git.tmbundle
git clone http://git.gitorious.org/git-tmbundle/mainline.git Git.tmbundle
git clone https://github.com/masterzen/puppet-textmate-bundle.git Puppet.tmbundle
git clone https://github.com/drnic/Chef.tmbundle.git Chef.tmbundle
osascript -e 'tell app "TextMate" to reload bundles'

Using VIM:

If textmate is not your thing, here is how you can pimp up your vim:

When you look around for puppet/vim integration there seem to have been some re-incarnations:

The most advanced vim-puppet integration I could currently(Dec/2011) find is : - the vim-puppet Tim Sharpe created. - https://github.com/rodjek/vim-puppet

To use the vim-puppet plugin, you're best to use pathogen written by Tim Pope. I've followed the instructions at http://tammersaleh.com/posts/the-modern-vim-config-with-pathogen.

I've enabled the following plugins in my update_bundles script

git_bundles = [
  "git://github.com/astashov/vim-ruby-debugger.git",
  "git://github.com/ervandew/supertab.git",
  "git://github.com/godlygeek/tabular.git",
  "git://github.com/hallison/vim-rdoc.git",
  "git://github.com/msanders/snipmate.vim.git",
  "git://github.com/pangloss/vim-javascript.git",
  "git://github.com/scrooloose/nerdtree.git",
  "git://github.com/timcharper/textile.vim.git",
  "git://github.com/tpope/vim-cucumber.git",
  "git://github.com/tpope/vim-fugitive.git",
  "git://github.com/tpope/vim-git.git",
  "git://github.com/tpope/vim-haml.git",
  "git://github.com/tpope/vim-markdown.git",
  "git://github.com/tpope/vim-rails.git",
  "git://github.com/tpope/vim-repeat.git",
  "git://github.com/tpope/vim-surround.git",
  "git://github.com/tpope/vim-vividchalk.git",
  "git://github.com/tsaleh/taskpaper.vim.git",
  "git://github.com/tsaleh/vim-matchit.git",
  "git://github.com/tsaleh/vim-shoulda.git",
  "git://github.com/tsaleh/vim-tcomment.git",
  "git://github.com/tsaleh/vim-tmux.git",
  "git://github.com/vim-ruby/vim-ruby.git",
  "git://github.com/vim-scripts/Gist.vim.git",
  "git://github.com/scrooloose/syntastic",
  "git://github.com/rodjek/vim-puppet.git",
  "git://github.com/vim-scripts/Specky.git"
]

Most notable plugins:

  • Tabular gives you automatic => alignment
  • Syntastic gives you syntax feedback while you edit files
  • Snipmate gives you the snippets on tab expansion
  • Specky gives you functionality for rspec files
  • vim-ruby gives you extra functionality for ruby files
  • vim-cucumber gives you functionality for cucumber files

For more information on the vim-puppet project go to:

https://github.com/rodjek/vim-puppet/

The snippets that are expanded in the vim-puppet plugin can be found at:

https://github.com/rodjek/vim-puppet/blob/master/snippets/puppet.snippets

Tip 2: don't create modules structure by hand

I keep forgetting the correct structure, files etc.. when I create a new module. Luckily there is an easy way to generate a puppet module structure using the puppet-module gem

$ gem install puppet-module
$ puppet-module
Tasks:
  puppet-module build [PATH_TO_MODULE]                 # Build a module for release
  puppet-module changelog                              # Display the changelog for this tool
  puppet-module changes [PATH_TO_MODULE]               # Show modified files in an installed module
  puppet-module clean                                  # Clears module cache for all repositories
  puppet-module generate USERNAME-MODNAME              # Generate boilerplate for a new module
  puppet-module help [TASK]                            # Describe available tasks or one specific task
  puppet-module install MODULE_NAME_OR_FILE [OPTIONS]  # Install a module (eg, 'user-modname') from a repositor...
  puppet-module repository                             # Show currently configured repository
  puppet-module search TERM                            # Search the module repository for a module matching TERM
  puppet-module usage                                  # Display detailed usage documentation for this tool
  puppet-module version                                # Show the version information for this tool

Options:
  -c, [--config=CONFIG]  # Configuration file
$ puppet-module generate puppetmodule-apache
=========================================================================================
Generating module at /Users/patrick/demo-puppet/modules/puppetmodule-apache
-----------------------------------------------------------------------------------------
puppetmodule-apache
puppetmodule-apache/tests
puppetmodule-apache/tests/init.pp
puppetmodule-apache/spec
puppetmodule-apache/spec/spec_helper.rb
puppetmodule-apache/spec/spec.opts
puppetmodule-apache/README
puppetmodule-apache/Modulefile
puppetmodule-apache/metadata.json
puppetmodule-apache/manifests
puppetmodule-apache/manifests/init.pp

Tip 3 - Geppetto: a Puppet IDE

The folks from cloudsmith have created an Eclipse based editor called Geppetto. It integrates the syntax highlighting, module creation etc... and has nice integration with the Puppet forge

Note: this NOT related with the Gepetto (one P) project by Alban Peignier

James Turnbull was so kind to make a quick screencast on how it works:

But remember it's Java based, so it might take a while to fire it up :)

Manage Non-Cloud Dev+Test Environment With a Cloud Hat On

(2011-11-23) - Comments

Over the years I've moved from Production to Test to Development and back to Production.

This presentation is a summary of my learnings along the way:

  • After running larger production environments I wondered how it would be different in the cloud
  • What would be extra if you already have configmt, continouous delivery etc..
  • I learned that selfservicing and API's can make a huge difference
  • and I plea for more IT companies to apply these principles in their internal IT.
  • not only for machine provisioning , but also for logging, monitoring, etc ..

Here's my presentation from cloudcamp Ghent 2011 that was brilliantly organized by Frederik Van Hecke

Manage your dev-test-prod like a cloud
View more presentations from Patrick Debois

Advanced Virtualbox and Vagrant tips for VLAN,PXE and Hostonly networking

(2011-10-04) - Comments

I've raved about Vagrant many many times before.

Initially it was concieved for single machines running behind a NAT interface. Later support for hostonly-networking was added, opening the possibility to have multiple host talk together over a private network.

But what if the network setup of production is much more complex than this? In this blogpost I'm going to explain to reproduce the following network setup in a vagrant setup:

The network setup

          HOST
            |
            |
          eth2                          eth2
          (NAT)                         (NAT)
            |                             |
        [ Gateway ]      <---->      [ Mgmt Host]  <----->   [Servers1,2]
               br10              br10          eth3       br0
              (eth0.10,        (eth0.10,                  (eth0
               eth1.10)         eth1.10)                   eth1)
              vboxnet0          vboxnet0     vboxnet1    vboxnet1

This is quite a typical production setup:

gateway

  • it acts as a incoming/outgoing firewall, squid caching proxy , nginx etc...
  • it has two production interfaces eth0,eth1 which are VLAN-ed (ID 10) and bridged together

management host

  • connected to the gateway for in and outside traffic
  • contains all necessary tools to manage the servers, dhcp, tftpboot, etc..

servers

  • all servers get installed via the mgmt host
  • server1 will only get installed via PXE
  • server2 will have no disk attached and will boot from the network everytime

Preparing a basebox

Almost all baseboxes out there assume that the DHCP interface for the NAT interface resides on the eth0 interface. Moving the NAT interface to eth2, requires you to change the /etc/network/interfaces to indicate that dhcp is done on eth2, and eth0,eth1 are not dhcp. Otherwise bootup will be very slow, waiting for a timeout, resulting in no ip addresses.

We've used Veewee to add another postinstall script to change the /etc/network/interfaces file to the state we needed. Resulting in a 'squeeze64'

sed -i -e "s/eth0/eth2/" /etc/network/interfaces

Hostonly networking in Vagrant

While Vagrant has support for hostonly networking, it assumes that it has to assign the interface an IP-address. In our case the interfaces eth0,eth1 do not get an IP-address, only the bridge gets an IP-address.

Also using the hostonly networking support built into vagrant, we found that it changed the order of the interfaces: it seems to dynamically remove all host interfaces and recreate them.

We resorted to :

  • adding our hostonly interfaces to our 'basebox' via the VBoxManage commands
  • exporting it via 'vagrant basebox export' subcommand

And thus creating a 'gateway.box' and 'manager.box' vagrant box.

For the gateway node:

IFTYPE="virtio"
VBoxManage modifyvm squeeze64 --nic1 hostonly
VBoxManage modifyvm squeeze64 --nic2 hostonly
VBoxManage modifyvm squeeze64 --nic3 nat
VBoxManage modifyvm squeeze64 --nic4 none
VBoxManage modifyvm squeeze64 --macaddress1 auto
VBoxManage modifyvm squeeze64 --macaddress2 auto
VBoxManage modifyvm squeeze64 --macaddress3 auto
VBoxManage modifyvm squeeze64 --nictype1 $IFTYPE
VBoxManage modifyvm squeeze64 --nictype2 $IFTYPE
VBoxManage modifyvm squeeze64 --nictype3 $IFTYPE
VBoxManage modifyvm squeeze64 --cableconnected2 off
VBoxManage controlvm squeeze64 setlinkstate2 off
VBoxManage modifyvm squeeze64 --hostonlyadapter1 vboxnet0
VBoxManage modifyvm squeeze64 --hostonlyadapter2 vboxnet0

rm squeeze64.box
bundle exec vagrant basebox export squeeze64
mv  squeeze64.box boxes/manager.box
bundle exec vagrant box remove 'manager'
bundle exec vagrant box add 'manager' 'boxes/manager.box

For the manager node:

IFTYPE="virtio"
VBoxManage modifyvm squeeze64 --nic1 hostonly
VBoxManage modifyvm squeeze64 --nic2 hostonly
VBoxManage modifyvm squeeze64 --nic3 nat
VBoxManage modifyvm squeeze64 --nic4 hostonly
VBoxManage modifyvm squeeze64 --macaddress1 auto
VBoxManage modifyvm squeeze64 --macaddress2 auto
VBoxManage modifyvm squeeze64 --macaddress3 auto
VBoxManage modifyvm squeeze64 --macaddress4 auto
VBoxManage modifyvm squeeze64 --nictype1 $IFTYPE
VBoxManage modifyvm squeeze64 --nictype2 $IFTYPE
VBoxManage modifyvm squeeze64 --nictype3 $IFTYPE
VBoxManage modifyvm squeeze64 --nictype4 $IFTYPE
VBoxManage modifyvm squeeze64 --cableconnected2 off
VBoxManage controlvm squeeze64 setlinkstate2 off
VBoxManage modifyvm squeeze64 --hostonlyadapter1 vboxnet0
VBoxManage modifyvm squeeze64 --hostonlyadapter2 vboxnet0
VBoxManage modifyvm squeeze64 --hostonlyadapter4 vboxnet1

mv squeeze64.box manager.box

bundle exec vagrant box remove 'manager'
bundle exec vagrant basebox export squeeze64
mv  squeeze64.box boxes/manager.box
bundle exec vagrant box add 'manager' 'boxes/manager.box'

Adapting Vagrant to use eth2 for NAT

Not specifying the hostonly network in the Vagrantfile, kept the order and the interface names of the hostonly networks intact. But by default, vagrant assumes it has to put the NAT ssh-port mapping on eth0. To make it point to eth2, you can specify the adaptor by explicitly adding the ssh portforwarding mapping to the Vagrantfile.

  config.vm.forward_port("ssh",22,2222,{:auto => true,:adapter => 2})

VLAN support in Virtualbox

I would have expected VLAN-ing to work out of the box with Virtualbox, but it didn't.

For some reason, both gateway and manager were not able to talk to each other. I would see packets come in from both sides on the interfaces, but it just would not work.

After some digging , I found this blogpost on VLAN Stripping in Virtualbox and the research he did .

It turns out that the Intel Pro Network drivers providers, provided by Virtualbox, strip the VLAN tags. The blog suggested changing the interface types to the older AMD PCNet Fast III. This indeed worked.

But on large bursts of network traffic, I was confronted with a kernel warning NETDEV Watchdog: eth0: transmit timed out .

Luckily there is a third option for networking listed on the Virtualbox network documentation: virtio drivers. At first I thought this was not usable because I was running an a Mac, but the virtio are drivers inside the Guest OS, not on the host.

That is why in the above box modifications we set the nictype to virtio

VBoxManage modifyvm squeeze64 --nictype1 virtio

Bridging in Virtualbox

While recreating the vms, many times, I noticed that bridging sometimes would not work. When I looked in the VirtualBox gui, I noticed that sometimes virtualbox indicated, that the active network connection was on eth0 in the gateway and on eth1 in the manager. To fake the behavior and have them both on the same interfaces, I disconnected the cable on the eth1.

VBoxManage modifyvm squeeze64 --cableconnected2 off
VBoxManage controlvm squeeze64 setlinkstate2 off

Now the bridging network interfaces could always see eachother.

PXE booting in Virtualbox

If you want to use pxe booting in virtualbox, be sure to install the Extension pack. This will include the PXE boot functionality.

$ VBoxManage list extpacks
Extension Packs: 1
Pack no. 0:   Oracle VM VirtualBox Extension Pack
Version:      4.1.4
Revision:     74291
Description:  USB 2.0 Host Controller, VirtualBox RDP, PXE ROM with E1000 support.
VRDE Module:  VBoxVRDP
Usable:       true 
Why unusable:

More details can be found in the documentation of Virtualbox

Creating a pxe-booting basebox

For server1 and server2, we want them to pxe boot from the manager node. This means that we want to have an empty disk, but with network boot enabled.

VBoxManage modifyvm pxe-server1 --boot4 net

Veewee is not yet capable of doing this, but here are the commands we used to create an empty netboot basebox.


VMPATH="/"$(VBoxManage list systemproperties|grep "^Default"| cut -d '/' -f 2-)"/pxe/"
DISKPATH="/"$VMPATH/"/pxe.vdi"
VBoxManage unregistervm pxe --delete
rm -rf "$VMPATH"
VBoxManage createvm --name pxe --ostype Debian_64 --register
VBoxManage storagectl 'pxe' --name 'IDE Controller' --add ide
VBoxManage storagectl 'pxe' --name 'SATA Controller' --add sata --hostiocache off --sataportcount 1

VBoxManage createhd --filename "$DISKPATH" --size 20000 --format VDI 
VBoxManage storageattach 'pxe' --storagectl 'SATA Controller' --port 0 --device 0 --type hdd --medium "$DISKPATH"

PXE boot install on server1

Now that we have a pxe boot basebox, we can start modifying it to:

  • fix the macaddress, so we can put them in dhcp
  • create the necessary hostonly interfaces

Because the first pxe boot takes a long time, we increased the ssh timeouts in the Vagrantfile

    config.ssh.timeout = 1000000
    config.ssh.max_tries = 50

Again, the virtio driver is used

IF_TYPE=virtio
VBoxManage unregistervm pxe-server1 --delete
VBoxManage clonevm pxe --name pxe-server1 --register

VBoxManage modifyvm pxe-server1 --memory 256 --ostype Debian_64

VBoxManage modifyvm pxe-server1 --nic1 hostonly
VBoxManage modifyvm pxe-server1 --nic2 hostonly
VBoxManage modifyvm pxe-server1 --nic3 nat
VBoxManage modifyvm pxe-server1 --nic4 none
VBoxManage modifyvm pxe-server1 --macaddress1 080027BA2DAE
VBoxManage modifyvm pxe-server1 --macaddress2 080027BE8E74
VBoxManage modifyvm pxe-server1 --macaddress3 auto
VBoxManage modifyvm pxe-server1 --macaddress4 auto
VBoxManage modifyvm pxe-server1 --nictype1 $IF_TYPE
VBoxManage modifyvm pxe-server1 --nictype2 $IF_TYPE
VBoxManage modifyvm pxe-server1 --nictype3 $IF_TYPE
VBoxManage modifyvm pxe-server1 --nictype4 $IF_TYPE
VBoxManage modifyvm pxe-server1 --cableconnected2 off
VBoxManage controlvm pxe-server1 setlinkstate2 off
VBoxManage modifyvm pxe-server1 --cableconnected3 off
VBoxManage controlvm pxe-server1 setlinkstate3 off
VBoxManage modifyvm pxe-server1 --hostonlyadapter1 vboxnet1
VBoxManage modifyvm pxe-server1 --hostonlyadapter2 vboxnet1
VBoxManage modifyvm pxe-server1 --boot4 net
rm boxes/pxe-server1.box
bundle exec vagrant box remove 'server1'
bundle exec vagrant basebox export pxe-server1
mv 'pxe-server1.box' boxes/
bundle exec vagrant box add 'server1' 'boxes/pxe-server1.box'

Fixing the mac address of a bridge

Even though we can control the mac address from the physical interfaces, the macaddress of the bridge is randomly assigned.

The trick is to specify the macaddress on if-up of the bridge interface

auto br0
iface br0 inet dhcp
  bridge_ports eth0
  bridge_maxage 12
  bridge_hello 2
  bridge_fd 6
  bridge_stp on
  post-up ip link set br0 address 08:00:27:ba:2d:ae

PXE boot for server2

The main difference with server1, is that server2 always boots from memory.

Vagrant basebox export, expects a disk to be there. So we have to assign it one, even if we don't need one.

We also found that the internal pxe support is limited, using of ipxe.org, allowed us to boot with many special pxe options.

This lead us to put the ipxe boot code on the first disk.

VBoxManage convertfromraw ../ipxe/ipxe.usb "$DISKPATH"

And replace the disk with it, making it play nice with vagrant export

The full creation script looks like this

VMPATH="/"$(VBoxManage list systemproperties|grep "^Default"| cut -d '/' -f 2-)"/pxe-server2/"
DISKPATH="$VMPATH/pxe-server2.vdi"
VBoxManage storageattach 'pxe-server2' --storagectl 'SATA Controller'  --port 0 --device 0 --medium none
VBoxManage closemedium disk  "$DISKPATH" --delete
VBoxManage unregistervm pxe-server2 --delete
rm -rf "$VMPATH"
VBoxManage clonevm pxe --name pxe-server2 --register
VBoxManage modifyvm pxe-server2 --memory 1024 --ostype Debian_64
VBoxManage modifyvm pxe-server2 --nic1 hostonly
VBoxManage modifyvm pxe-server2 --nic2 hostonly
VBoxManage modifyvm pxe-server2 --nic3 nat
VBoxManage modifyvm pxe-server2 --nic4 none
VBoxManage modifyvm pxe-server2 --macaddress1 080027C5DAB8
VBoxManage modifyvm pxe-server2 --macaddress2 0800274020F8
VBoxManage modifyvm pxe-server2 --macaddress3 auto
VBoxManage modifyvm pxe-server2 --macaddress4 auto

VBoxManage modifyvm pxe-server2 --nictype1 virtio
VBoxManage modifyvm pxe-server2 --nictype2 virtio
VBoxManage modifyvm pxe-server2 --nictype3 virtio
VBoxManage modifyvm pxe-server2 --nictype4 virtio
VBoxManage modifyvm pxe-server2 --cableconnected2 off
VBoxManage controlvm pxe-server2 setlinkstate2 off
VBoxManage modifyvm pxe-server2 --cableconnected3 off
VBoxManage controlvm pxe-server2 setlinkstate3 off
VBoxManage modifyvm pxe-server2 --hostonlyadapter1 vboxnet1
VBoxManage modifyvm pxe-server2 --hostonlyadapter2 vboxnet1
VBoxManage modifyvm pxe-server2 --boot4 net
rm "$DISKPATH"
VBoxManage storageattach 'pxe-server2' --storagectl 'SATA Controller'  --port 0 --device 0 --medium none
VBoxManage closemedium disk  "$DISKPATH" --delete
VBoxManage convertfromraw ../ipxe/ipxe.usb "$DISKPATH"
VBoxManage storageattach 'pxe-server2' --storagectl 'SATA Controller' --type hdd --port 1 --device 0 --medium "$DISKPATH"

rm boxes/pxe-server2.box
bundle exec vagrant box remove 'server2'
bundle exec vagrant basebox export pxe-server2
mv 'pxe-server2.box' boxes/
bundle exec vagrant box add 'server2' 'boxes/pxe-server2.box'

Direct ssh into vagrant VM via Hostonly IF

Instead of doing the portmapping to server1 or server2 via NAT, we found it useful to put an interface of our host into the hostonly network. This allowed for connection to services in server1,server2 for example.

VBoxManage hostonlyif ipconfig vboxnet1 --ip 172.168.0.1

And then adjust for example the ssh properties of server2, as follows:

    config.ssh.host = "172.168.0.22"
    config.ssh.port = 22

Conclusion:

It takes a bit of massaging to get it working but it's possible:

  • use virtio driver where possible
  • use ipxe to boot complex pxe setups
  • use hostonly networking for direct access to services

October is Devops Month

(2011-09-29) - Comments

So you are looking for a devops place to be in October? Be quick to register to one or both of these conferences

Europe: Devopsdays Goteborg

On Oct 14 - 15 in Goteborg, we're organizing the third! Devopsdays Europe (goteborg)

Here's a list of the talks:

For more information on the schedule , goto the complete program

US: Camp Devops Chicago

On Oct 22 - 23 in Chicago, the first Camp Devops will be organized by @martinjlogan.

Here's a list of the talks:

  • Continuous Delivery Breakdown - Jez Humble
  • Automate Everything: 100% Continuous with Jenkins and Gerrit - Tyler Croy
  • Overcoming Organizational Hurdles - Chris Read
  • Configuration Management: Puppet - Teyo Tyree
  • Monitoring and Alerting - Zenoss
  • Hands on Graphite - Nicholas Leskiw
  • Hal Snyder - Is CMDB Dead?
  • Head in the Clouds - Matt O'Keefe

For more information, goto the complete program

Devops tools fools and other smart things

(2011-09-29) - Comments

In this phrase I try to tackle the phrase "Devops is not about the tools, it's about culture". In this presentation I gave at gotocon I try to explain the importance of tools within devops:

I believe working with tools can change your behavior which eventually leads to a change in culture. The tool itself is no guarantee, lots depends on the person, the context. But culture alone will not cut it. In that way I think the importance of tools is underrated.

Devops Tools Fools and Other smart things
View more presentations from Patrick Debois
Prev Next