User Acceptance Testing and Automation as Atomic App

The germ of the technical problem

Is there a problem? Looking at some recent internet activity about automation testing was confirmed that there is widespread instances of test environment stability problem. (Definition of Done for Regression Test Automation Suite – Quick,Reliable and Credible by Alex Lavrenov, How Many Test Failures Are Acceptable? by Dave Farley). My philosophy is that before we start writing scripts to re-run failed test, we have to think a little bit whether we have done everything to provide a favorable environment for our test suite.

There is a lot of tools for acceptance testing and continuous integration. All of them are using browser specific API to drive the testing process.  Here is the list of drivers for most popular browsers if you want to read about:
* Yeah, putting this list here make no sense but it will help Google to deliver this guideline to a larger audience.

Desktop:

Mobile:

We trust that browser vendor provided drivers are stable software but it is not.

They are very sensitive to the operating system and windowing system. So when we say that we gonna use Selenium it’s not enough. Even more if we switch to parallel testing without some special settings it becomes a nightmare.

The germ of the spiritual problem

Very often this part of development process can make a developer unhappy. After spending a lot of time writing test scenarios for your application and waiting for satisfaction of well done job – Oh, that never happens. Looking at the failing tests every morning is really depressingly.

Another trend is that when you work in a team, not all developers care about the tests. But the team should not force a spike developer to fix tests when he got inspiration. Often this task remains for a special breed of programmers who I greatly respect.

How to be a happy developer when your task is User Acceptance Testing?

Respect

Build it so that no one can say “Tests are broken”. The test suite must instill respect. If there is failures it should mean that there is a real issue or the interface is not passing the requirements.

Guideline

First you need a stable, simple and easy to maintain test environment. Do not falter in the development of tools on top of test environment. Prepare and test the test environment. Ask the company for a virtual server specifically provided for those purposes or build your own. Install the low level tools and run a simple test 100 times or all the night. If there is no 100% success, something is wrong. “Unable to obtain stable firefox connection in 60 seconds” or “Unable to connect to host 127.0.0.1 on port 7055 after 45000 ms” are typical error messages for unstable environment.

Once you find the right configuration, fix the versions of all modules. Remember all your steps by creating a scripts that can set up the test environment from scratch.

Solution

A nice solution that I want to share is using a Vagrant box. Vagrant is cross OS tool that creates and configures virtual development environments. Vargant Provisioning can help you to document and preserve the Test Environment setup process. With Vagrant Synced Folders you can have your project outside the test environment, work with your favorite editor and finally – get the test execution results back to your host machine.

I’m a fan of Fedora Linux and I was pleasantly surprised when I found these two articles:
Using Fedora 22 Atomic Vagrant Boxes
Running Vagrant on Fedora 22

I was not familiar what “Atomic” means and reading more about Atomic App and Nulecule Specification (describing a container-based application) encouraged me that this is the right direction.

Technical Part

I’m a Rubyist, so I will give an example configuration with Ruby project, but you can adapt it for other project natures too.
* Part of my plan to be a happy developer was to program in Ruby. (This is not a joke. When a Java or PHP developer becomes a Ruby and Ruby on Rails developer – he becomes a happy developer)

Create an empty project and Gemfile including Rspec, Capybara and Selenium WebDriver.

Gemfile

source 'https://rubygems.org'

gem 'rspec'
gem 'capybara'
gem 'selenium-webdriver'

Run from your project directory:

bundle install
bundle exec rspec –init

The last command will generate .rspec and spec/rspec_helper.rb files.
Now create a separate file spec/test_helper.rb for the initialization code and add require ‘test_helper’ on top of existing rspec_helper.rb:

spec/test_helper.rb

require 'capybara/rspec'
require 'selenium-webdriver'

Capybara.default_driver = :selenium

Lets create the a test feature:

spec/features/uat_spec.rb

describe 'UAT', type: :feature do

  it "Find Definition from Wikipedia" do
    visit 'https://en.wikipedia.org'
    
    within '#simpleSearch' do
      fill_in 'search', with: 'UAT'
      click_on 'Go'
    end
    
    within '#bodyContent' do
      expect(page).to have_selector 'a', text: 'User acceptance testing'
      click_on 'User acceptance testing'
    end
    
    expect(page).to have_content 'verifying that a solution works for the user'
  end
end

Now we have a ready test environment and we can run the test. (I told you Ruby is fun). Run it and you will see Firefox opening Wikipedia and executing the scenario.

bundle exec rspec
.

Finished in 7.25 seconds (files took 0.2968 seconds to load)
1 example, 0 failures

FirefoxDriver is default, but you can easily switch to Chrome having ChromeDriver installed, by changing the scpec/test_helper.rb:

Capybara.register_driver :chrome do |app|
  Capybara::Selenium::Driver.new(app, :browser => :chrome)
end

Capybara.default_driver = :chrome

We can run the test several times with little Bash script help and I’m sure all will pass

for i in 1 2 3 4 5; do bundle exec rspec; done;

But let’s go parallel!
Clone the test 4 times (change decribe title too) and add parallel_tests gem to the Gemfile (don’t forget bundle install):

Gemfile

gem 'parallel_tests'

Now if you run ‘for i in 1 2 3 4 5; do bundle exec parallel_rspec spec/features; done;‘ it will run all the tests simultaneously 5 times (if you have 4 CPUs) and you’ll be lucky if there is no failures.

Atomic App

Install Vagrant and prepare the f22atomic box image as described in http://fedoramagazine.org/using-fedora-22-atomic-vagrant-boxes or use a box of your choice.

Add Vagrant file to the project by executing:

vagrant init f22atomic
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! …

I decided to use libvirt Vagrant Provider, but the ‘virtualbox’ provider shows a great result too.

Vagrantfile

Vagrant.configure(2) do |config|
  config.vm.box = "f22atomic"
  config.vm.provider "libvirt" do |libvirt|
    libvirt.memory = 2048
    libvirt.cpus = 4
  end
end

Lets create vagrant provisioning scripts. We will provide headless browser testing with Xvfb and in my case – rvm and ruby.

Vagrantfile

config.vm.provision :shell, :path => "vagrant-install-xvfb.sh"
config.vm.provision :shell, :path => "vagrant-install-firefox.sh"
config.vm.provision :shell, :path => "vagrant-install-rvm.sh",  :args => "stable"
config.vm.provision :shell, :path => "vagrant-install-ruby.sh", :args => "2.2"

vagrant-install-xvfb.sh

#!/usr/bin/env bash

dnf install -y Xvfb

vagrant-install-firefox.sh

#!/usr/bin/env bash

dnf install -y firefox liberation-sans-fonts

vagrant-install-rvm.sh

#!/usr/bin/env bash

dnf install -y which #fix missing bash command for f22atomic
gpg2 --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
curl -sSL https://get.rvm.io | bash -s $1

vagrant-install-ruby.sh

#!/usr/bin/env bash

source /usr/local/rvm/scripts/rvm

rvm use --install $1

shift

if (( $# ))
  then gem install $@
fi

usermod -a -G rvm vagrant
rvm --default use $1
gem install bundler

Vagrant commands:

vagrant up #start Vagrant (with provisioning if for the first time)
vagrant ssh #establish SSH session to running machine
vagrant halt #stop it
vagrant provision #re-run provision scripts
vagrant destroy #stop and delete all resources

Now add the headless gem to Gemfile and modify spec/test_helper.rb to use Xvfb display.

Gemfile

gem 'headless'

spec/test_helper.rb

require 'headless'

ENV['UAT_TEST_NUMBER'] ||= "#{ (ENV['TEST_ENV_NUMBER']!='' ? ENV['TEST_ENV_NUMBER'] : 1).to_i }"

headless = Headless.new(
  display: "#{ 100 + ENV['UAT_TEST_NUMBER'].to_i }",
  reuse: true,
  dimensions: "1280x900x24"
) if ENV['HEADLESS']
RSpec.configure do |c|
  c.before(:suite) do
    if (ENV['HEADLESS'])
      p 'Starting Headless...'
      headless.start
    end
  end
  
  c.after(:suite) do
    if (ENV['HEADLESS'])
      headless.destroy
    end
  end
end

The headless configuration is triggered only if HEADLESS environment variable passed, so we can run the tests in usual way too.
Note the code display: “#{ 100 + ENV[‘UAT_TEST_NUMBER’].to_i }”. With Parallel Tests gem, we have Environment variable TEST_ENV_NUMBER, which is used to start separate Xvfb display for each set of tests.
* If you going to use Jenkins to run tests, include Jenkins provided variable BUILD_NUMBER in the calculation too.

Inside the box, we have the project mounted in /vagrant folder by default (thanks to Vagrant Synced Folders) Now we can run the test suite like:

vagrant ssh
cd /vagrant
bundle install
HEADLESS=1 bundle exec parallel_rspec spec/features

And if added –format ParallelTests::RSpec::SummaryLogger –out tmp/spec_summary.log to the parallel_rspec command it will provide the test results out of the box in the host’s  <project>tmp folder.

Vagrant is a friend with Docker, so you can create a Dockerfile in your project and add more color to the test environment by providing aliases for the most common commands, but I do not intend to go deep into.

Now the project is portable. You do not even need to have development tools installed to run the tests.

Is it stable?

It depends on your configuration. But this is a big step towards stability.

PS:

Oh, and there is an extra that other test environments can’t do so easy – video recording. Here is a quick example:

spec/test_helper.rb

headless = Headless.new(
  display: "#{ 100 + ENV['UAT_TEST_NUMBER'].to_i }",
  reuse: true,
  dimensions: "1280x900x24",
  video: {
    provider:   :ffmpeg,
    frame_rate: 12,
    codec:      :libx264,
    pid_file_name:  "/tmp/.headless_ffmpeg_#{ENV['UAT_TEST_NUMBER']}.pid",
    tmp_file_name:  "/tmp/.headless_ffmpeg_#{ENV['UAT_TEST_NUMBER']}.pid"
  }
) if ENV['HEADLESS']

RSpec.configure do |c|
  c.before(:each) do
    page.driver.browser.manage.window.resize_to(1280,900)
    headless.video.start_capture if ENV['HEADLESS']
  end
  
  c.after(:each) do |e|
    headless.video.stop_and_save "video/video_#{ENV['UAT_TEST_NUMBER']}_#{File.basename(e.metadata[:file_path])}.mp4" if ENV['HEADLESS']
  end
end

10x for reading!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s