[Digital humanities] Wittgenstein Scholarship internals: testing and continuous integration

At the Center for Information and Language Processing (CIS) I'm currently working on the Digitised Manuscripts to Europeana EU-awarded Wittgenstein finder application.

Therefore we decided to build up a proper testing and integration infrastructure to test the whole web application and the search engine backend. As we used the great web-based git repository mangement software called GitLab I built a continuous integration setup so that every commit to our repository is automatically built and tested in a kind of chroot jail. For that setup we are using the GitLab Continuous Integration package and many different GNU/Linux based distributions for testing.

Virtualization setup

For the whole continuous integration setup we use three virtual machines. The (hardware) virtualization is done with QEMU with kernel-based virtual machine support. Furthermore we use the latest 13.2 series of openSUSE on the hostsystem. Inside of all virtual machines an GNU/Linux Arch system is running. Despite of all negative comments about the "new" (I've been using it over 2 years on servers and desktop now) init system called systemd, I personally say it's working and all reported bugs, e.g. in systemd-networkd for using static ipv6 addresses in systemd version smaller than 213, have been fixed very soon and in a very friendly manner. At the moment QEMU in version 2.2.90 is running. As we detect a hard disk problem with the 2.1/2.2 series of QEMU and the latest 3.19 kernel (inside the virtual machine) the virtio support is enabled. So a typical QEMU start command for running a virtual machine looks like:

  qemu-system-x86_64 \
    -daemonize \
    -enable-kvm \
    -drive file=vm1.img,cache=none,if=virtio,format=raw \
    -m 15G \
    -vnc :1 \
    -k de \
    -net nic,macaddr=52:54:00:00:00:51 \
    -net tap,ifname=tap1,script=no,downscript=no \
    -monitor telnet:localhost:7001,server,nowait,nodelay \
    -smp cores=2,threads=2
So we are using a kind of "tapped" networking with a (telnet) based monitor console.

Notice: We do not use virtualization for the sake of security (never do that!). We have limited (and shared!) hardware/ servers available so this kind of virtualized setup gives the possibility to run more and individual configured machines on one single server. So hardware donations are very welcome ;)


Inside of one virtual machine some container-virtualization - we use the popular docker - is done. At the moment 13 containers are running and listening for new builds. Here's a short overview of the different GNU/Linux operating systems we use for testing our software on:
# runner id Distribution GCC Boost CMake clang
1 Arch 4.9.2 1.57.0 3.2.1 3.6.0
2 Debian 7 4.7.2 1.49.0 2.8.9 3.0
3 Ubuntu 14.04 4.8.2 1.54.0 2.8.12 3.4
4 Ubuntu 14.10 4.9.1 1.55.0 3.5.0
5 Ubuntu 15.04 4.9.2 1.55.0 3.0.2 3.6.0
6 openSUSE 13.1 4.8.1 1.53.0 3.3
7 openSUSE 13.2 4.8.3 1.54.0 3.0.2 3.5.0
8 Fedora 20 4.8.3 1.54.0 3.4.2
9 Fedora 21 4.9.2 1.55.0 3.0.2 3.5.0
10 CentOS 7 4.8.2 1.53.0 2.8.11 3.4.2
11 Debian Testing 4.9.2 1.55.0 3.0.2 3.5.0
12 Debian Testing with latest GCC 5.0.0 20150320 1.55.0 3.0.2 3.5.0
13 Arch ARM 4.9.2 1.57.0 3.2.1 3.6.0

The 13th machine is not a virtual or container-virtualized machine: It is a new Raspberry PI 2. That means our software/search engine backend is also tested on ARM.

For all 12 container-virtualized machine I wrote so-called Dockerfiles in order to get the GitLab CI runner built and running. Here's a small example of how our Dockerfiles look like:

  from archlinux
  maintainer Stefan Schweter <stefan@schweter.it>

  run pacman --noconfirm -Syu
  run pacman-db-upgrade

  run echo "de_DE.UTF-8 UTF-8" > /etc/locale.gen
  run locale-gen
  env LANG de_DE.UTF-8

  workdir /root

  run curl --silent -L https://gitlab.com/gitlab-org/gitlab-ci-runner/repository/archive.tar.gz | tar xz

  # Install dependencies for building the ci-runner
  run pacman --noconfirm -Sy ruby

  # Default location for gems is per-user, not system-wide -> change this first
  # More information: https://wiki.archlinux.org/index.php/ruby#Bundler
  run echo "gem: --no-user-install" > /etc/gemrc

  # Now bundler can be installed (system-wide)
  run gem install bundler

  # Dependencies for single gems
  # charlock_holmes
  run pacman --noconfirm -Sy make icu gcc patch

  workdir /root/gitlab-ci-runner.git

  # Some stupid ruby 2.2 workarounds for compiling the json gem
  run sed -i "s/\('json'\), '~> 1.7.7'/\1, '~> 1.8.2'/" Gemfile
  run sed -i "s/\(.*json\) (1.7.7)/\1 (1.8.2)/" Gemfile.lock
  run sed -i "s/\(.*json\) (~> 1.7.7)/\1 (~> 1.8.2)/" Gemfile.lock

  # Now we can build the ci-runner
  run bundle install --deployment

  run mkdir -p /root/.ssh

  cmd test -z $RUNNER_TOKEN && bundle exec ./bin/setup_and_run ||\
      echo "---" > config.yml &&\
      echo "url: $CI_SERVER_URL" >> config.yml &&\
      echo "token: $RUNNER_TOKEN" >> config.yml &&\
      bundle exec ./bin/runner

  # Include git to fetch/clone the repositories
  run pacman --noconfirm -Sy git

  # Project-related dependencies will be installed here
  run pacman --noconfirm -Sy clang cmake boost recode


The finder application backend - internally it is called wf - consists of a few unit tests. In combination with the cmake and the Boost Test Library all test cases are automatically executed after building the search backend in one of the virtual containers. That ensures that the search backend builds on different Linux distributions and takes care about distribution specific bevahiour, e.g. Toolchain changes.

Here's an example of a test case for the search engine backend:

  #define BOOST_TEST_MODULE Query_test

  #include <boost/test/unit_test.hpp>

  #include "automaton/SimpleQuery.hpp"
  #include "automaton/UserQuery.hpp"
  #include "automaton/SkopeQuery.hpp"

  using wf::Query;
  using wf::SimpleQuery;
  using wf::UserQuery;
  using wf::SkopeQuery;
  using WordVec = wf::Query::WordVec;

  static void
  testQuery(Query& query, const WordVec& gold)
          auto res = query.token();
          BOOST_CHECK_EQUAL(res.size(), gold.size());
          for (auto i = 0U; i < std::min(res.size(), gold.size()); ++i)
                  BOOST_CHECK_EQUAL(res[i], gold[i]);

          SimpleQuery query;
          query.setQuery("((A B) *) | C");
          testQuery(query, {"(", "(", "A", "B", ")", "*", ")", "|", "C"});

End-to-end testing

Another important test case is to make sure that the application frontend works like expected. For that purpose we use a full web stack called phantomjs in combination with CasperJS. Here's a small example (with coffee script) to do a kind of frontend testing:

  testhost = casper.cli.get "testhost"
  timeout = casper.cli.get "timeout"

  casper.options.waitTimeout = timeout;
  casper.test.begin "Wittfind search works as expected", 12, suite = (test) ->
    casper.start "http://#{testhost}/", ->
      @test.assertTitle "WiTTFind — CIS∕WAB 2015", "homepage title is the one expected"
      @test.assertTextExists "Regelbasiertes Finden", "'Regelbasiertes Finden' exists"
      @test.assertExists ".form-control", "main input field is found"

      # Leave it and you'll spend hours to find out that it must be loaded before!
      @waitForResource "include/main.js"
      @waitForResource "include/search.js"

      @fill ".form-group", pattern: "denken", false

    casper.then ->

      # Necessary, as we allow prevent the default form submit behaviour
      # And: It is also necessary to use a casper.then before
      # Leave it, and you get to the strange
      # behavior. This *costs* hours to solve...
      @click ".btn-default"

    casper.then ->
      @waitForSelector ".number_all_hits"

    casper.then ->

      @test.assertHttpStatus 200, "search response code good"
      @test.assertTextExists "Es wurden 207 Treffer gefunden", "Retrieval text for 「denken」 appears."
      @test.assertTextExists "Wenn Sie noch mehr Treffer sehen wollen", "Get CIS account message after search appears."


Testing (and automatically executing tests) is very important so we extended the test scenario in order to test on many different GNU/Linux distributions. Moreover we wrote unit and end-to-end tests to cover test cases for the search engine backend and also for testing our JavaScript code on the frontend. An overview of the used technologies and software was given. Feel free to give comments or ask questions: stefan at schweter dot eu.


I want to thank Dr. Maximilian Hadersbeck and Alois Pichler for giving me the opportunity to work on this awesome and interesting project! Thanks to Daniel for giving me sublime help on things like docker, Makefiles, web technologies (the list goes on and on) and to my colleagues also working hard on the Wittgenstein project: Florian, Matthias, Angela, Roman and Yuliya (and many more!). Thanks to our sysadmin (and ipv6 expert) Thomas for providing us the necessary hardware and support for software installation, network configuration and his expertise on administration and (not to forget) ipv6 networking!