openQA – or how we test EuroLinux

openQA – how we test EuroLinux

One of the elements of the EuroLinux test suite is the openQA software. It’s an automated test tool that allows an extensive analysis of the whole operating system installation process, as well as visual tests for individual applications. In this article we’ll show, how we utilize it.

openQA utilizes virtual and physical machines for replaying an installation process, checking the output (of both serial console and a screen) and sending the required keystrokes, commands and mouse movements. openQA tests are written in the Perl programming language, the knowledge of which is required for handling them. The openQA suite is build upon the Mojolicious Perl framework, which is used for building interactive web pages.

Interfejs openQA
openQA interface

openQA can conduct several test combinations for each version of an operating system, reporting the detected errors for each combination of hardware configuration, installation options and an operating system variant. It utilizes the capabilities of KVM/QEMU as well as nested virtualization in the case of EuroLinux. openQA is free software licensed under GPLv2.

A standalone application named os-autoinst is the heart of the test engine. The application creates a virtual machine and uses it to run a set of testing scripts. It generates videos, screenshots and JSON files with in-depth results of the tests.

A sample video of EuroLinux 8 Minimal installation process generated by openQA engine.

openQA provides a web-based user interface and the infrastructure for running a distributed os-autoinst. The web-based interface also provides a REST API that allows the utilization of external scripts and is used by so-called workers. They gather data and input files from openQA for os-autoinst to launch the tests. A single host system can run several such jobs. The web application and workers can both be running on the same machine as well as be networked on multiple machines.

Machines

To run the tests, we need to have at least one machine configured as part of YAML definition files. The files represent types of the virtual machines we want to test. For the tests to run, at least one openQA worker which satisfies the specification below must be online:

  • name – a user-defined string – an operator needs it to identify a machine’s configuration
  • backend – qemu is the recommended value, since it’s the most widely used engine, however other options (such as kvm2usb or vbox) are available as well
  • variables – most of machine’s variables affects the behavior of os-autoinst on how the test machine is configured. Some examples:
    • QEMUCPU ('qemu32' lub 'qemu64') – specifies the architecture of the virtual CPU
    • QEMUCPUS – an integer specifying the number of cores we want to include in the tests
    • LAPTOP – if set to 1, QEMU will create the profile: laptop
    • USBBOOT – if set to 1, an image will be loaded through an emulated USB drive.

Below are some sample definitions

of a machine:

   "Machines": {
        "64bit": {
            "backend": "qemu",
            "settings": {
                "ARCH_BASE_MACHINE": "64bit",
                "PART_TABLE_TYPE": "mbr",
                "QEMUCPU": "Nehalem",
                "QEMUCPUS": "2",
                "QEMURAM": "2048",
                "QEMUVGA": "virtio",
                "QEMU_VIRTIO_RNG": "1",
                "WORKER_CLASS": "qemu_x86_64"
            }
        }

of a distribution:

   "Products": {
        "EuroLinux-8-x86_64-*": {
            "arch": "x86_64",
            "distri": "eurolinux",
            "flavor": "DVD",
            "settings": {
                "+QEMURAM": 3072,
                "TEST_TARGET": "ISO"
            },
            "version": "*"
        }
    }

of a profile:

    "Profiles": {
        "EuroLinux-8-x86_64-*-64bit": {
            "machine": "64bit",
            "product": "EuroLinux-8-x86_64-*"
        }
    }

Needles

One of the main mechanisms that lets openQA know a virtual machine’s state is checking the presence of several elements on a “screen” of that machine. This is performed as a fuzzy match of that “screen” and so-called “needles”. A needle specifies the elements to search for as well as a list of tags to decide, which needles shall be used at a given moment.

A needle consists of a complete screenshot in PNG format as well as a JSON file with the same filename (eg. el12.png and el12.json). It contains the associated data such as area inside the complete screenshot that are vital for the tester process.

Przykładowa igła w interfejsie openQA
A sample needle in openQA’s interface

The appropriate areas are visually marked directly at OpenQA – marking an area results in the creation of a relevant JSON file with the coordinates of the areas. In the example above a needle’s JSON file has the following form:

{
  "area": [
    {
      "xpos": 7,
      "ypos": 86,
      "width": 536,
      "height": 33,
      "type": "match"
    },
    {
      "xpos": 31,
      "ypos": 96,
      "width": 10,
      "height": 10,
      "type": "match"
    },
    {
      "xpos": 23,
      "ypos": 36,
      "width": 74,
      "height": 29,
      "type": "match",
      "click_point": {
        "xpos": 37,
        "ypos": 14.5
      }
    }
  ],
  "properties": [],
  "tags": [
    "timezone-next"
  ]
}

Areas

Three types of areas exist:

  • regular areas – they define the vital parts of a screenshot. They must match with at least a specified similarity percentage. They are displayed as green fields in the needle editor and as either green or red frames in the needle view (green for matching areas, red for non-matching)
  • OCR areas – they also define the vital parts of a screenshot, but an OCR algorithm is used for matching. OCR areas are displayed as orange areas in the needle editor
  • exclude areas – they can be used to ignore parts of the reference picture. They are displayed as red frames in the needle editor. In the needle view exclude areas are displayed as gray fields.
needleedit
Area editing is carried out visually directly in the web interface

Access management

Several operations in openQA require special privileges. By default an openQA instance is configured to use openID, but can be easily reconfigured to use another provider. A user profile contains an openID identity and two access control flags:

  • operator – meaning that a user is able to manage jobs, performing activities such as creating new jobs, canceling them, etc.
  • admin – meaning that a user is able to manage users (granting or revoking operator and administrator rights) as well as job templates and other related information.

Many openQA operations are not performed via the web interface but with a REST API. The most common example of using this API are workers and scripts that download new operating system releases and plan ahead appropriate tests.

Testing an ISO image

Te begin testing a new ISO, it must be placed in the /var/lib/openqa/share/factory/iso directory and the following sample command must be invoked:

openqa-cli api -X POST isos \
	ISO=EL-8.5-x86_64-20211201-appstream.iso \
	DISTRI=eurolinux8 \
	VERSION=5 \
	FLAVOR=appstream \
	ARCH=x86_64 \
	BUILD='EL8.5_20211201'

Job groups

Job groups are the area where the actual test scenarios are defined by selecting a type, the test suite and the machine with a priority value. If multiple jobs are schedules and their running requirements are met, those with a lower priority are run. Job groups can be created via the web interface as well as via REST API. They can be optionally nested into categories. The display order of job groups and categories can be configured with a “drag and drop” method in the web interface.

groups
Job groups list

Tests based on a Support Server

Having a dedicated “Support Server” is required for advanced network-based test procedures. A Support Server utilizes the basic parallel setup where the Support Server is the parent test “A” and the test that needs it is the child test “B”.

Using text consoles and a serial terminal

Usually an operating system boots to a graphical shell. In the case of OpenQA it is useful if we want to test a system with a GUI but in many situations we need to input commands directly to a shell, a TTY, a text terminal, a commandline, etc. openQA has two basic methods for interacting with a text shell. The first one utilizes the same input and output methods as with the case of interacting with a GUI, plus a serial port for gathering a raw text output. The second method uses another serial port for input as well as for output. Therefore, the whole communication is text-based.

Summary

In the article we present just a fragment of openQA’s capabilities. This tool is well-developed and offers a range of other test solutions, which we systematically implement into our chain of tests. Our proprietary Jenkins integration expands openQA’s capabilities even further and makes testing EuroLinux a fully automated and consistent process.