Integrating TestCafe with TestRail

Tags, , ,

This article describes how to integrate TestCafe, a popular end-to-end testing tool, with TestRail test case management system by using Agiletestware Pangolin product.


Installing TestCafe and xUnit reporter

To be able to send TestCafe result files into TestRail, we have to install TestCafe itself and also TestCafe xUnit reporter. This reporter can generate test reports in XML format which Pangolin server can understand and upload into TestRail.

TestCafe installation:

npm install -g testcafe

xUnit reporter installation:

npm install -g testcafe-reporter-xunit

Creating TestCafe test

For this article we will copy a simple dummy test from TestCafe documentation and enhance it a bit to illustrate pass and fail cases:

import { Selector } from 'testcafe'; // first import testcafe selectors

fixture `Getting Started`// declare the fixture
    .page ``;  // specify the start page

//then create a test and place your code there
test('Passing test', async t => {
    await t
        .typeText('#developer-name', 'John Smith')

        // Use the assertion to check if the actual header text is equal to the expected one
        .expect(Selector('#article-header').innerText).eql('Thank you, John Smith!');

test('Failing test', async t => {
    await t
        .typeText('#developer-name', 'Jane Doe')

        // Use the assertion to check if the actual header text is equal to the expected one
        .expect(Selector('#article-header').innerText).eql('Thank you, John Smith!');

Pangolin Server setup

In order to start using Pangolin features, a Pangolin Server application should be installed somewhere in your network. To do so, please follow the steps below:

Once the Pangolin Server is in place, you can start setting up your CI system for sending test results into TestRail. Currently, Pangolin has plugins for the following CI systems:

In this article, we will show how to set up integration in Jenkins, but it’s more or less the same for all other CI Systems. See Pangolin documentation for more details.

Jenkins setup

To start using Pangolin plugin for Jenkins, it needs to be installed from the official Jenkins plugin repository. To do that, please follow the steps below:

  • Open Jenkins Web GUI in your favorite browser and navigate to Manage Jenkins -> Manage Plugins
  • Click on Available tab, search for “Agiletestware Pangolin Connector for TestRail” plugin, tick the checkbox and click “Download now and install after restart
  • Restart Jenkins

Plugin global configuration

Once the plugin has been installed, a new configuration section is available in Global configuration (Manage Jenkins -> Configure System):

Pangolin plugin’s global configuration

The following fields must be filled:

  • Pangolin URL – a URL of your Pangolin server
  • TestRail URL – a URL of your TestRail server
  • TestRail User – default user to be used across all related Jenkins jobs
  • TestRail Password – password for the TestRail user
  • Upload Timeout – upload timeout or zero for infinite timeout

Once all required fields are filled, click on “Save Configuration” button to persist configuration values.

Creating a Jenkins agent (Windows)

It is always a good idea to run your jobs not on a Jenkins Master machine, but on some dedicated execution agent. It is also important to run your TestCafe job on a separate agent if your Jenkins server runs as a Windows Service under Local System account as it does not have interactive session.
To setup Jenkins agent, navigate to Manage Jenkins -> Manage Nodes -> New Node, check “Permanent Agent” option, type a node name and click OK.

In new window, set Remote Root directory, make sure that Launch method is “Launch agent via Java Web Start” and click Save

Now you can see newly created agent in the list of agents:

Click on your agent’s name and then on Launch button, save .jnlp file and run it:

Once the agent is launched, its status should change to “Idle”:

Now your agent is ready to run your jobs

Creating a new Jenkins job

Once the global configuration is done, we can start configuring a new Jenkins job. You can do it in two ways: using Freestyle project and configuring it with UI or having Pipeline script. Let’s try both ways:

Jenkins Freestyle project

First of all we need to create a new Freestyle Project in Jenkins and mark that this project has to run on the articular Jenkins agent:

Then add our Git repository containing TestCafe script file into Source Code Management section:

Then, we need to add shell/batch script to run our TestCafe script and put results into result.xml file:

The next step is to add “Pangolin: Upload test results into TestRail” post-build action which sends the results into TestRail:

There are several configuration parameters to set:

  • Project – name of a project in TestRail
  • Test Path – path to a section in TestRail to add test definitions into
  • Format – test results format type. Pangolin supports many formats, but for this article we will use “JUnit”
  • Test Run – name of a Test Run in TestRail to upload results into
  • Test Plan – name of a Test Plan in TestRail to upload results into
  • Milestone Path – specify that if you want to attach your results to some milestone
  • Case Fields – this field can be used to provide values for custom fields of your tests in TestRail.

Pipeline script

Here is a Jenkins Pipeline script which does the same as Freestyle job above:

pipeline {
    agent {label 'my-agent'}
    stages {
                git branch: 'TestCafe', url: ''
        stage('Build') { 
            steps {
                bat 'testcafe chrome test.js -r xunit:result.xml'
    post {
        always {
                    testRailProject: 'Pangolin Demo',
                    configs: [[
                        testPath: 'Master\\TestCafe',
                        format: 'JUnit',
                        resultPattern: 'result.xml',
                        testPlan: 'TestCafe plan',
                        testRun: 'TestCafe run',
                        milestonePath: 'Milestone 1\\Milestone 2',
                        customProperties: '''custom_required_field=Hello\ncustom_optional_field=World'''

Execution and results in TestRail

Once the job is configured and run, it runs test.js file and export results into TestRail:

So, as you can see, Pangolin allows you to integrate your TestCafe tests with TestRail easily without any custom coding.


Software Developer with 10+ years of experience in Java and related technologies