Integrating JUnit/TestNG/NUnit frameworks with TestRail

Tags, , , , , ,

Many companies use various unit testing frameworks like JUnit, TestNG, NUnit, etc for testing their applications on different layers together with manual testing. The problem that might appear is that there is no single source of information about testing results: results of automated testing resides in a corporate CI system, while manual tests are performed in TCM systems, like TestRail.

To address this issue, we have developed Pangolin – a set of tools for integrating various test frameworks with TestRail and in this article we going to show how easy this issue can be solved.

Prerequisites

  • JDK 8
  • Maven 3
  • TestRail server
  • Jenkins

Creating Maven project with JUnit tests

In order to send some test results, we need to produce them somehow, so let’s create a simple Maven project with a few test methods.
First of all, create a new Maven project with the following pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.agiletestware.demo</groupId>
	<artifactId>pangolin-demo</artifactId>
	<version>1.0-SNAPSHOT</version>
	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<java.version>1.8</java.version>
		<junit.version>4.12</junit.version>
	</properties>

	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>3.8.1</version>
				<configuration>
					<source>${java.version}</source>
					<target>${java.version}</target>
				</configuration>
			</plugin>
		</plugins>
	</build>
	<dependencies>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>${junit.version}</version>
			<scope>test</scope>
		</dependency>
	</dependencies>
</project>

and add a simple test class:

package com.agiletestware.pangolin.dummytest;

import org.junit.Assert;
import org.junit.Ignore;
import org.junit.Test;

public class LoginFormTest {

	@Test
	public void loginCorrectCredentials() {
		System.out.println("login with correct credentials");
	}

	@Test
	public void loginErrorIncorrectUserName() {
		Assert.fail("Error message was not shown");
	}

	@Test
	public void loginErrorIncorrectPassword() {
		throw new RuntimeException("Unexpected exception");
	}

	@Test
	public void loginRememberMeIsChecked() {
		System.out.println("login with remember me set to true");
	}

	@Ignore
	@Test
	public void loginFailedServiceUnavailable() {
		System.out.println("login failed");
	}

}

You can find a complete example code in the BitBucket repository:
https://bitbucket.org/sergey_oplavin/pangolin-demo

That’s it, now verify that your Maven project is fine by running “mvn clean test” in your console/terminal. You should see JUnit test results file in target/surefire-reports folder.

Pangolin Server setup

In order to start using Pangolin features, a Pangolin Server application should be installed somewhere in your network. To do so, please follow the steps below:

Once the Pangolin Server is in place, you can start setting up your CI system for sending test results into TestRail. Currently, Pangolin has plugins for the following CI systems:

In this article, we will show how to set up integration in Jenkins, but it’s more or less the same for all other CI Systems. See Pangolin documentation for more details.

Jenkins plugin setup

To start using Pangolin plugin for Jenkins, it needs to be installed from the official Jenkins plugin repository. To do that, please follow the steps below:

  • Open Jenkins Web GUI in your favorite browser and navigate to Manage Jenkins -> Manage Plugins
  • Click on Available tab, search for “Agiletestware Pangolin Connector for TestRail” plugin, tick the checkbox and click “Download now and install after restart
  • Restart Jenkins

Plugin global configuration

Once the plugin has been installed, a new configuration section is available in Global configuration (Manage Jenkins -> Configure System):

Pangolin plugin’s global configuration

The following fields must be filled:

  • Pangolin URL – a URL of your Pangolin server
  • TestRail URL – a URL of your TestRail server
  • TestRail User – default user to be used across all related Jenkins jobs
  • TestRail Password – password for the TestRail user
  • Upload Timeout – upload timeout or zero for infinite timeout

Once all required fields are filled, click on “Save Configuration” button to persist configuration values.

Jenkins job configuration

Now, everything is ready to create a new Jenkins job which does three simple operations: checks out a Maven project from remote GIT repository, runs Maven test phase and sends test results into TestRail.

First, we have to add a new Freestyle project and specify a GIT repository containing our Maven project, e.g. https://bitbucket.org/sergey_oplavin/pangolin-demo

Then, add an “Invoke top-level Maven targets” build action and type “clean test”:

This will invoke Maven test phase and execute your unit tests.

And finally, add a new “Pangolin: Upload test results into TestRail” post-build action which does all the magic and sends the results into TestRail:

There are several configuration parameters to set:

  • Project – name of TestRail project
  • Test Path – path to a section in TestRail to add test definitions into
  • Format – test results format type. Pangolin supports many formats, but for this article we will use “JUnit”
  • Test Run – name of a Test Run in TestRail to upload results into
  • Test Plan – name of a Test Plan in TestRail to upload results into
  • Milestone Path – specify that if you want to attach your results to some milestone
  • Case Fields – this field can be used to provide values for custom fields of your tests in TestRail.

Once the job is configured and run, it will upload all test results into TestRail:


Conclusion

As it can be seen, this type of integration is very easy and requires no coding or difficult configuration. Following this approach of using TestRail as a single source of truth for software quality metrics, we can publish all unit and functional testing results in TestRail and allow engineering management to make informed decisions.

Author

Software Developer with 10+ years of experience in Java and related technologies