TestNG and ALM Integration

This setup requires a little bit of code adjustment. You should only do this is if there is a need to map java tests to alm tests. If you need simple upload of test results, use the Jenkins plugin.

JUnit along with TestNG are the most widely used test frameworks for writing unit and functional tests using a variety of testing tools like Selenium, Playwright, REST Assured, etc.

Bumblebee provides TestNG annotations pacakge that allows users to tag their Junit tests with ALM specific metadata and map JAVA tests to ALM tests in a variety of flexible workflows. When these mapped tests are run via CI or via the IDE, Bumblebee takes care of uploading both the tests and the results automatically. This setup is incredibly useful for teams that use ALM for test case management but are using opensource tools like Selenium for test automation and saves hours of work.

Configuration

Bumblebee server has alredy been installed and configured per instructions.

This example is for a Maven TestNG project

Bumblebee server and clients provide a rich set of configurations to help users create appropriate workflows. This section goes over a variety of configuration schemes.

Modify the project pom.xml file and add agiletestware repository and bumblebee-annotations dependency

<dependencies>
  <dependency>
      <groupId>com.agiletestware</groupId>
      <artifactId>bumblebee-annotations</artifactId>
      <version>0.1.11</version>
      <scope>test</scope>
   </dependency>
</dependencies>

Configure Maven surefire plugin:

<build>
	<plugins>   
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-surefire-plugin</artifactId>
			<version>2.19</version>
			<configuration>
				<properties>
					<property>
						<name>listener</name>
						<value>com.agiletestware.bumblebee.annotations.testng.BumblebeeTestNGListener</value>
					</property>
				</properties>
			</configuration>
		</plugin>
	</plugins>
</build>

Add bumblebee_config.xml to the project root folder

<?xml version="1.0"?>
<bumblebee>
  <!-- URL of the Bumblebee Server -->
  <bumblebee_url>http://server_name:port/bumblebee</bumblebee_url>
  <!-- URL of HP ALM Server -->
  <alm_url>http://server_name:port/qcbin</alm_url>
  <!-- Name of HP ALM User -->
  <alm_user>qcuser</alm_user>
  <!-- Encrypted password: please use http://server_name:port/bumblebee/password/encrypt to encrypt your plain text password  -->
  <alm_encrypted_pass>fd4OMOXLJjkMR6e64RJh3Q==</alm_encrypted_pass>
  <!-- HP ALM Domain -->
  <alm_domain>DEFAULT</alm_domain>
  <!-- HP ALM Project -->
  <alm_project>annotations_demo</alm_project>
  <!-- Asynchronous (offline) update -->
  <async_update>false</async_update>
  <update_testplan>false</update_testplan>
  <enabled>false</enabled>
</bumblebee>

ALM Credentials in Server

Some security senstive users may not want to expose ALM credentials in the project configuration. Bumblebee allows users to specify ALM credentials within the Bumblebee Server in the alm-mappings.xml configuraiton section.

When specifying ALM credentials on the Bumblebee server side, they can be specified at project, domain or URL level as shown in the following examples.

Server side ALM credentials at project level can be specified as follows:

<?xml version="1.0" encoding="UTF-8"?>
 
<mappings>
    <url url="http://server_name:port/qcbin">
        <domain name="DEFAULT">
            <project name="annotations_demo">
                <credentials username="qcuser" encrypted_password="fd4OMOXLJjkMR6e64RJh3Q==" />
            </project>
        </domain>
    </url>
</mappings>

Server side ALM credentials at domain level can be specified as follows:

<?xml version="1.0" encoding="UTF-8"?>
<mappings>
    <url url="http://server_name:port/qcbin">
        <domain name="DEFAULT">
			<credentials username="qcuser" encrypted_password="fd4OMOXLJjkMR6e64RJh3Q==" />
            <project name="annotations_demo">
            </project>
        </domain>
    </url>
</mappings>

Server side ALM credentials at URL level can be specified as follows:

<?xml version="1.0" encoding="UTF-8"?>
<mappings>
    <url url="http://server_name:port/qcbin">
		<credentials username="qcuser" encrypted_password="fd4OMOXLJjkMR6e64RJh3Q==" />
        <domain name="DEFAULT">
            <project name="annotations_demo">
            </project>
        </domain>
    </url>
</mappings>

When ALM credentials are configured on the Bumblebee server side, the alm_user and alm_encrypted_pass attributes are not required on the client side bumblebee_config.xml file. However you may still specify these attributes, which will override the server side configuration.

Server side ALM credentials can also be specified to use API keys for authentication instead of username and password. To use this, create an ALM API key and specify it in the alm-mappings.xml as follows.

Server side ALM API keys at project level can be specified as follows:

<?xml version="1.0" encoding="UTF-8"?>
 
<mappings>
    <url url="http://server_name:port/qcbin">
        <domain name="DEFAULT">
            <project name="annotations_demo">
                <credentials username="qcuser" client_id="api-key-client-id" secret="api-key-secret" />
            </project>
        </domain>
    </url>
</mappings>

Server side ALM API keys at domain level can be specified as follows:

<?xml version="1.0" encoding="UTF-8"?>
 
<mappings>
    <url url="http://server_name:port/qcbin">
        <domain name="DEFAULT">
			<credentials username="qcuser" client_id="api-key-client-id" secret="api-key-secret" />
            <project name="annotations_demo">
            </project>
        </domain>
    </url>
</mappings>

Server side ALM API keys at URL level can be specified as follows:

<?xml version="1.0" encoding="UTF-8"?>
 
<mappings>
    <url url="http://server_name:port/qcbin">
		<credentials username="qcuser" client_id="api-key-client-id" secret="api-key-secret" />
        <domain name="DEFAULT">
            <project name="annotations_demo">
            </project>
        </domain>
    </url>
</mappings>

Note that the api-key-client-id and api-key-secret in the above examples should be replaced with the clientId and secret of the relevant ALM API key.

JAVA Properties

Values provided with Java Properties override the ones provided in XML configuration.

The Bumblebee properties can also be passed using java properties using us -D<property_name>

Name	Description
bumblebee.config	Path to a configuration file. If not present, then `${project.basedir}/bumblebee_config.xml` is used
bumblebee.config.bumblebee_url	URL of the Bumblebee Server
bumblebee.config.alm_url	URL of ALM Server
bumblebee.config.alm_user	Name of ALM User
bumblebee.config.alm_password	Encrypted user password
bumblebee.config.alm_domain	Name of ALM Domain
bumblebee.config.alm_project	Name of ALM Project
bumblebee.config.async_update	Asynchronous (offline) update
bumblebee.config.take_screenshot	Boolean flag indicating whether a screenshot of a failure should be automatically done (for WebDriver tests only)
bumblebee.config.enabled	Boolean flag indicating whether Bumblebee functionality is enabled or not
bumblebee.config.update_testplan	Boolean flag indicating whether Bumblebee should update existing tests in ALM TestPlan module

Example

mvn clean test -Dbumblebee.config.bumblebee_url=https://company.alm.com:8080/bumblebee

Disable Exporting

To disable exporting all test results into HP ALM, add <enabled>false</enabled> into bumblebee_config.xml

<?xml version="1.0"?>
	<bumblebee>
	  <!-- URL of the Bumblebee Server -->
	  <bumblebee_url>http://server_name:port/bumblebee</bumblebee_url>
	  <!-- URL of HP ALM Server -->
	  <alm_url>http://server_name:port/qcbin</alm_url>
	  <!-- Name of HP ALM User -->
	  <alm_user>qcuser</alm_user>
	  <!-- Encrypted password: please use http://server_name:port/bumblebee/password/encrypt to encrypt your plain text password  -->
	  <alm_encrypted_pass>fd4OMOXLJjkMR6e64RJh3Q==</alm_encrypted_pass>
	  <!-- HP ALM Domain -->
	  <alm_domain>DEFAULT</alm_domain>
	  <!-- HP ALM Project -->
	  <alm_project>annotations_demo</alm_project>
	  <enabled>false</enabled>
	</bumblebee>

During the execution the following message will be shown in console output: Bumblebee: Bumblebee has been disabled in configuration --> Results will not be exported to HP ALM

Disable TestPlan Update

Sometimes users may not want to update existing tests in the TestPlan and only want to update test results in the TestLab. To do this, add the <update_testplan>false</update_testplan> configuration switch. By default (true), Bumblebee updates test in TestPlan for each run.

<bumblebee>
	<bumblebee_url>http://bbe:8888/bumblebee</bumblebee_url>
	<alm_url>http://alm:8080/qcbin</alm_url>
	<alm_user>user</alm_user>
	<alm_encrypted_pass>fd4OMOXLJjkMR6e64RJh3Q==</alm_encrypted_pass>
	<alm_domain>DEFAULT</alm_domain>
	<alm_project>demo</alm_project>
	<update_testplan>false</update_testplan>
</bumblebee>

In such a case, Bumblebee will not update existing tests in TestPlan and add a new Comments sections into a run in TestLab:

Result in HP ALM

while existing test steps will get No Run status:

Result in HP ALM

Override Server Configs

It is also possible to override server side configuration with the bumblebee XML configuration. To do that, just add a new set of mapping elements into configuration:

<bumblebee>
	<bumblebee_url>http://bbe:8888/bumblebee</bumblebee_url>
	<alm_url>http://alm:8080/qcbin</alm_url>
	<alm_user>user</alm_user>
	<alm_encrypted_pass>fd4OMOXLJjkMR6e64RJh3Q==</alm_encrypted_pass>
	<alm_domain>DEFAULT</alm_domain>
	<alm_project>demo</alm_project>
	
	<mapping field_type="test" field_label="user field" bumblebee_label="test user field" default_value="something"/>	
</bumblebee>

mapping element field_type has the following attributes:

field_type: a type of entity in ALM for which mapping is provided. The following are supported:
test: test in TestPlan
test-set: test set in TestLab
test-instance: test instance in TestLab
run-step: step of a test in TestPlan
step: step of a test in a run in TestLab
run: run in TestLab
field_label: label of a field in ALM (not a system name as in mapping configuration on server), e.g. 'Status' or 'User field', etc...
bumblebee_label: this is a label which should be used as @Parameter.name to pass a different value
default_value: default value

Example: If you wish to provide value for test's User field, the following element should be used:

<mapping field_type="test" field_label="user field" bumblebee_label="test user field" default_value="something"/>

When running Maven build, the following notification is displayed in the console output:

Bumblebee: Passing mappings configuration to the server: [Mapping [fieldLabel=user field, fieldType=TEST, defaultValue=something, bumblebeeLabel=test user field]]
Bumblebee: Mapping configuration on Bumblebee Server will be ignored

And results in ALM:

Result in HP ALM

Disable specific tests

To disable exporting results for a particular test, specify the enabled = BooleanValue.FALSE attribute to Bumblebee annotation:

@Bumblebee(testlab = "Root\\junit_simple", testset = "class_annotations", testplan = "Subject\\junit_simple", enabled = BooleanValue.FALSE)
	public class SomePassSomeFailTest {
 
	  @Test
	  public void test1() {
	    Assert.fail("test1 failed");
	  }
	}

Add Bumblebee annotations

@Bumblebee Java annotation can be added on class or method level. Bumblebee annotation on method level overrides values set by Bumblebee annotation on class level

Bumblebee Annotation Reference

Name	Description	Applicable	Required
testplan	Path to test in a Test Plan in HP ALM. e.g. Subject\Test1	Class, method	Yes
testlab	Path to test set in a test lab in HP ALM. e.g. Root\Test1	Class, method	Yes
testset	Name of test set in HP ALM	Class, method	Yes
testname	If specified, then the value of testname parameter will be set as the,name of test in HP ALM. If not set, test name will be set to fully,qualified method name, e.g. com.annotations.Demo.method1	Method	No
almid	Defines id of a test in a HP ALM test plan which needs to be updated. If specified then testplan and testname are ignored	Method	No
description	Description for test in HP ALM	Method	No
overwriteAlmSteps	If set to BooleanValue.TRUE, all existing test steps will be deleted	Class, method	No
parameters	An array of custom parameters which are passed to the bumblebee server and then mapped to HP ALM custom fields. Please refer to documentation documentation on how to setup mappings on Bumblebee server	Class, method	No

Setting Custom Fields

Bumblebee supports setting of custom fields for HP ALM Test, Test Set, Run, Design Step, Run Step, Test Instance

Bumblebee annotation allows users to pass custom parameters to Bumblebee Server which then are mapped to custom user fields in HP ALM.

Set the Parameter name to match the label attribute of mapping in mapping XML on the server. For example, if the server has this mapping: <mapping alm_field="TS_USER_01" label="user field"/>, the java code would look like with look up happening on the label which is more user friendly.

	public class UpdateExistingTest {
		@Bumblebee(testplan = "Subject\\custom fields", description = "Test description",
				testlab = "Root\\custom fields", testset = "map custom fields",
				parameters = {@Parameter(name = "user field", value = "value from JUnit") })
		@Test
		public void test1() {
			Assert.fail("test1 failed");
		}
	}

After export TS_USER_01 field of the test in HP ALM test plan gets the value from Java class:

Result in HP ALM

Dynamic Annotations

The following ALM parameters can contain dynamic values

testplan
testlab
testset
testname
description

To define a dynamic value, just add a placeholder in format: ${property_name}, where property_name is the name of property containing value for the parameter. Bumblebee resolves dynamic values in run time using Java System Properties or XML configuration file.

To set a value for dynamic parameter in bumblebee_config.xml file, just add new property element. E.g. to provide a value for $example placeholer

@Bumblebee(testset="Test set: ${example}")

<property name="example">some value</property>

To provide a value with Java System Property, just pass them into your Maven build

mvn clean test -Dexample=Something

If both XML configuration and Java System Property are provided, value in Java System Property will be used.

Attachments and Steps

Adding Attachments

Many times, tests need to output logs or screenshots or configs next to results to provide appropriate context. Bumblebee annotations package provides some convienent static utility class com.agiletestware.bumblebee.annotations.util.CurrentTest to help with this. The com.agiletestware.bumblebee.annotations.util.CurrentTest consist of a few methods

addAttachment(File): Add a single file attachment
addAttachment(String, InputStream): Add a single attachment from a stream
addAttachments(List<File>): Add list of files

Example of adding attachments:

@Bumblebee(testlab = "Root\\junit_simple", testset = "class_annotations", testplan = "Subject\\junit_simple")
public class SomePassSomeFailTest {
 
	@Test
	public void test1() {
		CurrentTest.addAttachment(new File("test.log"));
	}
}

Results in ALM:

Result in HP ALM

Adding Steps

Sometimes it is ncessary to programtically add design steps to ALM tests. Bumblebee provides com.agiletestware.bumblebee.annotations.util.CurrentTest.addStep(String) method which returns a builder object which provides convenient way to set attributes of step.

@Bumblebee(testlab = "Root\\junit_simple", testset = "class_annotations", testplan = "Subject\\junit_simple")
public class SomePassSomeFailTest {
 
	@Test
	public void test_additional_step() {
		final Step step = CurrentTest.addStep("Additional step")
				.description("description")
				.expected("expected")
				.actual("actual")
				.status(Status.FAILED)
				.build();
		step.addAttachment(new File("test.log"));
	}
}

Results in ALM:

Result in HP ALM

Log to Test Step

To add some log messages into test step's actual field, use com.agiletestware.bumblebee.annotations.util.CurrentTest.log(String) method:

import org.junit.Test;
 
import com.agiletestware.bumblebee.annotations.Bumblebee;
import com.agiletestware.bumblebee.annotations.util.CurrentTest;
 
@Bumblebee(testlab = "Root\\junit_simple", testset = "class_annotations", testplan = "Subject\\junit_simple")
public class SomePassSomeFailTest {
 
	@Test
	public void test1() {
		CurrentTest.log("Log line one");
		CurrentTest.log("Log line two");
	}
}

Results in ALM:

Result in HP ALM

Run Tests and Enjoy

Once the project has been configured with Bumblebee annoations, simply run via mvn test and view the results in ALM.

Example Scenarios

We have created a few example mapping and configuration scenarios. There are a lot of possibilities due to the flexible nature of Bumblebee

Class Level Annotations

@Bumblebee(testlab = "Root\\testng_simple", testset = "class_annotations", testplan = "Subject\\testng_simple")
public class SomePassSomeFailTest {
 
  @Bumblebee(testname = "Test with changed name")
  @Test
  public void test1() {
    Assert.fail("test1 failed");
  }
 
  @Test
  public void test2() {
    System.out.println("test2 passed");
  }
 
  @Test
  public void test3() {
    System.out.println("test3 passed");
  }
 
  @Test
  public void test4() {
    Assert.fail("test4 failed");
  }
 
  @Test
  public void test5() {
    System.out.println("test5 passed");
  }
}

Results in ALM: Result in HP ALM

Method Level Annotations

 
public class SomePassSomeFailTest {
 
  @Test
  public void test1() {
    Assert.fail("test1 failed");
  }
 
  @Bumblebee(testname = "TestNG test", testlab = "Root\\testng_simple", testset = "method_annotation", testplan = "Subject\\testng_simple")
  @Test
  public void test2() {
    System.out.println("test2 passed");
  }
}

Results in ALM: Result in HP ALM

Mapping Tests to ALM

Lets say we have some test in ALM with id=50 and we want to map it to our new java TestNG automation tests

Map to existing test

It contains two design steps:

Map to existing test

To map test method in java class to the test in HP ALM, almid property can be used. Also it is necessary to define testlab and testset properties :

public class UpdateExistingTest {
  @Bumblebee(almid = 141, description = "Updated description", testlab = "Root\\testNG existing test", testset = "map existing test")
  @Test
  public void test1() {
    Assert.fail("test1 failed");
  }
}

Results in HP ALM: Test description in Test Plan is updated with the value of Bumblebee description field:

Results in HP ALM

There is also a new test step with name Test Results:

Results in HP ALM

And results are exported to the HP ALM Test Lab. In such a case, all steps get the same status:

Result in HP ALM

Bumblebee almid field can accept an array of test IDs almid={42,24}. In such a case, test result will be mapped into two tests in ALM.

Map all methods to one test

Sometimes test classes are designed in the way that each of them represents a single test case and test methods are just a test steps. To export all methods of a single test class to HP ALM, mark the class with @Bumblebee annotation with hasSteps attribute set to BooleanValue.TRUE

Bumblebee relies on test method execution order provided by TestNG, so you have to provide a fixed test methods execution order somehow (e.g. via priorities). Otherwise test steps will have incorrect execution order and consequently wrong order in ALM.

import org.testng.annotations.Test;
import com.agiletestware.bumblebee.annotations.BooleanValue;
import com.agiletestware.bumblebee.annotations.Bumblebee;
 
@Bumblebee(testname = "SingleTest", testplan = "Subject\\testng_simple", testlab = "Root\\testng_simple", testset = "method_annotations", hasSteps = BooleanValue.TRUE)
public class SingleTest {
 
	@Test(priority = 1)
	public void methodOne() {
		System.out.println("Passed");
	}
 
	@Test(priority = 2)
	public void methodTwo() {
		throw new RuntimeException("ooops");
	}
 
	@Test(priority = 3)
	public void methodThree() {
		System.out.println("Passed");
	}
}

In such a case, the new test with three steps will be created in HP ALM Test Plan:

Result in HP ALM

and results will also appear in Test Lab:

Result in HP ALM

Change Step Name

By default, test step names are the same as names of corresponding test methods in your Java code.
To change test step names and some other export settings, use the @BumblebeeTestStep annotation. This annotation can be applied to test methods only and has the following properties:

Name	Description	Required
name	Name of a test step in HP ALM	Yes
description	Description of a test step in HP ALM	No
expected	Value for `Expected` field of a test step in HP ALM	No

import org.testng.annotations.Test;
import com.agiletestware.bumblebee.annotations.BooleanValue;
import com.agiletestware.bumblebee.annotations.Bumblebee;
import com.agiletestware.bumblebee.annotations.BumblebeeTestStep;
 
@Bumblebee(testname = "SingleTest", testplan = "Subject\\testng_simple", testlab = "Root\\testng_simple", testset = "method_annotations", hasSteps = BooleanValue.TRUE)
public class SingleTest {
 
	@Test(priority = 1)
	public void methodOne() {
		System.out.println("Passed");
	}
 
	@Test(priority = 2)
	public void methodTwo() {
		throw new RuntimeException("ooops");
	}
 
	@BumblebeeTestStep(name = "method three", description = "method three description", expected = "expected")
	@Test(priority = 3)
	public void methodThree() {
		System.out.println("Passed");
	}
}

Result in HP ALM

JUnit Privacy Policy