TestNG

Description

Bumblebee provides a set of Java annotations which allows you to send your TestNG test XML reports right after test execution in your favorite IDE.

Technically it consists of three parts:

  • @Bumblebee annotation
  • TestNG test run reporter: com.agiletestware.bumblebee.annotations.testng.BumblebeeTestNGReporter
  • XML configuration file which contains ALM connection and other details needed for functioning of bumblebee-annotations

Prerequisites

  • Prerequisites
  • Maven project with TestNG tests

Installation and configuration

1) Modify your pom.xml file and add agiletestware repository and bumblebee-annotations dependency:

<dependencies>
    <dependency>
        <groupId>com.agiletestware</groupId>
        <artifactId>bumblebee-annotations</artifactId>
        <version>0.0.1-SNAPSHOT</version>
        <scope>test</scope>
    </dependency>
</dependencies>
<repositories>
    <repository>       
        <id>com.agiletestware</id>
        <name>Agiletestware Maven Repository</name>
        <url>http://www.agiletestware.com/maven/</url>
    </repository>
</repositories>

2) Configure maven surefire plugin:

<build>
  <plugins>   
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.19</version>
        <configuration>
          <properties>
            <property>
              <name>listener</name>
              <value>com.agiletestware.bumblebee.annotations.testng.BumblebeeTestNGReporter</value>
            </property>
          </properties>
        </configuration>
      </plugin>
  </plugins>
</build>

3) Create bumblebee_config.xml file and put into your project root folder. Here is a description of configuration file:

<?xml version="1.0"?>
<bumblebee>
  <!-- URL of the Bumblebee Server -->
  <bumblebee_url>http://server_name:port/bumblebee</bumblebee_url>
  <!-- URL of HP ALM Server -->
  <alm_url>http://server_name:port/qcbin</alm_url>
  <!-- Name of HP ALM User -->
  <alm_user>qcuser</alm_user>
  <!-- Encrypted password: please use http://server_name:port/bumblebee/password/encrypt to encrypt your plain text password  -->
  <alm_encrypted_pass>fd4OMOXLJjkMR6e64RJh3Q==</alm_encrypted_pass>
  <!-- HP ALM Domain -->
  <alm_domain>DEFAULT</alm_domain>
  <!-- HP ALM Project -->
  <alm_project>annotations_demo</alm_project>
</bumblebee>

Add Bumblebee annotations to your TestNG test classes/methods

@Bumblebee Java annotation can be added on class or method level When class or method is marked with Bumblebee annotation, its result will be exported to HP ALM.

Bumblebee annotation parameters

Name Description Applicable Required
testplan Path to test in a Test Plan in HP ALM. e.g. Subject\Test1 Class, method Yes
testlab Path to test set in a test lab in HP ALM. e.g. Root\Test1 Class, method Yes
testset Name of test set in HP ALM Class, method Yes
testname If specified, then the value of testname parameter will be set as the,name of test in HP ALM. If not set, test name will be set to fully,qualified method name, e.g. com.annotations.Demo.method1 Method No
almid Defines id of a test in a HP ALM test plan which needs to be updated. If specified then testplan and testname are ignored Method No
description Description for test in HP ALM Method No
overwriteAlmSteps If set to true, all existing test steps will be deleted Class, method No
parameters An array of custom parameters which are passed to the bumblebee server and then mapped to HP ALM custom fields. Please refer to documentation documentation on how to setup mappings on Bumblebee server Class, method No
almReqIds An array of HP ALM requirements IDs to which this test should be linked Class, method No
almReqRecursive If set to BooleanValue.TRUE, test is linked to all child requirements of requirements defined y almReqIds parameter Class, method No

Bumblebee annotation on method level overrides values set by Bumblebee annotation on class level.

Run Maven build

To run TestNG tests and send results to HP ALM, just run Maven test phase on your project: e.g.: mvn test

Examples

Bumblebee annotation on class level:

@Bumblebee(testlab = "Root\\testng_simple", testset = "class_annotations", testplan = "Subject\\testng_simple")
public class SomePassSomeFailTest {

  @Bumblebee(testname = "Test with changed name")
  @Test
  public void test1() {
    Assert.fail("test1 failed");
  }

  @Test
  public void test2() {
    System.out.println("test2 passed");
  }

  @Test
  public void test3() {
    System.out.println("test3 passed");
  }

  @Test
  public void test4() {
    Assert.fail("test4 failed");
  }

  @Test
  public void test5() {
    System.out.println("test5 passed");
  }
}

After export to HP ALM:

Result in HP ALM

Bumblebee annotation on method level:

public class SomePassSomeFailTest {

  @Test
  public void test1() {
    Assert.fail("test1 failed");
  }

  @Bumblebee(testname = "TestNG test", testlab = "Root\\testng_simple", testset = "method_annotation", testplan = "Subject\\testng_simple")
  @Test
  public void test2() {
    System.out.println("test2 passed");
  }
}

Results in HP ALM:

Result in HP ALM

Overriding annotation values on methods:

public class AllPassedTest {

  @Bumblebee(testlab = "Root\\testng_simple", testset = "method_annotation", testplan = "Subject\\testng_simple")
  @Test
  public void test1() {
    System.out.println("test1");
  }

  @Bumblebee(testlab = "Root\\testng_simple", testset = "method_annotation", testplan = "Subject\\testng_simple")
  @Test
  public void test2() {
    System.out.println("test1");
  }

  @Bumblebee(testlab = "Root\\testng_simple", testset = "method_annotation", testplan = "Subject\\testng_simple")
  @Test
  public void test3() {
    System.out.println("test1");
  }

  @Test
  public void test4() {
    System.out.println("test1");
  }
}

Results:

Result in HP ALM

Map TestNG test method to existing test in HP ALM:

Here is the existing Manual test in HP ALM Test Plan with id = 11:

Map to existing test

It contains two design steps:

Map to existing test

To map test method in java class to the test in HP ALM, almid property can be used. Also it is necessary to define testlab and testset properties :

public class UpdateExistingTest {
  @Bumblebee(almid = 11, description = "Updated description", testlab = "Root\\testNG existing test", testset = "map existing test")
  @Test
  public void test1() {
    Assert.fail("test1 failed");
  }
}

Results in HP ALM: Test description in Test Plan is updated with the value of Bumblebee description field:

Results in HP ALM

And results are exported to the HP ALM Test Lab. In such a case, all steps get the same status. Note that the new step was added:

Results in HP ALM

Setting custom fields in HP ALM from TestNG class

Bumblebee annotation allows users to pass custom parameters to Bumblebee Server which then are mapped to custom user fields in HP ALM. Please refer to documentation on how to setup mappings on Bumblebee server. Parameter name shall match to the "label" attribute of mapping in mapping XML on the server:

<mapping alm_field="TS_USER_01" label="user field"/>

To map some value to TS_USER_01 field in HP ALM, define a parameter with name="user field":

public class UpdateExistingTest {
  @Bumblebee(testplan = "Subject\\custom fields", description = "Test description",
      testlab = "Root\\custom fields", testset = "map custom fields",
      parameters = {@Parameter(name = "user field", value = "value from TestNG") })
  @Test
  public void test1() {
    Assert.fail("test1 failed");
  }
}

After export TS_USER_01 field of the test in HP ALM test plan gets the value from Java class:

Results in HP ALM

Currently Bumblebee supports setting of custom fields for HP ALM Test, Test Set and Run.

Mapping tests to HP ALM requirements

Bumblebee annotations allows users to map their test classes/methods to HP ALM requirements with a help of two parameters: * almReqIds - an array of requirements IDs in HP ALM * almReqRecursive - defines if test should be also linked to child requirements

e.g.:

import org.testng.annotations.Test;
import com.agiletestware.bumblebee.annotations.BooleanValue;
import com.agiletestware.bumblebee.annotations.Bumblebee;

public class AllPassedTest {

    @Bumblebee(testlab = "Root\\testng_simple", testset = "method_annotations", testplan = "Subject\\testng_simple", almReqIds = { 1,   4 }, almReqRecursive = BooleanValue.TRUE)
    @Test
    public void test1() {
        System.out.println("test1");
    }
}

During running maven test phase, Bumblebee creates test in HP ALM and maps it to the requirements with ID 1 and 4, and also to their children:

Result in HP ALM

If requirement mappings are specified on a class-level annotation, they will be applied to all test methods of the class.

Exporting all test methods into one HP ALM test

Sometimes test classes are designed in the way that each of them represents a single test case and test methods are just a test steps.
While some can say that it is not a best test design practice, Bumblebee supports such a use case.
To export all methods of a single test class to HP ALM, this class shall be marked with @Bumblebee annotation with hasSteps attribute set to BooleanValue.TRUE.
e.g.:

import org.testng.annotations.Test;
import com.agiletestware.bumblebee.annotations.BooleanValue;
import com.agiletestware.bumblebee.annotations.Bumblebee;

@Bumblebee(testname = "SingleTest", testplan = "Subject\\testng_simple", testlab = "Root\\testng_simple", testset = "method_annotations", hasSteps = BooleanValue.TRUE)
public class SingleTest {

    @Test(priority = 1)
    public void methodOne() {
        System.out.println("Passed");
    }

    @Test(priority = 2)
    public void methodTwo() {
        throw new RuntimeException("ooops");
    }

    @Test(priority = 3)
    public void methodThree() {
        System.out.println("Passed");
    }
}

In such a case, the new test with three steps will be created in HP ALM Test Plan:

Result in HP ALM

and results will also appear in Test Lab:

Result in HP ALM

Note: Please note that Bumblebee relies on test method execution order provided by TestNG, so you have to provide a fixed test methods execution order somehow (e.g. via priorities).
Otherwise your test steps will have incorrect execution order and consequently wrong order in HP ALM.

By default, test step names are the same as names of corresponding test methods in your Java code.
To change test step names and some other export settings, a @BumblebeeTestStep annotation was introduced.
This annotation can be applied to test methods only and has the following properties:

Name Description Required
name Name of a test step in HP ALM Yes
description Description of a test step in HP ALM No
expected Value for Expected field of a test step in HP ALM No

e.g.:

import org.testng.annotations.Test;
import com.agiletestware.bumblebee.annotations.BooleanValue;
import com.agiletestware.bumblebee.annotations.Bumblebee;
import com.agiletestware.bumblebee.annotations.BumblebeeTestStep;

@Bumblebee(testname = "SingleTest", testplan = "Subject\\testng_simple", testlab = "Root\\testng_simple", testset = "method_annotations", hasSteps = BooleanValue.TRUE)
public class SingleTest {

    @Test(priority = 1)
    public void methodOne() {
        System.out.println("Passed");
    }

    @Test(priority = 2)
    public void methodTwo() {
        throw new RuntimeException("ooops");
    }

    @BumblebeeTestStep(name = "method three", description = "method three description", expected = "expected")
    @Test(priority = 3)
    public void methodThree() {
        System.out.println("Passed");
    }
}

And it produces the following test steps in HP ALM Test Plan:

Result in HP ALM