Atlassian Bamboo®

Why you need Bumblebee

In many organizations, Atlassian Bamboo® is used to build software, run unit tests, and run various kinds of testing frameworks. Example: Selenium, JUnit, Pytest, TestNG, Visual Studio Test, etc. These test metrics are very important and should be reflected in HP ALM to show the true health of any project.

Bumblebee add-on for Bamboo allows users to integrate any testing framework with HP ALM without making any code changes or writing custom tools. In addition, the add-on allows you to run HP ALM tests directly from Bamboo. Many organization use Bumblebee and Bamboo to achieve CICD.

Integrating Bamboo with HP ALM can be easily achieved using the Bumblebee add-on for Bamboo. Once configured, Bamboo can:

  1. Export test results to HP ALM and automatically create TestPlan, TestLabs, TestsSets, and TestRuns. This is extremely useful for running test frameworks that run on Bamboo and showing those test reports in HP ALM.
  2. Trigger test runs in HP ALM and show their status in Bamboo
  3. Run local UFT tests and report execution results
  4. Run tests in HP Performance Center

- Install add-on

To install Bumblebee add-on, navigate to Administration → Add-ons → Find new add-ons

Install Bumblebee Server before configuring the Bamboo add-on

Bumblebee Bamboo add-on communicates with HP ALM via the Bumblebee server. You must install Bumblebee server before configuring the Bamboo add-on. Bumblebee server setup instructions

bamboo

Search Bumblebee and install HP ALM - Bumblebee for Bamboo add-on

bamboo

Restart Bamboo server after it has finished installation of Bumblebee add-on.

- Configure Add-on

Configure Bumblebee add-on. Administration → Overview → Bumblebee → Global Settings

bamboo

Configure the following fields in Bumblebee Global Settings screen:

Field Name Description
BumbleBee URL URL for bumblebee server. Must end with /bumblebee. Example: http://some-server:8888/bumblebee
HP ALM URL URL for your HP ALM instance. Must end with /qcbin. Example: http://your-alm-server:8080/qcbin
PC URL URL for your HP Performance Center. Leave it empty if you will not be running PC tests.
HP ALM User Name of the HP ALM user to connect to ALM instance.
HP ALM User Password Password for HP ALM user to login into ALM
Upload timeout The number of minutes to wait for the Bumblebee server to process the request. 0 means wait indefinitely. Uploading of test results into HP ALM might take a few minutes depending upon the number of tests that must be processed.
PC timeout The number of minutes to wait for the PC test to finish. 0 means wait indefinitely.

bamboo

Save Configuration to save add-on configuration changes. Saving may take a few seconds because of Bumblebee server-side validation.


- Export Tests Results

Bumblebee add-on allows automatic export of Bamboo build's test results to HP ALM. Bumblebee supports a variety of test results formats like JUnit, NUnit, testNG, Fitnesse, and Cucumber. To enable this feature, configure the Bamboo plan and add and configure Bumblebee Task.

bamboo

Field Name Required Description
Task Description Yes Standard description for Atlassian Bamboo® task
Domain in HP ALM Yes The name of HP ALM domain where to export test results
Project in HP ALM Yes The name of HP ALM project
TestPlan in HP ALM Yes TestPlan path in HP ALM where tests should be saved. Must start with "Subject\"
TestLab in HP ALM Yes TestLab path in HP ALM where test results should be saved. Must start with "Root\"
Test Set Yes The name of test set in HP ALM
Format Yes Format of test results generated by previous tasks in the same job. Available values: junit, nunit, testng, fitnesse, cucumber
Results File Pattern Yes Provide path to the XML file(s) generated during the build.This is a comma separated list of test result directories.You can also use Ant style patterns such as **/surefire-reports/*.xml
HP ALM Mappings No If ALM Mappings are configured on the Bumblebee server, their values can be specified in this field.The format is [alm field label 1]=[value 1], [alm field label 2]=[value 2]. You can also use Atlassian Bamboo® environment variables.You can also directly specify the default value on the Bumblebee server.Please refer to Bumblebee server documentation for details on configuring HP ALM mappings.
Fail build if upload unsuccessful N/A If checked, the build will be marked as failed if for any reason the add-on was not able to upload the results.This could be due to Bumblebee server issues, HP ALM server issues, network issues, etc.
Process offline N/A If checked, Bumblebee will send test reports to the special asynchronous,processing queue on Bumblebee server and process it afterwards. Use,this when upload of test results takes much time and you don't want the,build to wait until it finishes

bamboo

Note

It is recommended to put Bumblebee Task into "Final tasks" section as they are always executed even if previous tasks fails, so if e.g. surefire maven plugin failed build because of some tests failed, Bumblebee task will be executed and send failed test results to HP ALM anyway.

configure ALM mappings (optional)

If your ALM project has custom user fields and you have already configured ALM Mappings, you can easily specify them in the Bamboo post-build step as name-value pairs. If you don't specify them in the Bamboo post-build step, Bumblebee will use the custom field values from the server (if they exist).

bamboo

Once the Bamboo plan is configured, simply build the plan. The build console will show the Bumblebee add-on to Bumblebee REST API activity. This output is quite useful for troubleshooting.

bamboo

All the results will be processed by Bumblebee server and corresponding TestPlan, TestLab, Testset, and TestRun will be created automatically.

bamboo

bamboo


- Offline processing

Processing of test results can add a few minutes to your build time. If this is not desirable, Bumblebee provides an option for processing tests results in a background thread on the Bumblebee server. Simply check the Process offline checkbox in the job configuration to enable this feature.

bamboo

When Process offline is enabled, the console output will be

bamboo

The console output will contain a link to the processing queue item.

bamboo

To view all the offline tasks being processing, simply navigate to Offline Processing Queue from the Bumblebee Server main page

bamboo

bamboo


- Running ALM Tests from Bamboo

Bumblebee Run Test Set in HP ALM task allows you to run tests stored in HP ALM (UFT, LeanFT, QTP, etc) directly from Bamboo and view the results in both Bamboo and HP ALM.

- Prerequisites

  • Job must run on Bamboo agent running Windows OS
  • Bamboo agent must NOT run as windows service. It should run as a console application (run BambooAgent.bat)
  • Appropriate version of HP ALM Connectivity Tool must be installed on Bamboo agent. Tool is available at http://your_alm_server_and_port/qcbin/PlugIns/TDConnectivity/TDConnect.exe
  • Appropriate version of HP ALM Client must be installed on Bamboo agent. Available at http://your_alm_server_and_port/qcbin/start_a.jsp?common=true

To run HP Testsets from Bamboo, add Bumblebee HP ALM Test Set Runner build step to Bamboo plan configuration

bamboo

bamboo

Field Description
Domain The name of HP ALM domain
Project The name of HP ALM project
Test Sets A list of test sets to execute. Each test set path must start with new,line. Patch must start with Root and contains full path to the test set,in HP ALM TestLab. E.g. Root\folder1\testset1, where Root\folder1 is,TestLab folder and folder1 is the name of the test set to execute
Run Mode How to run test sets. Possible values: LOCAL - run all tests on agent's,machine, SCHEDULED - run tests on planned host, REMOTE - run on remote,host
Run Host The name of host on which tests shall be run. May be blank if Run Mode is LOCAL or SCHEDULED
JUnit Results Directory Directory where JUnit-like execution reports will be placed. If it does not exist, Bumblebee will create it
Timeout The number of minutes to wait for test sets execution. 0 means wait indefinitely.

When Bamboo build is executed, the Bumblebee HP ALM Test Set Runner step connects to HP ALM server and runs the specified ALM Testsets. The build waits till the test runs is completed in HP ALM.

Example: Bamboo build log

bamboo

Once the build is complete, Bumblebee HP ALM Test Set Runner transforms the HP ALM test results into a JUnit report compliant schema so that it can be published within Bamboo, using the standard JUnit Parser step.

bamboo

bamboo

- Adding tests from HP ALM TestPlan to TestSet

If a single test or all tests in a TestPlan folder need to be added to HP ALM TestSet, then Bumblebee: Add Test to Set build step can be used:

bamboo

The step has the following configuration fields:

Field Description
Domain The name of HP ALM domain
Project The name of HP ALM project
Test Plan Path Path to a test or test folder in HP ALM TestPlan, e.g. Subject\testfolder\test1
Test Set Path Path to a HP ALM TestSet into which test(s) shall be added. If TestSet does not exist, Bumblebee will try to create it on the fly. Example: Root\testfolder\testSet

Console output:

bamboo

Results in HP ALM:

bamboo


- Running ALM Tests inside Bamboo

Bumblebee also allows you to run HP Unified Functional Testing tests directly from Bamboo and reports resutls back to Bamboo.

- Prerequisites

  • Bamboo Agent runs on Windows machine and have appropriate HP Unified Functional Testing version installed. Please see UFT requirements for a particular version of OS and other software.
  • Bamboo agent must run as a console application (not as a windows service)
  • UFT Batch Runner capability is set on agent Administration → Agents → Shared Remote Capabilities

bamboo

On Add capability screen:

  • Select and add Capability type: UFT Batch Runner
  • Path: Path to UFTbatchRunnerCMD.exe on agent system (by default it is c:\Program Files (x86)\HP\Unified Functional Testing\bin\UFTBatchRunnerCMD.exe)

bamboo

To run UFT tests locally inside of Bamboo, add the Bumblebee: Run Local UFT Test task in the Bamboo plan configuration.

bamboo

Bumblebee: Run Local UFT Test step has the following configuration parameters:

  • Test Path: The path to a test folder or test batch file (.mtb) which needs to be executed
  • Results Directory: Directory inside your project where Bumblebee can add JUnit-like execution reports. If it does not exist, Bumblebee will create it automatically.

mtb file paths

If your .mtb files are located in some GIT repository, make sure paths to tests are correct and point to tests in build directory. You can use windows batch script for this.

bamboo

When Bamboo runs Bumblebee UFT task, it will trigger local HP UFT Batch runner and record its output:

bamboo

Bumblebee UFT task transforms the results of test execution as a standard JUnit schema compliant. This JUnit report can be published in Bamboo using the standard JUnit Parser step.

bamboo

bamboo


Running HP Performance Center Tests from Bamboo

HP Performance Center Tests + Bamboo

HP Performance Center is a powerful set of tools for composing and running performance tests and used by many companies. Bumblebee Bamboo add-on allows an easy way to schedule HP Performance Center tests and report results back to Bamboo.

Prerequisites

Before creating a Bamboo job for running HP Performance Center tests, validate the Bumblebee Global Settings

Configure Bumblebee HP PC Test Runner step

To configure a Bamboo plan to run HP Performance Center tests, add the Bumblebee:Run HP PC Test step. bamboo

Bumblebee: Run HP PC Test step has the following configuration parameters

Parameter name Description
Domain Domain name in HP ALM
Project Project name in HP ALM
Results Directory Directory to which test result files will be saved
Path To Test Path to a test in HP ALM TestPlan, e.g. "Subject\folder1\test", where "Subject\folder" is a path to a test folder and "test" is the name of a test to run
Test Set Path to a test set in HP ALM TestLab, containing correspondent test instance, e.g. "Root\folder1\testSet", where "Root\folder1" is a path to a test lab folder and "testSet" is the name of a test set. If test set does not exist or test is not assigned to it, Bumblebee task will try to create a new test set and assign a test to it.
Post Run Action Defines what PC should do after a test run. Available options: Collate And Analyze, Collate Results and Do Not Collate.
Time Slot Duration Time to allot for the test (PC parameter). It cannot be less than 30 minutes (limitation by PC).
Use VUD Licenses If true, the test consumes Virtual User Day (VUD) licenses.
Timeout overrides a global PC timeout value and represents the number of minutes to wait for the Performance Center test to finish. 0 means wait indefinitely.
Retry Attempts Number of retry attempts, before task completely fails.
Retry Interval Number of seconds to wait between retry attempts.
Interval Increase Factor Increase factor for retry interval. E.g. if it is set to 2, then each subsequent wait interval between attempts will be twice bigger than the previous one.
Polling Interval The number of minutes between two test state requests.
Fail Build If Task Fails If true and task has failed (or timeout has reached), then the whole build will be failed. If false, then build will not be failed even if task has failed.

bamboo

HP PC Results Configuration

HP Performance Center produces test reports that can be published inside Bamboo builds. Bumblebee:Run HP PC Test task automatically downloads these reports from the HP Performance Center server and copies them to the specified Results Directory. These HP Performance Center reports can also be attached as Bamboo build artifacts.

  • Bamboo plan configuration -- Artifacts → Create Definition
  • Configure Name, Location, and Copy pattern fields

bamboo

Running HP PC Bamboo Job

When the Bamboo job is triggered, it starts a new run in HP Performance Center for the test specified by Path To Test and TestSet properties of the task. Once the HP PC test has started, Bumblebee:Run HP PC Test task waits for it to finish and checks the run's possible Run States from time to time. If the test reaches one of the following states, Bumblebee assumes that the test has passed:

  • Finished
  • Before Collating Results (if Post Run Action = Do Not Collate)
  • Before Creating Analysis Data (if Post Run Action = Collate Results)

If test reaches one of the following states or timeout has occurred, Bumblebee treats the test as failed:

  • Canceled
  • Run Failure
  • Aborted
  • Failed Collating Results
  • Failed Creating Analysis Data

If a test has failed, Bumblebee marks the build as passed or failed based on the Fail Build If Task Fails job configuration property. If true, the build is marked as FAILED and build is aborted. If false, Bumblebee simply proceeds with the next test.

If an error occurs during fetching runs status from PC, Bumblebee will retry failed action according to the retry settings defined for a task.

Sample execution log:

bamboo

Artifacts published by the Bamboo plan:

bamboo