:::: MENU ::::

CICD with Jenkins and HP ALM / Quality Center – Part 3

Some of our customers use HP Performance Center for load/stress testing. Instead of running performance at the end of the release and finding critical issues, these customers run HP Performance Center tests as a regular part of the development process via Jenkins. Bumblebee’s Jenkins plugin allows organizations to schedule, run, and report HP Performance Center tests and implement their desired CICD vision.

In this post, we will explain you can use Bumblebee Jenkins plugin to run HP Performance Center tests.

Part 3: Running HP Performance Center tests from Jenkins

To configure a Jenkins job to run HP Performance Center tests, add the Bumblebee HP PC Test Runner build step.

HP Performance Center produces test reports that can be published inside Jenkins builds. Bumblebee HP PC Test Runner task automatically downloads these reports from the HP Performance Center server and copies them to the specified Results Directory.

To publish these reports inside Jenkins, add the Archive Artifacts post-build step to job configuration and define appropriate values for Files to Archive

When the Jenkins job is triggered it starts a new run in HP Performance Center for the test specified by Path To Test and Test Set properties of the task. Once the HP PC test has started, Bumblebee HP PC Task waits for it to finish and checks the run’s possible Run States from time to time.

If a test has failed, Bumblebee marks the build as passed or failed based on the Fail Build If Task Fails job configuration property. If true, the build is marked as FAILED and build is aborted. If false, Bumblebee simply proceeds with the next test.

If an error occurs during fetching runs status from PC, Bumblebee will retry failed action according to the retry settings defined for a task.

Sample execution log:

Bumblebee is very flexible and simple to setup. For a detailed step by step guide, please check out the latest Bumblebee docs

CICD with Jenkins and HP ALM / Quality Center – Part 2

If you are using HP ALM, chances are that there are significant amount of tests stored in ALM using a variety of HP Tools like UFT, LeanFT, etc. Instead of running HP tests via HP ALM and everything else via Jenkins and then trying to figure out what is failing in Jenkins vs. HP ALM, you can control everything via Jenkins while keeping things as in in HP ALM.

In this post, we will explain you can use Bumblebee Jenkins plugin to run existing HP Functional tests in ALM via Jenkins. The tests results will appear in both HP ALM and Jenkins.

Part 2: Running HP ALM Functional tests (UFT, LeanFT, etc) from Jenkins

To run ALM Testset from Jenkins, add Bumblebee HP ALM Test Set Runner build step to the Jenkins job configuration

When Jenkins build is executed, the Bumblebee HP ALM Test Set Runner step connects to HP ALM server and runs the specified ALM Testsets. The build waits till the test runs is completed in HP ALM.

Example: Jenkins build console

Once the build is complete, Bumblebee HP ALM Test Set Runner transforms the HP ALM test results in a JUnit report compliant schema so that it can be published within Jenkins, using the standard Publish JUnit test result report post-build step.

Bumblebee is very flexible and simple to setup. For a detailed step by step guide, please check out the latest Bumblebee docs

CICD with Jenkins and HP ALM / Quality Center – Part 1

In many organizations, Jenkins is used to build software, run unit tests, and run various kinds of testing frameworks. Example: Selenium, JUnit, Pytest, TestNG, Visual Studio Test, etc. These test metrics are very important and should be reflected in HP ALM to show the true health of any project. Bumblebee plugin for Jenkins allows users to integrate any testing framework with HP ALM without making any code changes or writing custom tools. In addition, the plugin allows you to run HP ALM tests directly from Jenkins. Many organization use Bumblebee and Jenkins to achieve CICD.

In these series of posts we will cover 3 main topics

  1. Exporting testing results from Jenkins to HP ALM
  2. Running HP ALM Functional tests (UFT, LeanFT, etc) from Jenkins
  3. Running HP Performance Center tests from Jenkins

1. Exporting testing results from Jenkins to HP ALM:

Integrating Jenkins with HP ALM is super simple by using Bumblebee’s plugin for Jenkins. Bumblebee’s Jenkins plugin allows you to Export test results to HP ALM and automatically create TestPlan, TestLabs, TestsSets, and TestRuns. This is extremely useful for running tests directly in Jenkins and automatically these test results in HP ALM.

Bumblebee Jenkins plugin can be installed via the Jenkins update center

Configure the Jenkins plugin global settings. manage Jenkins → Manage Jenkins → Configure System

Save Configuration to save your changes. Saving may take a few seconds because the Bumblebee plugin will validate the configuration before saving.

Bumblebee’s Jenkins plugin allows automatic export of Jenkins build’s test results to HP ALM. Bumblebee supports a variety of test results formats like JUnit, NUnit, testNG, Fitnesse, and Cucumber. To enable this feature, configure the Jenkins job and add Bumblebee HP ALM Uploader post-build step. In this step, configure a few fields and specify the results that need to be published.

Once the job is configured, simply build the job. The build console will show the Bumblebee Jenkins plugin to Bumblebee REST API activity. This output is quite useful for troubleshooting.

All the results will be processed by Bumblebee server and corresponding TestPlan, TestLab, Testset, and TestRun will be created automatically.


Bumblebee is very flexible and simple to setup. For a detailed step by step guide, please check out the latest Bumblebee docs

Achieving CICD with HP ALM and open source testing tools

The CICD Dilemma

Agiletestware works with a variety of engineering organization all over the world. It is always enlightening to see how different organization implement quality engineering discipline and try to achieve CICD (continuous integration and continuous delivery). Our experience shows that CICD is a long journey and requires a great amount of experimentation, discipline, and custom integrations with a variety of tools and frameworks. There is no silver bullet that will get you to the promised land 😢 CICD takes time, patience, and a good vision 😇

Many of our customers use a wide variety of testing frameworks, tools, CI systems, ALM solution, bug trackers, requirement management systems, etc. All of them want to achieve CICD but integrating all these technologies is a very challenging task when you need to do your day job and ship features to customers.

In these series of blog posts, we will go over how many of our customers have come one step closer to achieving CICD. The blog posts will explain how a variety of testing frameworks and tools and have been integrated with HP ALM (our customer’s ALM software for tracking projects) using Agiletestware’s Bumblebee product.

The typical technology landscape at most large companies

Before we explain the solution, let us explain the typical setup found in many of our customers. Hopefully, many of you will relate to this situation.

CI Applications – Jenkins, Bamboo, and TeamCity

CI system are at the center of achieving CICD. The three biggest and most mature CI applications are Jenkins, Atlassian Bamboo, and Jetbrain’s TeamCity. CI systems are used to build software, run tests, and do a variety of other tasks. In mature engineering organizations, CI systems are used to deploy software to achieve CD. All the information about these various tasks (build, test results, etc) are contained within the CI application.

What is HP ALM?

HP ALM is the leading solution in the industry for tracking complex software project. HP ALM is used for bug tracking, requirements management, test case management, and much more. HP ALM is a commercial product and integrates with various other tools from HP tools for testing web, mobile, and various other software products.

Web Testing and Selenium WebDriver

Selenium Webdriver is an open source tool for automating browsers. Primarily, it is for automating web applications for testing purposes. Selenium has the support of some of the largest browser vendors who have taken (or are taking) steps to make Selenium a native part of their browser. It is also the core technology in countless other browser automation tools, APIs and frameworks. If you are not using Selenium for testing web applications, you should re-think your testing tool strategy and take a closer look at Selenium WebDriver 😀

Mobile Testing and Appium

Appium is an open source test automation framework used for testing native, hybrid and mobile web apps. Appium aims to automate ios, Android, and Windows mobile app from any modern programming language and any test framework. Appium is supported by SauceLabs (leading provider of web and mobile test cloud). Appium is most widely used mobile testing framework and is free.

JAVA Test Frameworks – testNG and JUnit

Testing tools like Selenium and Appium require some test framework. In the JAVA world, two of the most widely used test frameworks are JUnit and testNG.

JUnit: JUnit is a simple framework to write repeatable tests. It is an instance of the xUnit architecture for unit testing frameworks.

testNG: TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use.

All sounds simple. So what is the problem?

If you are a HP ALM shop and using HP tools (UFT, LeanFT, QTP, etc) for testing all your applications, then there are no issues and life should be good. Every software development and testing activity should nicely be reflected in HP ALM.

However, the reality is that most organizations are rarely a homogeneous HP shop. Some teams may be using HP Tools (UFT, LeanFT, QTP), mobile testing team may be using Appium or other similar open source tools, browser application testing teams may be using Selenium WebDriver and cloud browser solutions like SauceLabs and BrowserStack, API testing team may be using using ReadyAPI, and unit test is hopefully done via some x-unit framework like Junit or testNG and most of these activities are controlled via Jenkins/Bamboo/TeamCity.

The end result is that you end up with islands of information within your organization. Some information is in HP ALM and some is in Jenkins/Bamboo/Teamcity and it is impossible to get a unified view of the project. As a consequence, engineering management often makes an uninformed decision because they don’t know how to collect all the data. There are just too much human element, too many reports to prepare, and it is just a pain and no one wants to do this.

How can Agiletestware help?

Agiletestware has been helping engineering organizations achieve CICD through many of our tools. We can’t solve all your problems but we may be able to solve many of them to get you closer to CICD. HP ALM is a significant investment and you should get the most out of your investment. We believe HP ALM is still the best ALM solution in the market and has the most mature bug tracking, agile project management, requirements management, and test case management modules in a single solution.

Agiletestware’s Bumblebee product can help integrate CI applications like Jenkins/Bamboo/Teamcity, JAVA tests frameworks like testNG/JUnit, and web and mobile testing tools like Selenium WebDriver / Appium with HP ALM with little to no coding. Bumblebee simply intercepts activities within your existing testing tools and CI frameworks and exports it to HP ALM in real-time.


ReadyAPI + TestRail ? Yes you can !


2016 was a great year for Agiletestware. We added record number of customers, implemented various new features in our products and thought about new ways to bring productivity to engineering teams around the world.

Many visitors to our website, who are using ReadyAPI, wanted to see if we can create a product similar to Dragonfly (ReadyAPI + HP ALM integration solution) that would integrate ReadyAPI with Gurock TestRail (test case management). Some of our team members had experience with TestRail and we think that TestRail is definitely a world class modern test case management system. Because we understood TestRail use cases very well, it made perfect sense for us to create a new product for integrating ReadyAPI and TestRail.

We are pleased to announce our newest product – Firefly. Firefly has been built from the ground up, using the same design principles as Dragonfly. Firefly has a big focus on usability, flexibility, and speed. Use Firefly to easily integrate ReadyAPI and TestRail with simple configs. Export ReadyAPI tests and tests results into TestRail with a simple action. Significantly boost engineering productivity by removing any manual uploading of test results. Watch real-time updates into TestRail as you run ReadyAPI tests interactively or via some CI system like Jenkins, Bamboo, or Teamcity.

Come try Firefly free for 14 days and see how you too can get the most from you ReadyAPI and TestRail setup.