There are manifold options for (functional) testing of REST- and Web Service interfaces, e.g., Java-Tool for REST interfaces or SoapUI for even both. The weapon of choice for this tutorial is JMeter, as it comes with a plurality of test flow configuration tools, parametrized testing, CLI execution, plugin integration and much more. Although JMeter’s main intent is load and performance testing, it can be used as “regular” interface testing tool by reducing the amount of generated requests.
This tutorial describes how to integrate JMeter test plans in a Jenkins environment, an award-winning CI/CD Java application for building and testing software projects. By doing so, the tutorial bases upon existing blog articles and extends them where necessary to address common problems and respect recent evolvment of JMeter and Jenkins.
The main approach for integration is twofold: 1) trigger JMeter test plan execution in a Maven build, and 2) create a Maven job in Jenkins for this prepared build. In particular, the test plan execution is triggered in the “verify” phase, the phase that runs any checks to verify the package meets quality criteria.
The tutorial is structured in three parts. The first part covers preparation of the environment, containing an execution environment, third-party software and Jenkins itself. The second part describes how to execute a JMeter test plan in a Maven build. The third part addresses the execution of the Maven build in Jenkins.
Prepare your environment
Quick read – Install Maven and a version control system using the Ubuntu package manager apt-get, install Jenkins as described here, and install the Performance Plugin (previously called JMeter plugin) via the Jenkins plugin manager.
The following paragraphs describe how to prepare the testing environment for integrating JMeter test plans in a Jenkins environment. The explanation assumes Ubuntu 14.04 and root privileges. To follow this tutorial, you can setup a virtual machine using the ISO image of Ubuntu 14.04 Desktop version. Although all tools described in this tutorial can be used via CLI, the web interface of Jenkins is quite comfortable. Disk size should be about 11 GB.
Start the environment preparation by installing Maven, the central element of the JMeter integration (see above). Use the package manager for installation
apt-get install maven
In case you don’t use a version control system yet, install one locally. This eases the transfer of your JMeter test plans and settings to the Jenkins workspace, as you can simply provide version control credentials and Jenkins performs a checkout automatically. It is also possible to copy your files manually to the Jenkins workspace, a task that might get cumbersome due to file permission and ownership flaws.
apt-get install git
Finally, install Jenkins using the Ubuntu package manager as described here. The installation contains the JDK 7 and the Jetty application container. Hence, it is self-contained and Jenkins can be found at http://localhost:8080 after installation.
After installing Jenkins, further configuration is required. First, set the Maven installation in Manage “Jenkins > Configure System”. Second, install the Performance Plugin (previously called JMeter plugin). For this, use the Jenkins Plugin Manager. In contrast, the Maven plugin is already contained in the Jenkins standard installation. After these confguration steps, restart Jenkins by opening http://localhost:8080/restart.
There are three third-party software components that can be installed optionally. First, a Java JDK in a version of choice (but greater or equal to 7.0). In case a specific version should be used, it can be installed manually, although the required JDK 7 is covered by the previously mentioned package installations or by the Jenkins installation, at the latest.
Since Jenkins is a Java application (see above), an application container is required for execution. Although Jenkins comes with Jetty as build-in container, any other application container can be used, like Tomcat or Glassfish.
The third optional third-party application is JMeter. It is included via a Maven dependency and hence, there is no need to install it manually or additionally. Although JMeter is required to edit your test plans – the *.jmx files – it is marked as optional, since a manual installation is not mandatory for the test environment.
Prepare Maven project
Executing JMeter test plans in a Maven build is achieved in three steps. First, prepare a certain file structure in your project, as depicted in the following screenshot.
In particular, place the JMeter test plans (*.jmx) to /src/test/jmeter, and copy the files jmeter-results-detail-report_21.xsl and jmeter-results-report_21.xsl from the JMeter extras folder to /src/test/resources. Although you can find the advice to copy the JMeter properties file to src/test/jmeter, you can run the Maven build without it, since it is only necessary for specific configurations, but not general functionality.
Second, trigger JMeter test plan execution using the aforementioned JMeter Maven plugin in the pom.xml. This is achieved by the following lines:
Now the JMeter test plans can already be executed. For this, execute Maven with the target phase “verify”. In Eclipse, run the project as “Maven build” with goal “verify”. The results will be placed in /target/jmeter/report.
Most test plans use a set of properties to parametrize the executed tests. These properties can be defined in the Maven POM in terms of profiles and passed to the JMeter test plan execution. In particular, an arbitrary set of profiles can be defined and the values of the selected profile are passed to JMeter using the “propertiesUser” node. The following example illustrates this approach.
<!-- In the profiles node -->
<!-- In the execution node -->
There are some important aspects about property use:
- Be aware that empty property entries will result in a NullPointerException in the Maven JMeter plugin. So if you do not provide a profile to the Maven execution or use empty property nodes, this will result in an exception.
Further configuration parameters for the JMeter call can be found in the example pom.xml of the JMeter Maven plugin.
The following code listing shows the complete Maven pom.xml.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
Configure project in Jenkins
The third and final step for integrating JMeter test plans in a Jenkins environment is to create a Maven project in Jenkins. This kind of project will parse and execute the provided pom.xml. Basically, there are only three things to specify in this new project: the version control credentials so the project can checkout all required files, the name of the root POM file, and the goals and options to execute.
In case you followed this tutorial without any adaptions, the root POM file is “pom.xml”, and the goal is “verify”. This is all and the first (simple) JMeter test plan can be executed within a Jenkins project. To select a profile defined above, provide the parameter “-P<profileName>” to the Jenkins job. If you do not select a profile, this will again result in a NullPointerException as explained above.
There are some important aspects about Jenkins project setup:
Now that the JMeter test plan gets executed in a Jenkins project, its time to enhance the ouput of results: in the project settings, add a post-build action “Publish performance test result report”, select as type “JMeter”, and provide the path to the report files. This will add the entry “Performance trend” in the left navigation of the Jenkins project.
Use own libraries
There might be situations that require the use of further libraries in JMeter. These libraries can be placed in the lib directory of JMeter using the common Maven functionality, i.e. defining them as plugin dependency. In case a local jar archive should be included, a local repository can be used.
Hints about proxies
If you are behind a proxy several settings are required. Since this is not directly related to the work with JMeter and Jenkins, this description is placed in the “appendix” of the tutorial.
Set the proxy information in the Maven file in .m2/settings.xml and ensure that this file is used by Eclipse in Window > Preferences > Maven > User Settings > Select the file.
Set proxy information for Eclipse in “Window > Preferences > General > Network Connections”.
Set proxy information in Jenkins at “Manage Jenkins > Manage Plugins > Tab Advanced”.