Hacker News

JMeter Assertions

JMeter

Following our JMeter series we are able to create different scenarios for testing multiple endpoints, using Thread Groups and Listeners, and avoid duplication of configuration using variables.

These tools allow us to evaluate the responses’ contents and the time they took to reach us back.

Summary

View

However, unless the server give us some response different than 200 OK, we will never see a failure.

In situations of heavy load, all sort of problems can happen on the server:

  • Response’s content may be corrupted;
  • Specific response can take a long time to return, which is not acceptable for some products.

Additionally, we may want to investigate how the server reacts in situations different from 200 OK:

  • If a resource does not exist, does the system return a 404 response fast enough?
  • If a server error occurs (500 code), does the system continues to work properly?

To answer such questions, it is necessary to run additional validations to each request.

In JMeter, these validations are called Assertions.

Let’s take a look at some of the simplest, but useful JMeter Assertions.

Types of Assertions

JMeter offers a vast range of Assertions:

Type Usage
Response Assertion Apply a string pattern to verify against the server response
Duration Assertion Check the response was received within a given elapsed time
Size Assertion Check the size of the server response contains the wanted number of bytes
XML Assertion Check the response is a valid XML document
Beanshell Assertion Execute your own logic using Beanshell scripting
MD5Hex Assertion Allows to check the MD5 hash of the response data (great for static files)
HTML Assertion Check html response syntax using JTidy
XPath Assertion Tests if a document is well-formed, with possible DTD validation, or putting the document through JTidy and testing with XPath
XML Shema Assertion Validate an XML response against an XML schema
JSR223 Assertion Run your own code logic using a JSR223 Script
Compare Assertion Compares results between themselves
SMIME Assertion Evaluate the sample results from the Mail Reader Sampler
JSON Assertion Execute JsonPath expressions and validate Json documents

For this post, we will focus on Response Assertion, Duration Assertion, and JSON Assertion.

Response Assertion

The Response Assertion performs content validations on the response, both on the data and meta-data.

Its goal is to detect bad responses generated during heavy server load and stress.

In the example below, we have defined a response validation for all Thread Groups (note the definition on the same level as the Thread Groups).

This Assertion will act (Apply to) to the first response (Main sample) and re-directs (sub-sample), if any.

The validation itself checks if the Response Code Equals to 200

Response Assertion

We can create as many All-Thread-Groups level Assertions as we want.

Below, we are checking if the Response Message Equals to OK.

Response Assertion

When we run again, all responses continue to be green, because they fullfil both criteria.

Response Assertion Results

Let’s change the first Assertion to an invalid value, 500 (Server Error code).

Response Assertion 2

Now we see the exact Assertion that failed and the diff between what was expected (comparison) and what we got (received).

Response Assertion Results 2

Duration Assertion

Duration Assertions investigate the timing the request.

Since this assertion is request specific, it only makes sense to have it under a Thread Group:

Duration Assertion

Duration Assertion

Duration Assertion allows to define the acceptable time threshold for the request, in our case, we defined 500ms for the Litecoin Orderbook endpoint and 1 second for the Litecoin Trades endpoint.

In case any request fails to meet this limit, JMeter will show a friendly error message:

Duration Assertion Results

JSON Assertion

JSON is an extension of the Response Assertion. It aims to facilitate the navigation in JSON responses, in order to do simply integrity verifications of the data.

In the Assert JSON Path exists, we set a JSON path and JMeter will throw an error if this does not exist.

In our case, we want to check that the asks key is composed of a 2-dimensional array: That’s why we entered on the first element of the first element of the array mapped to asks.

Additionally, we added a regular expression to match the element found on the path. In our case, I want to be sure we are talking about a decimal value (at least on numeric value, optionally followed by a dot ., followed by zero or more numeric values).

JSON Assertion

To demonstrate a failure, I’ve marked the “Invert assertion”, we raises a failure if the regex matches.

JSON Assertion

With these three assertions, you can investigate further the responses you get with JMeter.

You can see all posts of this JMeter series here.

And if you want to try out the resulting suite, you can download it here.

How would you implement the questions we asked at the beginning of this post?

  • If a resource does not exist, does the system return a 404 response fast enough?
  • If a server error occurs (500 code), does the system continues to work properly?

Leave a comment