Running Non-Functional Tests In Continuous Testing Mode – Part 2
An INFJ personality wielding brevity in speech and writing.
Before we go any further, let’s understand what Continuous Testing is.
What is Continuous Testing
During the past few years majority of the software projects use Continuous Delivery, a software engineering approach in which teams produce software in short cycles, ensuring that software can be reliably released at any time.
Apparently, to ensure high quality and reliability, such projects deploy Continuous Testing, which involves running many different types of tests (automated and manual) throughout the software delivery process to repeatedly validate and improve the quality of the software that is being built.
The below quadrant shows the different types of tests and the approach (manual, test automation) to run them:
Source: https://continuousdelivery.com/foundations/test-automation/
For the purpose of this blog, we’ll restrict our discussion to non-functional tests.
As we can see in the quadrant above, the recommended approach to functional acceptance tests is to automate them fully and most projects have successfully done that. However, non-functional tests such as Security and Performance are still considered to be executed post “Code Freeze” and are not run in Continuous Testing mode.
Since the objective behind building software in Continuous Delivery mode is to keep the code ready for deployment to production at any time, non-functional tests should be treated similar to functional tests and they should be “included early and tested often” to make sure the deployed code reveals not just functional issues but also performance, security and other non-functional issues.
How to setup Non-Functional Tests in Continuous Testing mode
The two major categories of non-functional testing are Performance Testing (includes Load, Stress etc.) and Security Testing. There are others such as Usability Testing which is also considered non-functional but we restrict our focus to the former two.
One of the major challenges in designing non-functional tests is usually the requirements for non-functional aspects of the software are not as straight forward when compared to the functional behavior of the software. This combined with other factors such as time, budget prevent non-functional tests from being executed in continuous testing mode.
However, with Continuous Delivery approach requirements are mostly stated in a ubiquitous format using Behavior Driven Development (BDD) and other similar practices where both functional and non-functional requirements are specified in feature files using a language type called Gherkin, which is as much as saying it in plain English. For example, look at this security testing requirement from https://devops.com using BDD:
Here’s another example of a Performance test requirement using Gherkin format:
Given there are 100,000 users registered on the system
When I create a new account
Then I should be taken to my dashboard within 5 milliseconds
These are some examples of non-functional requirements that can be identified early and incorporated in the Continuous Delivery pipeline, automated and run as Continuous Tests. However all requirements are not as simple as we see here and challenge us in running them as continuous tests. What kind of challenges are these?
(To be continued …)
Read more Running Non-Functional Tests in Continuous Testing Mode – Part 3
Running Non-Functional Tests in Continuous Testing Mode
Similar posts
Related Posts
