A brief guide to testing in DevOps

What is Testing in DevOps?

DevOps – as the term notes, is the practice that encourages partnership between development and IT operation teams to suit the aspirations of continuous integration and continuous deployment (CI/CD) and modern software delivery. However, this union of two teams is only half-done; to be able to reap the full benefits of DevOps, testing also needs to be tagged along in the continuous delivery chain.

So, what do we mean by testing in DevOps? Fundamentally, testing in DevOps aims to strike a right balance within the overlap of goals, feedback cycles, skills among the development, QA and the operations team. It promotes the collaboration of all three teams while automating the process of software delivery and infrastructure changes.

This article walks you through:

• How will testing be done in DevOps?
• Automated testing – the driver of DevOps
• Setting up test automation
• Getting QA started with DevOps
• DevOps testing strategy
• DevOps testing best practices
• DevOps testing tools

How will Testing be done in DevOps?

Testing in DevOps means that testing will no longer be happening in siloes. This implies, testing will happen in tandem with development, fostering for a quicker feedback cycle so that each team can stay in the know about how their work impacts rest of the delivery cycle.

As a result, we see a great emphasis on product quality is created throughout the DevOps cycle.

Dan Ashby, co-founder, of Software Testing Clinic, illustrates testing in DevOps as follows,

Having integrated agile methodology into teams with agile manifesto, testing in DevOps team has formulated guiding principles for testing software thoroughly and at scale. These came to be known as the “DevTestOps” manifesto.

Continuous testing over testing at the end

Embracing all testing activities over automating everything

Testing what gives value, based on customer usage, over testing everything

A whole team approach to testing over testing in siloed testing departments

Product coverage over code coverage

Leveraging the above DevTestOps practices ensure teams that regular deployments take place without much interruptions by providing greater visibility into the state of software development and help them achieve the desired product quality.

We’ll see how we can integrate testing in DevOps, plan for a fine DevOps testing strategy, best practices, and tools in the sections below.

Do you know?

Automated testing – the driver of DevOps

DZone research findings on test automation revealed that,

Reasons why DevOps need automated testing:

• Nearly impossible to continuously test the code and code fixes at the rate at which it is deployed
• Trailing of QA teams in the continuous software delivery chain
• Inconsistent test practices & setting up of one-off QA teams weaken the quality of the software being built
• Longer and mundane feedback cycles forcing development and QA to push the code down to production to achieve speed

Incorporating automated testing is the sure-shot way of integrating QA into DevOps workflow fruitfully. Without optimal automated testing in place, our pursuit of streamlining processes across the CI/CD pipeline for effective DevOps will be a distant dream. In this sense, automated testing acts as a driver of DevOps.

Setting up test automation

As we discussed earlier, automated testing is a crucial factor behind this entire DevTestOps phenomenon. Running big volumes of nightly regression suites every night or on weekends manually simply undercuts the goal of DevOps.

Let’s dive into the setting up of test automation.

Test automation skills are hard to acquire as it’s technically complicated. DevOps will continue to be a distant goal for companies that are not there yet or in the initial stages of adopting test automation in their delivery chain.

For successful DevTestOps implementation, a test automation engineer needs to have a good understanding of the functionalities of the software, inherent technical details, choice of automation and CI/CD tools, scripting language, automation framework design, reporting needs, ability to set up continuous testing needs and competence to develop test scripts in parallel with development.

Imagining you’re figuring out test automation for your company, there are 4 action items you can consider:

1. Have a champion internally or from the outside consulting firm, who understands the goals and has a plan to achieve them.
2. Pick someone who has experience in test automation but more of broad expertise in software development in general.
3. Engineering savvy champion- these champions have a clear vision, gather experiences from multiple sources, and enforce the right technical approach, processes, and guidelines that’ll significantly improve quality.
4. A right balance between manual and automated testing can actually help gain better returns.

In typical DevTestOps culture, all the teams will work together to streamline processes that promote in-built product quality in line with the culture.

All of these activities set up for a continuous integration, continuous testing, continuous feedback processes, helping teams build better software faster, and making the journey of DevOps worthwhile.

DevOps testing strategy

Generating an optimal DevOps testing strategy requires a detailed assessment of the software and a mix of other elements to plan for a streamlined testing structure.

Let’s check out the focus areas for a successful DevOps testing strategy.

Automate user behavior, not tests

One should aim to prevent defects arising in the production with the help of automated tests. One way to achieve this and contain defects earlier is by deploying automation frameworks and layer it with a Test-Driven Development (TDD) /Behavior Driven Development (BDD), which focuses on automating user behavior and not just tests. Following this approach ensures greater testability, maximum code coverage, and active collaboration among teams.

Example of automating user behavior using Gherkin

Optimize test environment

In addition to writing the right automation test scripts, we should plan to optimize a test environment that’ll encompass user conditions, relevant test data, and the capability to cope with varying peak times, product requirements, etc. to fit into the entire DevOps strategy.

These will help to mitigate quality issues much earlier in the process.

Analyze critical test data

It is evident that with the running of large volumes of test suites, the amount of test data generated will also be enormous. It becomes significantly necessary to check and analyze these test data to gain visibility into the state of quality of the software, discover the CI pipeline’s health, and identify test anomalies that help steady the continuous tests helping the software release cycles.

Measure continuously

To understand where we stand in DevOps maturity, we should continuously measure KPIs that contribute to the software’s quality,

A few of them include,

• Identify defects by phase, stage, cause, functionality, and trend
• Percentage of defects escaped to production
• Pinpoint manual vs. automated testing defects
• Trace flaky test trend

DevOps testing best practices

Here are some general best practices for implementing an efficient testing process into the DevOps structure

Smart automation

We have already addressed in detail about the significance of automation in DevOps; however, this doesn’t mean automated testing can alone help every time. The key to employing successful automation is to find automatable tests first.

Few tests like usability, post-deployment tests are better off when tested manually.

Agile iteration

Focus on rapid and quick iterations among each team leads to our goal of continuous delivery. A sequence of smaller and agile developments enables teams to test thoroughly and push for quicker deployments. These iterations give insight into the changes being made and the ability to ensure deployments happen more often and more early.

DevOps metrics and documentation

As the name suggests, DevOps is all about collaboration among teams. As we get started with it, there needs to be clear and defined metrics and documentation. It provides information about the processes and changes that are taking place in the system.

Additionally, it enforces transparency, and without proper documentation in place, it’s very difficult to review and understand what worked and what hasn’t in your DevOps structure. Also, it forms a base to enable decision-making for the future.

DevOps testing tools

As DevOps is built on top of the agile focused on the usual software testing life cycle and not an out of the box testing altogether, tools used for the usual testing process can be leveraged for testing in DevOps too.

Here are some of the commonly used testing tools,

• Selenium
• Jenkins
• Appium
• Cucumber
• TestNG
• Maven
• Appium
• Bamboo

Final thoughts

Though there are a plethora of resources available to guide you on the testing in DevOps, successful implementation and reaping full benefits of DevOps depend on the organization’s test maturity, and its external consulting/test partner as no one size fits all.

Related read/watch:

• What is DevOps testing?
• Adopting DevOps