A brief guide to testing in DevOps
An INFJ personality wielding brevity in speech and writing.
What is Testing in DevOps?
DevOps – as the term notes, is the practice that encourages partnership between development and IT operation teams to suit the aspirations of continuous integration and continuous deployment (CI/CD) and modern software delivery. However, this union of two teams is only half-done; to be able to reap the full benefits of DevOps, testing also needs to be tagged along in the continuous delivery chain.
So, what do we mean by testing in DevOps? Fundamentally, testing in DevOps aims to strike a right balance within the overlap of goals, feedback cycles, skills among the development, QA and the operations team. It promotes the collaboration of all three teams while automating the process of software delivery and infrastructure changes.
This article walks you through:
How will Testing be done in DevOps?
Testing in DevOps means that testing will no longer be happening in siloes. This implies, testing will happen in tandem with development, fostering for a quicker feedback cycle so that each team can stay in the know about how their work impacts rest of the delivery cycle.
As a result, we see a great emphasis on product quality is created throughout the DevOps cycle.
Dan Ashby, co-founder, of Software Testing Clinic, illustrates testing in DevOps as follows,
Having integrated agile methodology into teams with agile manifesto, testing in DevOps team has formulated guiding principles for testing software thoroughly and at scale. These came to be known as the “DevTestOps” manifesto.
Continuous testing over testing at the end
Embracing all testing activities over automating everything
Testing what gives value, based on customer usage, over testing everything
A whole team approach to testing over testing in siloed testing departments
Product coverage over code coverage
Leveraging the above DevTestOps practices ensure teams that regular deployments take place without much interruptions by providing greater visibility into the state of software development and help them achieve the desired product quality.
We’ll see how we can integrate testing in DevOps, plan for a fine DevOps testing strategy, best practices, and tools in the sections below.
Do you know?
|
Automated testing – the driver of DevOps
DZone research findings on test automation revealed that,
|
Reasons why DevOps need automated testing:
- Nearly impossible to continuously test the code and code fixes at the rate at which it is deployed
- Trailing of QA teams in the continuous software delivery chain
- Inconsistent test practices & setting up of one-off QA teams weaken the quality of the software being built
- Longer and mundane feedback cycles forcing development and QA to push the code down to production to achieve speed
Incorporating automated testing is the sure-shot way of integrating QA into DevOps workflow fruitfully. Without optimal automated testing in place, our pursuit of streamlining processes across the CI/CD pipeline for effective DevOps will be a distant dream. In this sense, automated testing acts as a driver of DevOps.
Setting up test automation
As we discussed earlier, automated testing is a crucial factor behind this entire DevTestOps phenomenon. Running big volumes of nightly regression suites every night or on weekends manually simply undercuts the goal of DevOps.
Let’s dive into the setting up of test automation.
Test automation skills are hard to acquire as it’s technically complicated. DevOps will continue to be a distant goal for companies that are not there yet or in the initial stages of adopting test automation in their delivery chain.
For successful DevTestOps implementation, a test automation engineer needs to have a good understanding of the functionalities of the software, inherent technical details, choice of automation and CI/CD tools, scripting language, automation framework design, reporting needs, ability to set up continuous testing needs and competence to develop test scripts in parallel with development.
Imagining you’re figuring out test automation for your company, there are 4 action items you can consider:
- Have a champion internally or from the outside consulting firm, who understands the goals and has a plan to achieve them.
- Pick someone who has experience in test automation but more of broad expertise in software development in general.
- Engineering savvy champion- these champions have a clear vision, gather experiences from multiple sources, and enforce the right technical approach, processes, and guidelines that’ll significantly improve quality.
- A right balance between manual and automated testing can actually help gain better returns.
In typical DevTestOps culture, all the teams will work together to streamline processes that promote in-built product quality in line with the culture.
Watch the video on “Test Automation: An enigma that continues to haunt” by our President, Vasudevan Swaminathan.
Getting QA started with DevOps
If you are attempting to get QA started with DevOps in your organization but don’t know-how, you can jump-start by measuring these 3 metrics
| Defect Containment | Log maximum of 95% defects in QA, UAT & Production |
|---|---|
| Team Productivity | Keep the backlog reworks to a maximum of 10% only |
| Lead Time | Aim to push code to production every day |
All of these activities set up for a continuous integration, continuous testing, continuous feedback processes, helping teams build better software faster, and making the journey of DevOps worthwhile.
DevOps testing strategy
Generating an optimal DevOps testing strategy requires a detailed assessment of the software and a mix of other elements to plan for a streamlined testing structure.
Let’s check out the focus areas for a successful DevOps testing strategy.
Automate user behavior, not tests
One should aim to prevent defects arising in the production with the help of automated tests. One way to achieve this and contain defects earlier is by deploying automation frameworks and layer it with a Test-Driven Development (TDD) /Behavior Driven Development (BDD), which focuses on automating user behavior and not just tests. Following this approach ensures greater testability, maximum code coverage, and active collaboration among teams.
Example of automating user behavior using Gherkin
Optimize test environment
In addition to writing the right automation test scripts, we should plan to optimize a test environment that’ll encompass user conditions, relevant test data, and the capability to cope with varying peak times, product requirements, etc. to fit into the entire DevOps strategy.
These will help to mitigate quality issues much earlier in the process.
Analyze critical test data
It is evident that with the running of large volumes of test suites, the amount of test data generated will also be enormous. It becomes significantly necessary to check and analyze these test data to gain visibility into the state of quality of the software, discover the CI pipeline’s health, and identify test anomalies that help steady the continuous tests helping the software release cycles.
Measure continuously
To understand where we stand in DevOps maturity, we should continuously measure KPIs that contribute to the software’s quality,
A few of them include,
- Identify defects by phase, stage, cause, functionality, and trend
- Percentage of defects escaped to production
- Pinpoint manual vs. automated testing defects
- Trace flaky test trend
DevOps testing best practices
Here are some general best practices for implementing an efficient testing process into the DevOps structure
Smart automation
We have already addressed in detail about the significance of automation in DevOps; however, this doesn’t mean automated testing can alone help every time. The key to employing successful automation is to find automatable tests first.
Few tests like usability, post-deployment tests are better off when tested manually.
Agile iteration
Focus on rapid and quick iterations among each team leads to our goal of continuous delivery. A sequence of smaller and agile developments enables teams to test thoroughly and push for quicker deployments. These iterations give insight into the changes being made and the ability to ensure deployments happen more often and more early.
DevOps metrics and documentation
As the name suggests, DevOps is all about collaboration among teams. As we get started with it, there needs to be clear and defined metrics and documentation. It provides information about the processes and changes that are taking place in the system.
Additionally, it enforces transparency, and without proper documentation in place, it’s very difficult to review and understand what worked and what hasn’t in your DevOps structure. Also, it forms a base to enable decision-making for the future.
DevOps testing tools
As DevOps is built on top of the agile focused on the usual software testing life cycle and not an out of the box testing altogether, tools used for the usual testing process can be leveraged for testing in DevOps too.
Here are some of the commonly used testing tools,
- Selenium
- Jenkins
- Appium
- Cucumber
- TestNG
- Maven
- Appium
- Bamboo
Final thoughts
Though there are a plethora of resources available to guide you on the testing in DevOps, successful implementation and reaping full benefits of DevOps depend on the organization’s test maturity, and its external consulting/test partner as no one size fits all.
Related read/watch:
Related Posts
