Building a scalable test automation framework for your project
An INFJ personality wielding brevity in speech and writing.
Suppose you are relatively new to test automation and want to get involved in building a framework. In that case, it’s important to be aware of components that are necessary for the framework and how they fit together. It can be helpful to think about your project’s specific needs and goals, as well as any constraints or limitations you may be working within.
For example, if you’re a test architect, ask yourself these questions
And there could be some more.
In few cases, there are frameworks that never really does its job like the ones that does not make any difference even when you turned it off.
An automation framework is more than just the tools. It’s people and processes put together to set up a framework from the ground up and allow teams to deliver better software quality faster.
We’ll talk more about it in the sections below.
Test automation framework types
Some common types of automation testing frameworks include:
Linear Testing Framework: This is the most basic type of automation testing framework, where test cases are executed in a linear sequence, one after the other.
Linear testing frameworks can be useful for testing applications with a relatively simple or straightforward architecture. However, they may not be as effective for testing more complex or interconnected applications, as they may not consider the interactions and dependencies between different parts of the application.
Modular Testing Framework: In this type of framework, the application is divided into smaller modules, and each module is tested separately before integrating them to test the complete application.
The advantage of using a modular testing framework is that it allows testers to focus on smaller, more specific areas of the software, which can make the testing process more efficient and effective. It also allows for greater flexibility, as modules can be tested in any order and reused in different contexts.
However, a modular testing framework may not be as effective at identifying issues that arise when different modules are integrated, as these issues may not be apparent until the full system is tested. As such, it is often used in conjunction with other testing frameworks to provide a more comprehensive testing approach.
Data-Driven Testing Framework: In this type of framework, test cases are executed by reading the test data from external sources, such as an Excel sheet or a database.
Here is a simplified example of a data-driven testing framework:
- The test script contains the steps for the test case, such as opening a web page, entering login credentials, and clicking a button.
- The test data is stored in a separate file or database and contains the login credentials that will be used as input to the test script.
- The test runner is the component that executes the test script and passes the test data to it.
- The test script uses the test data as input and performs the steps in the test case. The output of the test script is compared to the expected results to determine if the test has passed or failed. The test results are usually stored in a separate file or database.
Using a data-driven testing framework allows you to create a single test script that can be executed multiple times with different data sets, improving the efficiency and flexibility of your testing process.
However, data-driven testing frameworks can be more complex to set up and maintain, as they require creating and maintaining a separate data file for test cases. They may also be less effective at identifying issues unrelated to data input and output, such as UI or performance issues.
Keyword-Driven Testing Framework: This type of framework uses a table of predefined keywords to represent the actions to be performed in the test cases.
In this approach, keywords must be connected to the objects or user interface elements they correspond to. This is typically done by creating a shared object repository that maps objects to their required actions. It’s worth noting that these keywords can then be reused across multiple test scripts.
As the test suite grows, the keyword library can become large and difficult to maintain. It is important to have a system in place to keep track of the keywords and ensure that they are up to date.
Hybrid Testing Framework: As the name suggests, this type of framework combines elements of multiple automation testing frameworks.
A hybrid test automation framework combines the features and benefits of multiple types of test frameworks in order to increase the efficiency and cost-effectiveness of quality assurance testing. By addressing the limitations and utilizing the strengths of various test frameworks, hybrid frameworks offer a comprehensive solution for meeting the growing demand for test automation.
The appropriate framework should be chosen based on the nature of the project, the size of the project, the type of application being tested, and the skills and experience of the testing team.
Which automation framework is best?
Businesses have made a switch to Hybrid framework for a long time now as it encompasses the benefits of all other types of frameworks. Most popular hybrid frameworks are:
Test NG:
TestNG is a Java-based testing framework that is designed to cover a wider range of test categories: unit, functional, end-to-end, integration, etc. It is an open-source tool that can be used to write and execute test cases, and it is particularly useful for testing in a team environment, as it provides support for parallel test execution and the generation of test reports.
To use TestNG, you will first need to install it in your Java project. This can typically be done using a build tool such as Maven or Gradle. Once TestNG is installed, you can write test cases using the TestNG annotations and API. These test cases can then be executed by running the TestNG test runner, which will discover and run the tests according to the configuration specified in a TestNG XML file or via the command line.
In this example, the @Test annotation marks methods as test cases. When the TestNG test runner is executed, it will discover and run these test methods.
TestNG also provides a number of other annotations that you can use to control the execution of your test cases, such as @BeforeMethod and @AfterMethod, which can be used to set up and tear down test data before and after each test method is run.
To run your TestNG tests, you can use the TestNG Eclipse plug-in, or you can run the TestNG test runner from the command line using the following command:
java org.testng.TestNG testng.xml
The testng.xml file is an XML configuration file that specifies the test suite to be run and any additional configuration options.
BDD:
BDD (Behavior-Driven Development) is a software development methodology that aims to bring together developers, testers, and non-technical stakeholders by encouraging collaboration and communication throughout the development process.
A BDD framework is a tool or set of tools that support the BDD process. These frameworks typically provide tools for writing and executing human-readable acceptance tests, which define the system’s behavior under test from the end user’s perspective.
One of the most popular BDD frameworks is Cucumber, a tool for writing and executing acceptance tests in a language called Gherkin. Gherkin tests are written in plain English (or the language of your choice) and use a simple syntax to describe the steps of a test scenario.
In this example, the “Feature” and “Scenario” lines define the high-level behaviour being tested, and the subsequent lines describe the individual steps of the test in plain English.
To use the Cucumber framework, you will need to install it in your project and write your acceptance tests in Gherkin. You will also need to write code to implement the steps defined in your acceptance tests, which can be done using a variety of programming languages.
Once your tests and step definitions are written, you can execute your acceptance tests by running the Cucumber test runner. The test runner will parse your Gherkin tests, execute the step definitions, and report on the results.
Steps to building an in-house test automation framework
Define your goals: Clearly articulate what you want to achieve with your test automation framework. Do you want to improve the speed of your testing? Increase the reliability of your tests? Enhance your team’s productivity? Identifying your goals will help you determine the features and capabilities that your framework should have.
Tip: Start with understanding the application structure and environment and kickstart a pilot project. Select a few things you want to automate and set goals at the end of this experiment. Once you achieve those goals, it’s a sign that your experimentations are over, and you can kickstart your actual project. Don’t set off with the aim to automate everything.
Identify the tools and technologies you will use: Choose the programming language, test runner, test management tool, and other tools that you will use to build and maintain your test automation framework. Consider factors such as the skills and experience of your team, the type of software you are testing, and your budget.
Tip: Mentor your team to accommodate the paradigm shift
Develop your framework: Create a high-level design for your test automation framework that outlines its overall architecture and the relationships between its various components. This should include a description of the different layers of your framework (e.g., UI, business logic, data access), as well as the interfaces that will be used to communicate between these layers.
Create reusable elements, such as helper functions, utility classes, and common assertion methods, that can be used across multiple test scripts.
Implement your framework: Develop a detailed plan for the framework’s organization, including its modular design, control flow, design pattern, and the technology stack to be implemented. Use the tools and technologies you have selected to build the various components of your test automation framework. This may involve writing code to implement the different layers of your framework and creating test scripts and test data.
Tip: It typically takes 4-6 months for skilled professionals to implement a framework, with the technical aspects comprising approximately 50% of the work required. Working with automated testing experts can shorten path to implementation and avoid common missteps.
Logging & reporting: Implement a logging system in the framework to record information such as the progress of test execution, test outcomes, and any error messages that arise during testing.
Integrate a reporting system in the framework to generate test execution reports with an overview of test outcomes, including details such as the number of passed/failed test cases, test execution time, and any errors that occurred.
Maintain and update your framework: As your software evolves and your testing needs change, you will need to maintain and update your test automation framework.
Making test automation framework more effective
Use a modular approach: Divide your tests into smaller, independent units that can be combined to form larger tests. This will make it easier to maintain and update your tests.
Evaluate the ease of automating tests for each service and analyze the benefits of automation for each service. Consider factors such as maintainability and the limitations of automation in this process.
Use a programming language that is easy to learn and has a large community of users, such as Python or Java. This will make it easier to find help and resources when you need them. Try to use a language that devs also use.
Consider using test automation frameworks that are specifically designed for your testing needs. For example, if you are testing a web application, you might want to use a framework like Selenium.
Utilize a test runner or test management tool to execute your tests. Tools like PyTest, JUnit, or TestNG can help you run your tests more efficiently and generate reports on the results.
Use a version control system like Git to track changes to your test code. This will make it easier to collaborate with others and roll back changes if necessary.
Implement continuous integration (CI) to automatically run your tests every time you make changes to your code. Set up smoke tests for every build and regression on a nightly basis. This will help you catch and fix issues early on, before they become bigger problems. Popular CI tools include: GitLab, CircleCI, Azure pipelines, TravisCI…
Use test data management tools to store and manage your test data. This will help you keep your tests organized and ensure that you are using the most up-to-date data.
Use a test reporting tool to generate clear, easy-to-understand reports on the results of your tests. It should be easy to:
- See exception stack traces
- Quickly use failed assertions and associated inputs/logs
- Good overview of pass/fail/skipped metrics
- Should have historical results comparison
Some available reporting frameworks include Gradle reports, Cucumber reports, Report portals, Allure.
Foundations of a scalable automation framework: Zuci view
We asked our SDET Manager, Dhanalakshmi Tamilarasan about what should be the foundation of a scalable test automation framework and here’s what she listed.
Interested in watching a video instead? Click below to watch
Define scope
Are you going to automate web application or mobile or API or database? Ensure the automation framework can handle all these applications under test.
Automating critical test scenarios: Is it going to be lengthy or small?
Application type: Is it going to be data-intensive? Do I need to execute test cases for multiple set of data?
Quiz yourself on these parameters and define your scope of testing and framework.
Code flexibility
How flexible your code is so that you can add/update any new code and how easily can I maintain the existing code
Continuous Integration
In today’s agile world, we get (n) number of builds and, automation scripts are to be run on a daily basis. So, look at how easy it is for your framework to integrate with CI/CD pipeline.
Cross-platform testing
To carry out responsive testing by automating web and mobile apps that supports a combination of browsers/OS/devices, your automation framework should be able to integrate with cloud-based platforms like Sauce Labs, Browser Stack.
Test Management
Test management tools like JIRA and Test rail are important for tracking testing activities, analysing results, etc. So, your framework should allow you to automatically update test results without a manual support/intervention.
Parallel execution
An automation framework should support parallel execution of test suites so that test results are available quickly. Quick test results mean quick detection of defects.
Localization support
Today, most apps are built with localization to give seamless experience for users across the globe. We need to ensure that the framework is capable of handling different languages.
Test reports
Automation framework should have provisions for providing test reports from different test reporting tools so that it becomes for the test management team.
Final thoughts
Quite often when there’s a team restructuring or when someone new joins the team, there is this urge to rewrite the code base and build a framework from scratch.
And quite often, it’s the copy paste code base from the previous company without understanding the current team or company’s context. If you find yourself doing that, pause.
The best way to approach this is- context-driven, as simple as it sounds. You should talk to people in your team and have them walk through all the processes, tools, versions, CRMs and all of that.
Consider doing what’s best for your team and project while keeping in mind the underlying principles of automation and frameworks.
If testing is a means to the end of understanding the quality of the software, automation is just a means to a means.– James Bach
Looking to improve your test automation coverage? Take a look at Zuci’s test automation services and see how you can leverage Zuci for your business needs.
Related Posts
