Test Automation Strategy - Guide

Test Automation is increasingly being introduced to companies to solve issues and reduce the time to market for products. However, for test automation efforts to not be ad hoc and to get the return on investment that test automation brings, the team should work together to create a Test Automation Strategy.

A test automation strategy defines the tools and frameworks used to get reusable, automated scripts, as well as how to create those scripts and maintain them. It is most effective when it is understood by your entire team as they are going to be the ones to implement it. Test automation can provide valuable returns on investment, such as faster time to market, more reliable, faster tests. However, it takes a lot of time and effort and you don’t want to dive into writing automated scripts without first having a test automation strategy.

Your strategy should define how you’re going to go about doing automation, who is going to be writing it, when are you going to be running it and what environments you’re going to use. It fully defines the what, why, when, how and where of your test automation initiative. Let’s break down the what, why, when, how and where that will be in your test automation strategy.

The Why/Goal

When creating your test automation strategy, the first thing you should identify is the why. Why do you want to have automated tests? What is the goal that you’re trying to achieve by creating automated scripts? There are several goals that you may have in mind. You may want to have a faster time to market for your product. This can be achieved by having test automation in your process.

Other goals for test automation could be:

1. Having test automation created for features in the same sprint that they are developed.
2. Reduce manual testing. If there are a lot of regression tests that take a long time to execute, introducing automated tests will reduce the time spent executing the regression suite
3. Improve test coverage of the software. This can be done through automating a lot of unique scenarios and can be done easier and faster if we use a data-driven approach
4. Transition from waterfall to agile and incorporate continuous testing into software delivery

These along with multiple other reasons are your goal for test automation. This goal is going to help to identify the tools, techniques, members of the team and technologies that you will use. This goal is going to drive everything else that you include in your test automation strategy.

Who

Test automation scripts do not write themselves. You will need a team to write automated scripts, execute them, check for failures, update those scripts, create reports, maintain them, write regression tests, et cetera. All of these items are very important and you need a dedicated team to do this. Having a proper mix of manual testers and automated engineers to develop a balance between what is being automated and what is still being tested in each sprint will prevent your team from feeling overwhelmed and unable to achieve the goal that you have set out for doing test automation.

In addition to thinking about who will be on your team you need to think about:

1. Is training needed for members of your team and what do they need to be trained on? If you are transitioning manual testers to start doing test automation then they will need to know a programming language, locator strategies, coding best practices such as page object patterns and atomic tests along with the particular test automation tools and frameworks that you will choose.
2. Do you have the right persons on staff or do you need to hire? Start thinking about what that job description will be. The skills, knowledge, techniques and technologies that they will need to know and have.
3. Support for test automation. Will you create specific tickets for test automation? Will they be included in the different sprints? Do you have a project manager to help with this? Is the test automation effort supported by the entire organization so that setting up environments and creating applications more favourable to test automation will not be difficult?

Having the right set of persons with the needed knowledge and skills will positively impact the test automation strategy.

What

Deciding what to automate allows you to zone in on key areas that will give the best return on investment. Diving straight in without a plan on what you are going to automate and at what level of the test automation pyramid can result in a huge waste of time and other resources. When determining what to automate, think of the goal and ensure that you align what you are going to be automating with meeting that goal. For example, if your goal was that you wanted to automate more regression tests, then you are going to prioritize and automate the regression suite/features that already exist in production. If it is that you want to automate new features in the sprint, then you’re going to look at those new features and try to identify which ones can be automated. I generally have a rule of thumb regarding what you should automate. Here is the high-level overview:

1. Decide the automated test level at which you will be creating your tests. Automated Tests can be placed into three main levels: Unit Tests, Service/API Layer Tests, and User Interface Tests; you should evaluate the pros and cons of writing tests at each of these different levels before proceeding.
2. Automate your smoke tests.
3. Automate tests that are always run, such as regression tests or tests always found at the beginning of every workflow.
4. Automate extensive tests
5. Automate tests that require multiple configurations
6. Automate your performance tests.

These automated test cases will need to be designed to be reusable, maintainable, concise, easy to understand and prioritized to ensure a smooth process of creating and maintaining the test automation effort.

How

The next question that you should answer is how you’re going to go about running your automated tests. You need to find specific tools and frameworks to fit your project needs. In determining which tools you will need to use consider:

1. The level of test automation that you will be doing. If you are doing API automation vs Database or browser or Visual regression automated testing there are different tools and technologies that will be needed for each.
2. The language that will be used for test automation. Tools such as Selenium Webdriver can be used in several different languages such as Java, Python, JavaScript, C# etc. Selenium Webdriver does browser automation as well as mobile automation with Appium. In addition to Selenium Webdriver, there are other tools for different languages such as WebdriverIO and Cypress for Javascript browser automation. However, if you are going to be writing API or Database tests then there are different tools based on the language. RestAssured can be used for API automation in Java and Supertest can be used for Javascript.
3. Scenarios that will be tested. Consider unique scenarios or edge cases that you will want to be automated. If cross-browser testing is important to you then you may consider tools such as Saucelabs or Browserstack.
4. The goal for test automation. If it is that you have continuous integration as a goal, then you will need to start thinking about continuous integration systems, such as Jenkins, GitLab, GitHub Actions or some other CI tool. Think about how automated tests will be triggered (manually on your computer or via the continuous integration system).
5. Cost of the tool. There are paid tools and those that are open source. Based on the budget that you have you will need to weigh the cost and the support that you will receive for the tools that you need.

To evaluate and select the right tools and frameworks, create a table of all the features that will be automated as well as must-haves for the different tools and compare each to see which one will work best for you. I would also suggest that you do a Proof of Concept(POC) with the top 2 tools to ensure that the persons that will be using it are comfortable with it, they can create basic as well as edge case test automation and the tools are maintainable.

This stage is probably going to take the most time and research and you must take the time here to carefully evaluate each tool and the need that you have for the test automation project.

Where and when

The final thing that you need to consider for a solid test automation strategy is where and when. I strongly suggest that you have a dedicated test environment that you run your automated tests on. This allows you to manipulate the data so that you can get different scenarios that you would need to verify. You need to think about the work involved in getting this dedicated environment and have a plan to get it implemented.

I also suggest that you run tests on production. However, those tests should be properly marked or tagged. You want to tag your tests that should run in production so that you can control those specific tests. For example, you may not want to have the entire checkout flow running in production or make certain delete requests from your API or your database.

Have an execution plan for running your tests. Answer these questions and make a plan:

• Do you need to set up specific data for your test, make database connections or API calls?
• Does this data need to be masked?
• How do you tear down and set up your automated tests?
• Are you running these tests locally or on the cloud?
• Are you using a Continuous Integration system?
• Are there specific times that you want the tests to run, is there a particular trigger for the automated tests to execute?
• What repository will you be using to store code?
• What is the process for merging code to the repository?
• What will you be using to report automated test runs?

Ensure that you consider all these details so that you will have a solid test automation strategy.

_ _

Keep in mind that your team should understand the complete test strategy.

I would suggest that you use a mind map to start drafting out what your strategy will include.

This can be a living document that you and your team go through to update and talk about different points, regarding the tools, technologies, types of tests that are going to be run and the goal of the test automation project. Remember that automated tests should be maintainable and that will take time. Ensure that you give your team enough time to prioritize areas that should be automated, create scripts, run and evaluate them, create reports, maintain the project and have the adequate knowledge to carry out all of these activities. Therefore, if your team needs training, provide the relevant training and support.