5 steps creating a hands-on Test Automation Strategy
Are you looking to kickstart your software development teams automation efforts and make a solid move towards a more modern and mature way of working? Keep reading. When it comes to software development automation I believe that test is a great actionable entry point for change, and a good test automation strategy will pave the way towards more robust flexible ways to add and focus on real business value. With a high degree of engineering maturity your teams and the organizations abilities will quickly excel towards defect avoidance, rather that the old boring traditional slow ways of defect finding. If done right your test automation effort can add significant business value over time without being overly complex. Here is a guide to hopefully get you started on that journey.
One word of advice befor jumping in. Remember to keep it simple. One of the major reasons test automation fails, in my opinion, is that we do not dare to try things and be wrong. Fail fast and restart. MVP and prototype simple ideas, fast. The results of not trying at all are devastating. It doesn’t have to be perfect, just get some stuff going, experiment, see where it takes you. That, and focus on business value. Good luck!
The 5 steps to get your team on the right track
Set tangible goals and make them measurable
Make the test automation pyramid work for you
What to test and why
Importance of speed and accuracy
Culture & Ownership
1. Set goals and make them measurable
Test Automation is a development effort, you need to set goals for it in the same way you need goals for your team deliveries. The goals and KPIs will help you to prioritise and follow up on progress. Here are some questions to get you going.
Which problems, short term, are you trying to solve with test automation?
Which of your current test areas/cases do you want to automate?
In which area do you see the quickest return of investment?
What kind of turnaround times do your team require when is comes to testing?
How will your test automation development fit in with your existing processes and workflows?
Do you have the required skills (and tools) in our team today? or is there a need to hire/reskill/upskill.
Be aware that your specific area might have other significant aspects impacting Test Automation KPIs, make it a team effort to ellaborate on those. Make your goals quantifiable and make sure they measure success.
Additional: Stay away from KPIs measuring number of test cases in relation to your manual tests. This is a weak statistic assuming manual and automation testing is the same thing. Instead focus on business value and key business processes to determine progress.
2. Make the test automation pyramid work for you
In order to automate on the appropriate levels and as early as possible, you can use the test automation pyramid to increase focus on doing things right. The pyramids main layers are, Unit, API & UI which represent different ways to interact with your application.
The base of the pyramid, Unit tests, represents the fastest, least expensive and most stabile tests. This should be where we put most of our efforts aiming for high test/code coverage.
In the middle of the pyramid you typically find API tests, or service level tests. And at the top, UI tests, the narrow part of the pyramid. The UI tests represents test on the GUI or presentation layer.
*Cred for the shamelessly stolen picture goes to hackernoon.com
How do you leverage this in your strategy? Push to create as many tests as possible at the base of the pyramid safeguarding the basics with fast, reliable and cost effective testing. Smaller, more precise, fast tests will enable your teams speed and agility going forward.
My advice is to create your own version of the test automation pyramid. Use the one in the picture or Google another one. Make sure to find the levels that apply to your tech stack and adjust the entry points. As long as you keep building your test automation from the bottom up, keeping the large parts of the effort at the base, you will succeed.
Additional: Typically you cannot automate all of your testing. Your exploratory tests for example, where the craftmanship of testing plays a big part, also UX tests can be hard to automate efficiently. Do consider keeping these test areas manual for the time beeing. When your team and organization have reached a higher engineering maturity you should be able to test/observe these and other aspects of the delivery in production, together with your customers and their data.
3. What to automate and why
Here is a set of statements related to test automation that will help your team to prioritise and get the conversation started. Keep in mind to follow the general advice in the test automation pyramid through out, Unit before API, API before UI… Bottom up!
Do automate
Repetitive monotounous functional tests that require a lot of manual work today
Tests of all new development on unit level
High risk / high value use cases
Features that require large sets of data to test
Features or dependencies that you need alarms on if they break
Load/Volume testing of isolated components to monitor trends
Do not automate (typically)
Exploratory tests, that requires extended knowledge of the system
Tests requiring human intervention, we need the speed
GUI level tests on unstable early designs
Frequently changing interfaces
One time tasks or edge-cases
Test that have unpredicable results
All software development teams face different challenges which makes the statements more or less valid. It’s difficult to see every angle, please DM me if you are sitting on additional ideas/statements and perhaps I’ll add those too. 🙂
4. Why speed and accuracy is important
One of the key aspects in getting CI/CD up to speed is automation. To enable automation and built in quality long term we need our strategy to support and drive this change. There are three key aspects making CI/CD work that are universal.
Simplification – Fewer steps, each step as simple as possible
Automation – Automate each part of the process, wherever possible and eliminate wait states
Standardization – Each step is the same every time, regardless of the magnitude of the change
Used as an example, the picture below visualises a generic delivery pipeline (wikipedia). A change to the top left, triggers a chain of automated events aimed at creating a new release. Note that each test layer feeds back information to the entity triggering the change. That means once a change has been committed someone is waiting for feedback on that change. A team member will wait if the tests run fairly quick (< few minutes) and will act on clear indications of failure, including it’s origin. This not only requires speed in test automation execution buy also accuracy of test cases.
CI/CD pipelines differ drastically across different tech stacks when it comes to speed. Your aim is to adapt your strategy to your current environment and improve both speed and accurancy over time. Accuracy is typically adressed by keeping test cases simple. Single purpose test cases and not complex chains that are indeterministic. Make sure you compare to similar teams if possible and measure over time to see trends.
Additional: Good practice is to add speed(runtime) for each automation layer as a team KPI to continuously keep track of progress and keeping the test automation lean. Also keep in mind that the overarching product team purpose is to deliver business value continuously to your customers, that can end up meaning multiple deploys to production every day, adapt your test automation speed and accuracy levels to that fact from day 1.
5. Culture & Ownership
“Culture eats strategy for breakfast” – Peter Drucker
I see less of this now but want to mention it anyways, I know some of you are struggling with change towards more test automation in your contexts. Owners that are focusing on features rather than long term enabling development like this. If you join hackathons, hack fridays or similar you know what to do. If not, go stealth, this is in all fairness not rocket science. You will be able to push something valuable in no time. Don’t ask for permission, ask for forgiveness.
Test automation is no longer a nice to have, its a must have and it will lead to shorter time to market and a safer more agile environment for the engineers. As soon as the team knows how automation makes them awesome, things really start to happen. You need one break, that is all it takes!
Ownership
In order for any strategy to succeed it needs an active owner. Being able to deliver on it as a team requires clear ownership and accountability. The first goal is to make test automation KPIs a team priority. Everyone in the team should have good awareness on how test automation is (or will be) implemented in your delivery pipelines and what is required by them as individuals. The team will learn how to avoid defects rather than finding them as engineering practices mature.
Find or be the owner! Owning test automation in a team means to define, communicate and closely follow up on the performance and goals of related KPIs.
Coding + Testing = Development
To round things up. Test automation development is an integrated part of application development. Your product team might have other quality gateways between yourselves and a live production environment, but that doesn’t change the fact that the team specific tests should be developed internally, in the team, by the team. Test automation should preferably be implemented in the same programming language in a closely related tech stack to get developers up to speed quickly.
Try this at home!
Thank you!
Great work reading all the way here, please do not hesitate to DM/email me if you have questions or would like to make additions to this guide!
gustav.lindberg@qbox.se