The topic of this article is the most frequently asked questions throughout Keytorc’s software testing consulting and outsourcing projects. Our answer to this question is another question “How to Automate”.
Some of our clients perceive test automation as a silver bullet to solve their testing efforts and some of them perceive it as a nightmare. Why there is a huge gap between two different segments? Again, the answer lies between the lines of our second question “How to automate”. This question leads us to a paradigm shift where we have to consider our test processes other than just to focus on tool’s capabilities. This means that if the organization does not have a structured testing process, it has a chaos and automation of chaos gives more chaos faster. The good practices we have experienced in our projects have proved that behind every successful test automation tool implementation there are
- Well defined, structured test processes
- Know-how of different test types, test techniques
- And a high qualified test team who can implement these processes, test types and techniques with the help of test automation tools
The missing of any of above will lead to a chaos more than a success. But there is a sensitive balance between test automation and structured test process as in the below graphic:
Without structured test process we miss effectiveness where we apply useless tests very fast in a chaotic way. Vice versa without test automation we miss efficiency where we run out of time and budget. Only with a good balance between test automation and structured test process, we can achieve good job with reasonable costs. This balance depends on the nature of the project but in general we have experienced a 30% to 40% interval where tests can be automated. Above this interval will cause redundancy and repetition in testing efforts. If we agree on the importance of structured test processes, we can now move onto a next step: how to select the right tool for our processes? Here is the answer:
- Identify well the requirements we want to satisfy with the help of the tool ( e.g you may not need to buy a performance test tool with thousand fake users if you never experience a load of thousand real users). Requirements should draw our scope
- Market research: benchmarking among different tool alternatives ( you never know whether an open source tool can satisfy all of your needs, give a try)
- Compile a short list with costs and benefits
- Formally evaluate the tools in your short list (evaluation copy, pilot project etc.)
- Based on your evaluation, select the best suitable tool for your needs.
Well evaluation, well implementation and appropriate usage will:
- Achieve shorter software development life cycles with the synchronization of development and testing processes
- Enable (better) regression tests
- Prevent manual repetition errors
- Metric based, objective assessments
- “Special tests” such as performance and load
- Easier project tracking
- Save time and effort
Otherwise your silver bullet may bounce off of the wall and become your nightmare. But never forget the below risks that may show up in every test tool implementation project:
- Unrealistic expectations
- Underestimating time and effort for tool introduction and maintenance
- Over-reliance on the tool results
- One size fits all mistake
As it is the case in all tools we are using in our modern life, test automation tools should be treated as a way of doing our work more efficient. Never forget: it is a tool not a magician.
Koray Yitmen