THE VALUE OF AUTOMATED TESTING
Some clients ask why their application needs to be covered with automated tests. The basic truth is that if you want to ensure quality on your application it should be properly tested and both manual and automated tests are essential when you want to achieve that goal. However, some customers are not convinced to pay for automated tests. They sometimes think that if their application is relatively small, they will be able to test it manually and, therefore, automated testing is a waste of money.
For a while, they may be right. It is relatively easy to correct bugs in an application when the code base is small. Especially when there is only one or two developers working on the project. At a later stage, when the complexity of the application grows, controlling errors becomes ever more difficult. Not only because a greater code base translates into a greater number of bugs. In order to fix en error, someone has to spot it first. When the application expands, it is harder to identify the bugs that arise not only in the part of the system the developer is actually working on but also in the parts of the system they are not working on. The time needed for manual testing grows exponentially. The system reaches the size where it is simply impossible to ensure reliability merely with manual tests. It is only then that some realise they have incurred substantial technical debt that will have to be repaid. And the interest rate on the debt is painfully high.
HOW MANUAL TESTING WORKS?
Let’s assume there are no automatic tests implemented on the system you are working on as a tester. Imagine you have to check – manually – if you can create an account for a new user on an app under development. Like in Facebook, you specify the first name, last name, email, email confirmation, password, date of birth and sex. In order to test this form you have to enter correct data in all the text fields and select the proper option from every select box in the form. Finally, you click “Register” and the registration is completed. But your tests are not.
You have tested just one scenario of the sign-in form usage. Now it is time to check others like, for example: “what if an email given is already in the system”, “what if the user forgot to enter their first name”, “what if the user entered an invalid email” and so on and so forth. By the time you are finished, you will have noticed that it does take quite a lot of time. And this was only the registration form – what about the other parts of the system that should be tested before each deployment. Now, the bigger the system and the greater the pressure to deliver functionality on the app, the less likely you are to be involved with the time consuming process of manual testing.
HOW AUTOMATED TESTING WORKS?
Automated testing is not limited to testing user behaviour. Unit testing is focused on individual units of source code. It is used, for example, to test if a method of a class returns the correct value. Every method with complex calculations is an ideal candidate to be tested in this way. The developer should write enough tests to cover corner cases and thus ensure that the method will behave correctly even if it receives fairly absurd parameters.
On top of that, developers also write integration tests to, for example, verify the interactions between modules or classes. There are also more general testing levels: system testing to check if a fully integrated system meets the requirements or system integration testing focused on the integration between your application and third-parties’ systems.
ADVANTAGES OF AUTOMATED TESTING VS. MANUAL TESTING
Automated tests may not necessarily be able to solve each and every problem, but they surely provide you with substantial benefits when compared to manual tests. What are the most striking advantages automated tests offer?
1. AUTOMATED TESTING IS CHEAPER
On the whole, it is cheaper to pay once (writing a test suite for entering proper values into the form) rather than to pay multiple times for the same work (manually entering proper values into the form every time the app needs to be tested). I mean, the fully-loaded cost of automated tests in the SDLC is lower than those of “equivalent” manual tests (if the latter were in fact applied on the project).
2. AUTOMATED TESTING IS FASTER
After the initial effort of writing tests has been undertaken, automated tests are performed much faster than what it takes to perform the corresponding manual tests; this is one of the reason why they are actually run in the first place; manual tests on bigger systems take too long to be executed thoroughly and frequently.
3. AUTOMATED TESTING IS MORE RELIABLE
When the developer writes a test and adds it to the test suite that is performed on each deployment, the test just cannot be forgotten. A manual tester, on the other hand, can simply forget to perform some specific tests. He may even choose not to perform some tests at some deployments intentionally, assuming some parts of the system have been tested so many times before that there is no need to test them again / so often. By the way, this approach usually leads to the errors being discovered by the users of the app.
4. AUTOMATED TESTING REDUCES HUMAN AND TECHNICAL RISKS
The developers who have worked on your application may change work or else you may choose to bring the project in-house to have it developed and/or maintained by your own staff; both scenarios mean a project hand-off, i.e., the application will have to be taken over by another developer/team. Without tests, it will be a very laborious and risky process. Automated tests help the developers embarking on an existing project to start developing without worrying much about possible damage the changes they introduce may cause.
Additionally, such tests help them learn a lot about the application – some tests can act as documentation for programmers. For instance, the use of tools like CucumberFM or SteakFM enables developers or the Product Owner to draw on test scenarios as a form of system documentation.
Finally, automated tests facilitate code refactoring, e.g. when developers modify the code to improve the performance of the system rather than intend to change the behaviour of the app. With automated tests at hand, the developer can change the code and run the tests to easily find out if the changes made have impacted the system or not. A lot of time / workload can be saved in this way.
5. AUTOMATED TESTING IS MORE POWERFUL AND VERSATILE
A human tester cannot manually create 100.000 users to check if the app still meets the performance requirements; more specifically, for example, to check if the generation of statistics from that number of users will take less then 2 seconds. Likewise, the manual tester cannot “travel in time”. With automated tests you can, for instance, change the creation date for the objects in the database, e.g. user accounts. You can also check how the system will behave in the future, e.g. you can check if the user with an account created 49 weeks ago, will receive a congratulations email the following week. Automated testing may allow you to tackle some such challenges easily. Manual testing will not as it is restricted by the existing user interface.
WHAT SHOULD I CONSIDER BEFORE I MAKE THE DECISION TO USE AUTOMATED TESTING?
Test automation is surely not an ultimate remedy for ALL your problems. Such tests are developed by humans and humans make mistakes. Virtually any test so created can be wrong. Conversely, in some cases a human will spot an error where an automated test fails to do so. There is a splendid example of such a situation cited in “The Lean Startup” by Eric Ries (@ericries): if a designer changes the colour of a button to white and the background is white, automated tests will pass. The users – or for that matter, even the developers – will not be able to spot / use the button any more.
All in all, automated tests deliver great service but they are not a panacea for all evils.
A NOTE ON DEVELOPERS
It is worth noting that the tests are also code in themselves, developed with specific practices, varying in how well they perform / execute. Like code, they can be reused to cover other similar parts of the system to reduce the workload and the associated cost. So the benefits do depend on the quality of the tests applied. Make sure the development team you hire are able to ensure quality in this respect.
A NOTE ON COSTS
I mentioned that automated testing is cheaper than manual testing. Overall and in the long run. While implementing specific features you have to account for extra workload/cost – usually 10-50% (but in some cases even 100+) more than if you developed without a test coverage. In a nutshell, writing a feature and the tests for it takes longer than coding just the feature. Besides, when you change a part of the system that is already covered with tests, the existing tests may have to be modified, which again requires extra work. Another thing that adds to the paycheck is the cost of the associated infrastructure, e.g. a continuous integration server (e.g. Teamcity).
A NOTE ON TIME
The last thing to consider is time. With a fixed team velocity, features with tests emerge more slowly, which creates a challenge for those applications where time is critical or, in other words, where the cost of delay is high. What should we do if we are pressed for time to deliver an app? Ideally, developers write tests before coding each feature (Test-Driven Development) or shortly afterwards, but still – before deployment. If we really are pressed for time AND the quality of what is to be delivered is not absolutely crucial, we may choose to postpone writing tests until after deployment. In such cases though, we should strive to complete the missing tests as soon as possible.
And what if we need the benefits of automated testing and we still have to deliver fast? I think the best option is to determine the most critical paths in our system and cover them with integration tests, deploy the solution and only then progress to cover the rest of the system with tests. All such tactics involve a compromise that comes with trade-offs.
If you are after quality, I strongly advocate testing your application properly. Automated tests may contribute a lot in this respect. They come at an “up-front” cost and they do take more time to develop. Still the investment will pay off – in terms of workload and cost and time – later on in your product’s lifecycle. Overall, automated testing is faster and cheaper than manual testing. And it helps to ensure quality in a way manual testing never will do.
Even when the system is fully covered with automated tests, you may still perform some manual testing to identify, e.g. UI problems. Do not assume, however, that manual tests will ever be able to deliver the benefits that come with automated tests and can thus be used as a substitute for the latter.
Special thanks to Blażej Kosmowski for his critical feedback on the above article.