Testing is an integral part of any successful software project. In software testing, there are two main areas to complete the verification of a product – automated testing and manual testing.
Test Automation and Manual Testing
Manual software testing, as the name implies, is executed by a person without the help of any tool or script. Automated software testing, on the other hand, is executed with the assistance of tools, scripts and software.
Nowadays, there’s a question that keeps being heard around software companies – a question that’s in every manager’s head around the globe; a question that if answered incorrectly may end up hurting the quality of our products. The question is: should we manually test applications or should we implement automated tests?
The Big Question :
Almost all development companies get to a point where they have to decide on a test strategy. Do they go with pure BDD/TDD (behaviour driven development/ test driven development) which leads to one or no tester? Or do they hire/keep manual testers to do the product verification?
There’s a reason this discussion keeps being brought up – the need for “cutting costs.” From experience, when making decisions about budgeting, the area that takes the heavy hit is usually testing. Let’s face it, it’s not cheap to maintain a lot of resources to only test an application.
Companies can get into a real dilemma trying to answer this question. But is there really a rivalry between these two testing techniques?
Clarifying the difference :
TDD and BDD :
When comparing these two areas, we already know that automation testing has several benefits such as cost reduction, reusability, speed, and reliability. This technique helps reduce unnecessary manual effort and allows us to complete testing more quickly in the SDLC (software development life cycle) thanks to BDD/TDD, for those projects which do not have complex business rules.
Costly Decision :
The need to get apps on the market in a shorter timeframe has caused many companies to pull resources away from QA Staff on the assumption that automation is suitable for all testing needs. But is this really a good decision?
Manual and Automation are Brothers :
Even though BDD-TDD techniques have made significant advances, there’s no reason to think that manual testing cannot live alongside automated testing. Both testing techniques are 100% necessary, and can complement each other in multiple ways. Since no testing method is perfect, by using them together you can identify errors that the other is prone to missing.
Automation Execution :
We have to remember that automated testing was originally designed to reduce costs on repetitive test cases that needed to be run many times, for regression testing, and for building up the long term product quality. As we keep adding automated tests, the automation suite gets more robust, it takes on the responsibility of running test cases that can cause tester boredom and errors in execution.
The Other Way :
There’s another way of thinking about automated tests and about how we execute these test cases. We can consider them “confirmation tests,” because if the first execution doesn’t find a bug and we keep executing through the cycles, we probably won’t find new bugs. The next execution would only be to confirm that no new bugs are found. These automated test cases confirm that an application is working as expected when it was correctly built in a continuous integration environment.
The problem here is that the tests may start losing quality value because they stop providing meaningful data. Even more so if a reporting tool is not implemented along with the test – then it will only confirm code for how that particular feature (or an old feature) works. To avoid losing quality value, we need a resource to look after the whole automated suite – a resource with sufficient quality knowledge and project experience to track application changes with the support of a tool.
For One Reason:
This is one of the main reasons we need people to collaborate on testing our products: to attempt to “destroy” the application, to seek out errors, to think of all the possible ways to wreck digital security, databases and services. Test cases to complete these actions could definitely be done by automation, but speaking from experience, the cost of writing those scripts isn’t worth it for one fundamental reason: it might only be run once.
Planning :
When we’re planning our project testing, we need to answer some basic questions in order to decide the correct mix of both areas/techniques:
1. Project time frame – How much time do we have?
2. Core business – How critical is our software to the business running it?
3. Who is the targeted audience?
4. Costs – Are you running on non-free ware technologies and need to consider the cost of licensing?
5. How subjective is the application – Are the business rules the app serves too complex?
Deciding Factors :
Considering all the variables, usually the choice between a manual and automated test comes down to a few key issues:
Number of Test Cases :
If the test case execution occurs a small number of times, manual testing would be better. For example, projects with a static page, with no database connection, and few elements interacting with the page.
Number of Sub-Projects :
If the project is comprised of small projects of a similar type, automation can take care of the features they have in common and you can run a manual test after or before the automated test scenarios to complete the verification.
Time Constraints :
Exploratory testing (manually-executed) + automated test are always a good combination for projects where we don’t have much time or where requirement specification is poorly written. This type of testing technique requires experienced testers, with creativity and intuition. Automation comes in when we find the spare time to create regression tests for pre-existing features we need to keep an eye on.
To answer these three questions and help find the right mix of tests, we need to have a good vantage on upcoming projects and consider the pros and cons of both systems. It’s impossible to create automated tests for everything and as always, a minimum amount of tests should be executed for any product.
Pro’s and Con’s :
| Automation Testing | Manual Testing | |
|---|---|---|
| Automation testing performs the same operation each time. | Manual testing is not always reliable. Using this method of test execution is not accurate when testers have to verify the same scenario 20 times. | |
| Automation testing is very helpful with regressions in testing where code changes frequently. | Manual testing executed in regression testing might not catch defects for frequently changing requirements. | |
| Automation testing is useful when frequently executing the set of test cases. | Manual testing will be useful when the test case only needs to run once or twice. | |
| Automation testing can also be done on different machines with different OS platform combinations, concurrently. | With manual testing, testing on different machines with different OS platform combinations cannot be done concurrently. Executing each task requires different testers. | |
| Sometimes it’s not helpful for UI testing. | It is very helpful in UI testing. | |
| Automation testing is very useful for automating the Build Verification Testing (BVT) & it is not overly mundane or tiresome. | Executing the Build Verification Testing (BVT) is very mundane and tiresome in manual testing. | |
Conclusion :
Finally, we have to keep in mind that since professional testers and automation tools aren’t perfect, bugs can still end up in the final product. It’s almost inevitable that no matter how big or small your project is, these techniques still need to be matched, considered, and sensibly studied when making testing decisions. The right choice though can save you lots of time and offer improved results, minimizing the bugs found in production, giving your application more chances to succeed.
About the Author :
Bruno Soko – an ISTQB-CFTL certified QA Engineer. He has worked at a variety of software and technology firms, including Globant and HP. Bruno is currently a QA Automation Engineer at Monsoon, an Oakland, California-based software development company.
