Manual or Automated Testing – Which is Better?

talks about QA Wizard Pro on April 12, 2010

I am often asked a simple question that is anything but simple. “Which is better – manual testing or automated testing?” It’s impossible to answer without knowing what’s being tested. Everyone seems to want an easy answer so they can go to their boss and say, “If we only use ‘insert your favorite testing preference here’ testing then all will be well with our testing”. If it was that easy, there would only be one version of software, tested once and always working thereafter, with no revisions. Yeah right!Answering the following questions can help you determine if manual or automated testing is right for you.

  1. What are you trying to test? Manual testing is great when a person’s judgment is required. Automated testing excels in repetitive testing, such as regression testing.
  2. Are there specific repeatable results every time? Manual testing is usually better with dynamic conditions while automated testing is good for verifiable results.
  3. What are all the variations that can occur? Automated testing handles variables well and never gets bored catching issues that manual testing may miss.
  4. Is the result text-based? Automated testing generally works well with text strings and numeric values. Manual testing of image-intense projects is recommended.
  5. Is it a repetitive test? Repetitive tests are usually good candidates for automation.
  6. Is a judgment call required? Manual testing is recommended if you are verifying the quality of images in a web page or the clarity of an online video since automation lacks judgment in such testing.
  7. Are you testing large quantities of data? Without automation this kind of testing can be a resource hog and is also more prone to human error.
  8. Are you testing size limits or boundaries? These tests are good candidates for automation because computers don’t care how few, how many, or what characters are typed. Computers also don’t lose count. People have to manually count and visually check characters, which may lead to errors.
  9. Does the test include characters that are hard to distinguish? Automated testing can easily tell the differences between characters, such as an ‘o’ (lowercase letter) and a ‘0’ (number), which may be difficult for a tester to distinguish.
  10. How are pop-up error messages handled? Mixed testing may be best, unless you’re using test-driven development which is automated from the start. Use manual testing to determine the correct responses to the message, and then automate the error handling as part of the test.

OK, I think you get the idea. There is not a universal answer to the ‘Which is better – manual testing or automated testing?’ question. Good testing is about identifying what to test, the expected results, and how to make sure as many scenarios as possible are covered. At a minimum, good testing requires a well-thought-out strategy to ensure testing results are meaningful and help development progress. When you understand what you are testing and the expected results, then the type of testing needed starts to become clear.

2 Responses to “Manual or Automated Testing – Which is Better?”

  1. Sanjay Bhatta says:

    Is development better or automation testing better based on future prospect?

    • Geoff Schardein says:

      That is an interesting question and the answer depends on what you are trying to achieve. The type of automated testing will make a difference to the answer. There is White Box testing which is looking for coding logic errors and is often done as a bench test by the developer to ensure the code does as expected. This kind of testing is typically more developer centric and is done as development occurs.

      There is Black Box testing, as is done by automated regression tools such as Seapine’s QA Wizard Pro. The Black Box tool acts as a user taking over the keyboard and mouse, this kind of testing is better for more mature builds/versions to see if changes have broken existing functionality. Typically Quality Assurance would manage this kind of testing.

      There are also testing strategies and methodologies to consider. There is a more Agile process of Test Driven Development where the developer plans frequent unit test as part of the development cycle. There is Continuous Integration where the code is complied often and tested. This kind of testing is typically done via automated regression testing; testing previously existing functionality to make certain nothing was broken by the changes to the code. Manual testing of newly added functionality would be done initially with automation applied later as much as possible.

      Regardless of the process testing should be an integral part of all development, if you do not have an expected result and a way to test the result how will you know when you are done?