8.1 Effectiveness Measurements
Automation impacts can be measured beyond speed of test execution. The following areas provide metrics on use and effectiveness of automation.
Schedule
- Test Execution time improvement
- Test Setup time improvement
- Time to determine failures/defects
- Time to analyze data
Effectiveness
- Failures found
- Number of test required/number of system errors
- Defects found/number of test procedures executed
- Test procedures executed without defects/ total test procedures
Coverage
- Test Coverage - test procedures/test requirements
- Automation test coverage - automated test cases/total test cases
- New test capabilities
- Reusability improvement within project/program
Cost
- Man-hour reduction
- Total project or program savings
- Additional training requirement
- Additional resources
- Additional Maintainability
Each project or program should select those metrics that provide the most relevant feedback to assess quality and effectiveness of automation. Although test
execution times are dramatically shortened through the use of automation, test preparation steps and maintenance for automation need to be measured as part of the
overall effort and cost, as compared to manual testing.