9.1 Reporting
Test reporting is an important part of the process of automated testing. There are a number of different ways by which we can produce reports from automation.
Test execution tools represent the most critical component in reporting. It is at the execution of tests that data can be captured which is then used for reporting. Test execution tools will often have one reporting mechanism that shows that occurred during the test execution, including any verification steps and any failure points. The reporting provided by these tools is very focused to the sequence of events that occurred and often do not capture larger trends. Execution tools often have the ability to customize the reporting, often by inserting messages and custom output text to complement default reporting. Additionally, execution tools may have separate logs that show if any errors occurred as functions were called. Custom logs can be created in order to meet specific output requirements, like when data needs to be imported into other tools in a standardized format.
Test management tools often help to store and aggregate data and can report on an individual or group of tests. This is helpful in order to see trends in application quality. Management tools often have several mechanisms for reporting including standard table reports, summary table reports, and graphical data representations. These can include charts, graphs, and other visual elements that facilitate analyzing high level data without having to look at the more granular data.
Reporting tools exist that are not specific to testing but that can take a variety of input files (e.g. xls, cvs, etc.) and aggregate results for meaningful display. There is overlap with the management tools on some reports. However, reporting tools can also include project management tools that help with scheduling activities and task interdependencies.
As tools have a diverse set of reporting functionality, it may take more than one tool to get all reporting requirements in place. Therefore, understanding each tool's import/export capabilities and requirements will ensure that data can freely flow from one reporting tool to another.