Test automation reports are an important part of our test automation framework. Once you execute your automated tests, the test results will be your only window into what really happened during the runs, what passed, what failed and why. Being able to analyze these reports is the only way to find out actual problems, correct them and continue on your path to better quality. For this reason, every tester needs a good, robust, and understandable reporting mechanism to be a part of their test automation framework.
The Challenge
Most test automation tools focus more on execution and fail to focus on what comes after. While providing a pass/fail report is a good start, a tester analyzing the result needs much more information to be able to fix problems.
Failures may happen in test scripts due to actual defects or due to changes in the system causing scripts to fail (unable to find elements or change in workflow). Test automation reports are rarely that informative or even if they are, they are cumbersome and complicated.
Many open-source tools also have no in-built reporting mechanism and require testers to rely on third-party integrations. So, they have to set up extensions or plugins to be able to see a basic HTML style report and add it to their CI build system.
Here are the frequent problems testers face pertaining to their test automation reports:
- Not knowing if the script failed due to a wrong step or an actual defect in the system
- It takes ages to find the exact cause of the script failure and having to traverse multiple files of code.
- The errors or exceptions are not meaningful, and testers must google what they mean and how to resolve them.
- A tester needs to manually add taking a screenshot step every time there is a failure, to get to see the application state at the time
- A tester wants to see the complete run result for my test suite, instead, they must traverse each script run individually
- Testers would like a more visual representation of which scripts passed vs which failed in my test suite
- Testers need to share my test results with the entire team & the manager – they need a quick way to share a consolidated report without having to manually write it!
- Testers have many scripts dependent on each other- they do not want all dependent scripts to ‘Fail’ if the main script failed. It shows unnecessary failures and skews my reports!
The solution using Sahi Pro
Sahi Pro focuses on providing detailed test reports for both individual runs and suite runs.
Sahi Pro comes with in-built reporting mechanism and its run reports are visual, understandable, and very descriptive.
When a suite or script is executed, detailed reports of suite, script, steps, and errors are automatically created by Sahi and stored in a database.
- Sahi Reports can be viewed as a web page or downloaded as offline files in various formats.
- Reports can also be synced to a central database or between Sahi installations.
- The appearance of reports can be modified via XSL style sheets.
- Various other parameters of reports can be also be configured.
They help you not only see the exact failed step within the test but also lets you go to the exact line of code that failed, thus making debugging, and fixing very easy.
Sahi Pro also takes a screenshot automatically on step failures and errors.
Screenshots give visual information on the state of the application during failure.
Suite Reports
Suite Reports are a collective report representing all scripts executed as a part of the suite and their status, time taken, no. of failed steps in each and their pass percentage %
Suite Analysis
When suites fail, a good amount of time may be spent analyzing the test results to find the core problem.
Suite Analysis is designed to help quick fix of scripts causing most failures with a consolidated failure report. Historical results, for each script which are having at least one failure history, are also provided.
Sahi Pro has enhanced the Suite Analysis to analyze the reports and pinpoint the source of failures. When you know that the 23 failing scripts are all failing due to just 3 specific steps, it becomes very easy and fast to fix and rerun!
Downloading Reports
Multiple suite reports can be downloaded directly from the logs page. One can set the report type from the "Download as" dropdown.
The reports can be downloaded with varying formats including email, html, xml, excel, junit.
One can also download multiple script reports inside a suite, if there exists any script inside that suite.
Consolidated Reports
Sahi Pro also provides you a single-click Consolidated Report of Scripts and Testcases of selected Suites. This gives an easy way to provide a consolidated result of your runs to the management or the team in a simple, understandable format.
You can even download it as an excel to attach to your emails.
So, you do not spend time writing emails and consolidating your results manually in a excel!
Sahi Run Status
Sahi Test runs will always show meaningful Status to indicate the exact state of the run.
For example-In case of dependencies between scripts, if the main script failed, all ‘Dependent’ Scripts are marked as Skipped and not Failed.
Similarly, if the script is aborted due to browser or network issues, it is called ‘Aborted’ while the status is shown ‘User_Aborted’ if you manually stop it.
These different statuses make a huge difference when analyzing the overall Test Suite Reports and show a truer picture of what happened during the run without you having to manually analyze each false failure.
See it in action
Download a Free trial version of Sahi Pro to give BDTA a try.
A guide to Sahi Pro Test Reports.
Complete Playlist to take you end-to-end through BDTA concept and usage.
