`report`

Generate test execution reports.

Synopsis

provengo report [-o OUT] [--offline] [--suites :last-5] [--type {junit,testlog}] <path-to-project>

Description

Create reports based on data gathered during previous test runs.

Test Log

Shows execution history and results for tests in the executes test suites.

Parameters

-o/--output-file PATH

Path for the generated reports.

-t/--type TYPE

Type of the generated report. Either junit or testlog. Defaults to testlog.

--charset CHARSET

The report’s charset (see https://www.ietf.org/rfc/rfc2278.txt for the complete list).

--offline

Creates logs that do not use on-line resources. Increases log size, but allows full functionality in offline environments.

--suites

Suites to include in the report. Options include:

:all: Include all test suite executions,
:last: Include only the last test suite execution.
:last-nn: Include the last nn test suite executions. E.g., for the last 10 executions, use --suites :last-10
x,y,z: Include specific suite ids. E.g. for suites 5, 11, and 42 use --suites 5,11,42. Note that the numbers are separated by comma only! Adding additional spaces (as in --suites 5, 11, 42) may confuse the argument parser.

Status Lozenges and What They Mean

Reports contain status indicators for test actions, scenarios, and overall statistics. Here’s what they mean.

Indicator Status Meaning Example

Status	Meaning	Example
Success	Assertion or test scenario completed successfully.	a value returned from an API call matched an expected value.
OK	Action or test scenario completed successfully. No assertion was performed beyond the fact that the action was completed.	successfully clicked a button.
Failure	Assertion or action failed.	The text of a UI element does not match an expected value. E.g. expecting "Total: 500 USD" but the actual value is "Total: 400 EUR".
Error	An action failed because it was not possible. This may indicate an bug or a problem with the test setup.	Cannot connect to a certain endpoint because a server is down. Cannot assert a value of a label because that label is not there.
Skip	Action was not performed.	When a step in a scenario fails or creates an error, scenario execution stops. Any action after that action is skipped. These actions are still part of the report and can be executed if the script is executed again, e.g. by downloading it.
Terminated	A test scenario was stopped while is was running.	When Provengo is aborted while running a scenario, that scenario is marked as `terminated`.

status success

Success

Assertion or test scenario completed successfully.

a value returned from an API call matched an expected value.

status ok

Action or test scenario completed successfully. No assertion was performed beyond the fact that the action was completed.

successfully clicked a button.

status fail

Failure

Assertion or action failed.

The text of a UI element does not match an expected value. E.g. expecting "Total: 500 USD" but the actual value is "Total: 400 EUR".

status error

Error

An action failed because it was not possible. This may indicate an bug or a problem with the test setup.

Cannot connect to a certain endpoint because a server is down.
Cannot assert a value of a label because that label is not there.

status skip

Skip

Action was not performed.

When a step in a scenario fails or creates an error, scenario execution stops. Any action after that action is skipped. These actions are still part of the report and can be executed if the script is executed again, e.g. by downloading it.

status terminated

Terminated

A test scenario was stopped while is was running.

When Provengo is aborted while running a scenario, that scenario is marked as terminated.