Test Ensembles
Test Ensembles are collections of tests that cover specific test intentions. This tool allows developers and administrators to set different goals and run sophisticated optimization engines that will select for them the tests that are appropriate for the goals they have set.
Think of, for example, a test development team for an online store website. Once the scenarios that can be executed to the site have been defined, the team has a model that allows for thousands of tests. Using Provengo’s test ensembles generation tool, the team can automatically create ensembles of tests of relatively small size (say, several dozen of tests in an ensemble) that cover the model according to a coverage criterion that fits the tested application and the stage of the project. The team can, for example, choose to maximize the variability of tested scenarios and/or specify a set desires such as "we want tests that reach the 'order completion' stage" or "we want tests with 'backup' while there is 'site activity'".
Ranking Functions
The basic tool for specifying the goals that we want our test ensembles to achieve is ranking functions. Mathematically, a ranking function specifies symbolically, for each candidate ensemble, a number that specifies how much the ensemble serves our current goals. Recall that an ensemble is nothing but a set of tests, so the set is just the power set of the set of tests.
The ranking functions are defined in our model using a language that is very similar to the language used to define the events and the scenarios. Assume, for example that we want to define the function which is a type of generalized sequence coverage proposed in a recent scientific paper. We can add to our model the following piece of code:
function ranking_function(ensemble) {
let set = new Set();
for (var test of ensemble)
for (var i=0; i<test.length-1; i++)
set.add([test[i], test[i+1]]);
return set.size;
}This code demonstrates how we define the ranking function simply by adding a function called ranking_function
to our model. The parameter to this function is an ensemble represented by an array containing a set of tests where
each test is an array of events that represent the steps of the specific test. The function analyzes the ensemble and
returns a number that marks how well the given ensemble serves the goals that the testers want to achieve. In this
example, the function counts the number of different consecutive pairs of events exercised in the ensemble. This is an effective
general measure of sequence coverage proposed in the paper mentioned above.
Ensemble Generation
Provengo provides a two-step process for building test arrays called ensembles. In the first step a quantity of possible tests can be sampled using the command:
provengo sample --delete --size=<number> <project_directory>
The size parameter determines how many tests will be sampled. It is recommended to sample more than 1000 times the size
of the desired test set. The optional delete parameters tells Provengo to delete the old samples. Without it, the
sampler will add the samples to the existing file. Use the delete flag when the model has changed and the old samples
are no longer relevant. If in doubt, it is safer (but less efficient) to always use the delete flag.
In the second stage, an evolution-based optimization mechanism is activated for the purpose of selecting a subgroup that will maximize the value of the ranking function we defined above. This is done using the
provengo ensemble [--algorithm=<genetic,brute_force>] --size= <project_directory> provengo ensemble --algorithm=goals <project_directory>
command. The parameter suite-size defines the size of the ensemble, the number of tests in the array. This number is
determined by the needs of the system under test. It is recommended to run at least a few dozen tests, preferably
hundreds and thousands if there is time and resources.
An optional parameter ensemble-file allows to specify the name of the file that the ensemble generates.
The default name is ensemble.json. Use other names if you need to manage more than one ensemble for your project.
|