The rtv (Runtime Variables) Library

/* @provengo summon rtv */

The Runtime Variable library allows capturing, processing, and testing values during test execution. For example, when testing a search engine, this library can be used to capture the result count of a broad search (say, "Pizza"), the result count of a narrower search (say, "pizza mentions in rock ballads"), and to validate that the broad search had more results than the narrow one.

Other usages include:

  1. Using API tokens in tests - receiving one at the beginning of a test and then using it for the rest of the test.

  2. Reducing test graph size, by storing data that should not affect behavior on its own. E.g. selecting a town on some initial stage, and then asserting that the system displays the same town at a later stage.

Values are typically stored by methods in actuation libraries such as the Selenium Library's store method.

The RuntimeValues library is summoned by:

// @provengo summon rtv

Using Runtime Variables and the @{} Syntax

Accessing runtime variables is done by enclosing the variable name in @{}. For example, to have an event whose name is tha value of runtime variable rtv1, use Event("@{rtv1}").

Any JavaScript expression can go inside @{}. You can work with multiple variables, and even reference the top-level scope of your model, such as data stored in files at the data/ directory, or functions defined at the lib/ directory.

// ... inside a bthread, bws is a Selenium browser session
bws.store("firstName", XPATH_TO_FIRST_NAME_FIELD);
bws.store("lastName", XPATH_TO_LAST_NAME_FIELD);
bws.assertText(
    "Sincerely, @{firstName[0]+'. '+lastName}",
    XPATH_TO_PARTING_LINE); (1)
1 Assert that the greeting is in the form of "Sincerely, L. Cohen".
It is not possible to access local function variables. The reason is that these will be long gone when runtime arrives - the function runs during scenario generation, whereas test execution (runtime!) is done much later, typically after sampling the model and composing an optimal test suite.

Example

/* @provengo summon rtv */
bthread("sample-and-check", function(){
    // Generate two selenium sessions, s1 and s2.
    s1.store("c1", "//*[@id='counter1']"); (1)
    s2.store("c2", "//*[@id='counter2']"); (2)
    rtv.assertLt("@{c1}", "@{c2}");        (3)
    rtv.assertLe("@{c1}", "17");           (4)
    rtv.assertEq("250", "@{c2}");          (5)
});
1 Storing the text content of DOM element whose id is counter1 into runtime variable c1.
2 Same as above, but with counter2 and c2
3 Asserting that the numeric value of c1 is less then that of c2
4 Asserting that the numeric value of c1 is less then or equal to 17.
5 Asserting that the numeric value of c2 is equal to 250.

Events

rtv.anyStore(varName)

An event set containing all storage events that store values into varName. Useful for triggering scenarios when a certain variable is updated, blocking a variable from being updated, and defining test suite goals based on which values where updated.

Does not cover indirect storage events such as methods from the Selenium package.
varName

String. The name of the runtime variable we want to monitor.

Example

Below, whenever runtime variable customer_id is updated, the on handler prevents any updates to the variable until it completes a set of validations.

// @provengo summon rtv
on( rtv.anyStore("customer_num"), function(){
    block( rtv.anyStore("customer_num"), function(){
        request(Event("Validate Customer @{customer_num}"));
        request(Event("Validate customer count"));
    });
});

Methods

rtv.doStore(varName, value)

Stores value into runtime variable varName. In the typical use-case, runtime variables are updated by an automation layer, e.g. when a REST API requests a customer id from the system under test, and then uses that id during the rest of the test scenario. This command, which updates a runtime variable with a variable value from scenario generation time, serves a different use-case. It is typically used to reduce the model state space (see example) or to simulate sampled values from currently unavailable source. For example, if a certain value should be sampled from an API endpoint, but that endpoint is not implemented yet,

varName

String. The name of the runtime variable.

value

Any JavaScript value.

Example

The bthread below uses the runtime variable system to reduce states.

bthread("writer", function(){
    let food = select("food").from(              (1)
        "Glazed Donuts",
        "Jelly Donuts",
        "Bavarian Cream-Filled Donuts",
        "Bear Claws"
    );
    rtv.doStore("runtimeFood", food);            (2)
    food = null;                                 (3)
    request(Event("No, we're out of @{runtimeFood}")); (4)
});

  1. A food type is selected using select().from() an is place into food, a regular variable.

  2. The value from food is stored into the runtimeFood runtime variable. This is requests an event that will later appear in the test scenario, like any other event.

  3. The local variable is set to null, reducing the state space

  4. The runtime variable is used via the @{} syntax.

Scenario space for the above code. Note how all traces converge on the "no food" message node - if the example was using regular program variables, that node would have been replaces by four nodes.

rtv do store

rtv.assertEq(expr1, expr2, errorMessage)

Asserts that the values of expr1 and expr2 are equal.

expr1

Number or String. May contain @{} expressions.

expr2

Number or String. May contain @{} expressions.

errorMessage

Optional string. The message to display if the assertion fails.

Example

Checking that both the API and the Web UI agree on an account balance. Assume browser is a Selenium session, and api is a REST API session.

bthread("account-balance-correct", function(){
    browser.store("uiBalance", XPATH_TO_BALANCE);
    api.get("/balance", {
        callback: function(response, rtv){
            rtv.set(apiBalance, response.body);
            return true;
        }
    });
    rtv.assertEq("@{uiBalance}", "@{apiBalance}", "Balances on API and UI do not match");
});

rtv.assertLt(expr1, expr2, errorMessage)

Asserts that the numeric value of expr1 is less than that of expr2.

expr1

Number or String. May contain @{} expressions.

expr2

Number or String. May contain @{} expressions.

errorMessage

Optional string. The message to display if the assertion fails.

Example

Asserting that the user account balance is greater than 10.

bthread("account-balance-correct", function(){
    browser.store("balance", XPATH_TO_BALANCE);
    rtv.assertLt("10", "@{balance}", "User balance should be less than 10.");
});

rtv.assertLe(expr1, expr2, errorMessage)

Asserts that the numeric value of expr1 is less then or equal to that of expr2.

expr1

String that may contain @{} expressions.

expr2

String that may contain @{} expressions.

errorMessage

Optional string. The message to display if the assertion fails.

Example

Asserting that the user account balance is greater or equals to 10.

bthread("account-balance-correct", function(){
    browser.store("balance", XPATH_TO_BALANCE);
    rtv.assertLt("10", "@{balance}", "User balance should be equal or grater than 10.");
});

rtv.assert(expr1, expr2, testFunction, errorMessage)

A generalized way of asserting over runtime values. Asserts that testFunction return true when called with expr1 and expr2.

expr1

String that may contain @{} expressions.

expr2

String that may contain @{} expressions.

testFunction

Function. Takes two parameters (expr1 and expr2), returns a boolean (true/false). If it returns false, the assertion fails.

errorMessage

Optional string. The message to display if the assertion fails.

Example

Asserting that the sum of two runtime variables is even. Values were populated on different bthreads.

bthread("must be even", function(){
    rtv.assert("@{varA}", "@{varB}", function(a, b){ return (a+b)%2==0; } )
});

rtv.log.LEVEL(expr)

Logs runtime values during test execution. Logging is available at four levels: fine, info, warn, and error. Messages logged to fine are only printed when running in --verbose mode. All messages are stored in the logs. Useful for inspecting calculations performed and values sampled from tested system during test execution.

Evaluates the passed expression and prints it to the log at the specified logging level .

expr

Expression to print as log.

Example

Logging at the various levels during execution:

// Performs an API call and stores the data
bthread("performApiCall", function(){j;o
    session.post("/get-quote", {
        callback: function(r, rtvStore){
            let body = JSON.parse(r.body);
            rtvStore.set("apiKey", body.apiKey);  (1)
        }
    });
    rtv.log.info("Suggested quote is @{quote}");   (2)
    rtv.log.warn("Tax on quote is @{quote*0.17}"); (3)
});
1 Querying the system-under-test for a quote.
2 Logging the quote at the INFO level.
3 Logging an expression based on the quote at the WARN level.

rtv.run(processFunction)

Perform general processing of runtime values. For example, can be used to extract numbers hidden in text paragraphs, to allow assertions later.

processFunction

Function. Receives an object that can read and write to the runtime values table, using methods get(varName) and set(varName, value).

Example 
browser.store("userStats", XPATH_TO_USER_STATS)                  (1)
rtv.run(function(values) {
    let userCount = extractUserCount(values.get("userStats") (2)
    values.set("userCount", userCount);                      (3)
});
request(Event("User count is @{userCount}"));                (4)
1 Sample user statistics text from a web ui
2 Read that value, and call a function to extract user count
3 Store the clean number in a new runtime variable
4 Using the new variable value as usual.