The rtv (Runtime Variables) Library

/* @provengo summon rtv */

The Runtime Variable library allows capturing, processing, and testing values during test execution. For example, when testing a search engine, this library can be used to capture the result count of a broad search (say, "Pizza"), the result count of a narrower search (say, "pizza mentions in rock ballads"), and to validate that the broad search had more results than the narrow one.

The supported model API is documented below. Members whose names begin with __, along with the diagnostic toString() method, are internal implementation details.

Other usages include:

  1. Using API tokens in tests - receiving one at the beginning of a test and then using it for the rest of the test.

  2. Reducing test graph size, by storing data that should not affect behavior on its own. E.g. selecting a town on some initial stage, and then asserting that the system displays the same town at a later stage.

Values are typically stored by methods in actuation libraries such as the Selenium Library's store method.

Using Runtime Variables and the @{} Syntax

Accessing runtime variables is done by enclosing the variable name in @{}. For example, to have an event whose name is tha value of runtime variable rtv1, use Event("@{rtv1}").

Any JavaScript expression can go inside @{}. You can work with multiple variables, and even reference the top-level scope of your model, such as data stored in files at the data/ directory, or functions defined at the lib/ directory.

// ... inside a bthread, bws is a Selenium browser session
bws.store("firstName", XPATH_TO_FIRST_NAME_FIELD);
bws.store("lastName", XPATH_TO_LAST_NAME_FIELD);
bws.assertText(
    "Sincerely, @{firstName[0]+'. '+lastName}",
    XPATH_TO_PARTING_LINE); (1)
1 Assert that the greeting is in the form of "Sincerely, L. Cohen".
It is not possible to access local function variables. The reason is that these will be long gone when runtime arrives - the function runs during scenario generation, whereas test execution (runtime!) is done much later, typically after sampling the model and composing an optimal test suite.

Example

/* @provengo summon rtv */
bthread("sample-and-check", function(){
    // Generate two selenium sessions, s1 and s2.
    s1.store("c1", "//*[@id='counter1']"); (1)
    s2.store("c2", "//*[@id='counter2']"); (2)
    rtv.assertLt("@{c1}", "@{c2}");        (3)
    rtv.assertLe("@{c1}", "17");           (4)
    rtv.assertEq("250", "@{c2}");          (5)
});
1 Storing the text content of DOM element whose id is counter1 into runtime variable c1.
2 Same as above, but with counter2 and c2
3 Asserting that the numeric value of c1 is less then that of c2
4 Asserting that the numeric value of c1 is less then or equal to 17.
5 Asserting that the numeric value of c2 is equal to 250.

Events

rtv.storeEvent(varName, value)

Creates an event that stores value in the runtime variable named varName when the event is executed. This method only creates and returns the event; use request(rtv.storeEvent(…​)) to add it to a scenario. For a shorthand that creates and requests the event, use rtv.doStore(varName, value).

varName

String. The name of the runtime variable. The name is evaluated when the event executes; the resulting name must be a valid JavaScript identifier and must not be a reserved JavaScript word.

value

Any JavaScript value. Strings containing @{} expressions, and fields of JavaScript objects, are evaluated when the event executes.

Example 
bthread("store customer", function(){
    request(rtv.storeEvent("customer_num", 42));
    request(Event("Validate customer @{customer_num}"));
});

For an example that stores a value produced while a REST request is executing, see the REST library example. Runtime callbacks use pvg.rtv.set(key, value) because the value is available only during test execution; rtv.storeEvent(varName, value) is used when creating the scenario.

rtv.anyStore(varName)

Returns an event set containing all RTV storage events that store values into varName. It is useful for triggering scenarios when a certain variable is updated, blocking a variable from being updated, and defining test-suite goals based on which values were updated.

This event set does not capture indirect storage events such as methods from the Selenium package that store directly to the Runtime Value Store.
varName

String. The name of the runtime variable we want to monitor.

Example

Below, whenever runtime variable customer_id is updated, the on handler prevents any updates to the variable until it completes a set of validations.

// @provengo summon rtv
on( rtv.anyStore("customer_num"), function(){
    block( rtv.anyStore("customer_num"), function(){
        request(Event("Validate Customer @{customer_num}"));
        request(Event("Validate customer count"));
    });
});

rtv.assertEqEvent(v1, v2, errorMessage)

Creates and returns an event that compares the string representations of v1 and v2. The assertion succeeds when they are equal. Otherwise, the test fails.

v1, v2

Values to compare. Strings may contain @{} expressions, which are evaluated when the event executes.

errorMessage

Optional string. The failure message; it may contain @{} expressions. If omitted, a message showing the unequal values is used.

rtv.assertLtEvent(v1, v2, errorMessage)

Creates and returns an event that converts v1 and v2 to numbers and asserts that v1 < v2.

v1, v2

Values to compare numerically. Strings may contain @{} expressions, which are evaluated when the event executes.

errorMessage

Optional string. The failure message; it may contain @{} expressions. If omitted, a message showing the failed comparison is used.

rtv.assertLeEvent(v1, v2, errorMessage)

Creates and returns an event that converts v1 and v2 to numbers and asserts that v1 ⇐ v2.

v1, v2

Values to compare numerically. Strings may contain @{} expressions, which are evaluated when the event executes.

errorMessage

Optional string. The failure message; it may contain @{} expressions. If omitted, a message showing the failed comparison is used.

rtv.assertEvent(v1, v2, testFunction, alias, errorMessage)

Creates and returns an event that evaluates v1 and v2, passes them to testFunction, and succeeds or fails according to the returned boolean. Returning a non-boolean value produces an execution error.

v1, v2

Values passed to testFunction. Strings may contain @{} expressions, which are evaluated first.

testFunction

Function accepting the evaluated values and returning true or false.

alias

Optional string used to identify the assertion in analysis graphs.

errorMessage

Optional string. The failure message; it may contain @{} expressions. Defaults to Assertion failed.

rtv.log.infoEvent(text)

Creates and returns an event that logs text at the INFO level when requested and executed.

rtv.log.fineEvent(text)

Creates and returns an event that logs text at the FINE level when requested and executed. FINE messages are printed only in --verbose mode.

rtv.log.warnEvent(text)

Creates and returns an event that logs text at the WARN level when requested and executed.

rtv.log.errorEvent(text)

Creates and returns an event that logs text at the ERROR level when requested and executed.

For all four logging event methods, text must be a string or number. Strings may contain @{} expressions, which are evaluated when the event executes. As with the other event factory methods, call request(rtv.log.infoEvent(text)) (or fineEvent/warnEvent/errorEvent) to add the returned event to a scenario. An ERROR log entry does not by itself fail or error the test.

Methods

rtv.doStore(varName, value)

Stores value into runtime variable varName. In the typical use-case, runtime variables are updated by an automation layer, e.g. when a REST API requests a customer id from the system under test, and then uses that id during the rest of the test scenario. This command, which updates a runtime variable with a variable value from scenario generation time, serves a different use-case. It is typically used to reduce the model state space (see example) or to simulate sampled values from a currently unavailable source.

varName

String. The name of the runtime variable. The name is evaluated when the event executes; the resulting name must be a valid JavaScript identifier and must not be a reserved JavaScript word.

value

Any JavaScript value. Strings containing @{} expressions, and fields of JavaScript objects, are evaluated when the storage event executes.

Example

The bthread below uses the runtime variable system to reduce states.

bthread("writer", function(){
    let food = select("food").from(              (1)
        "Glazed Donuts",
        "Jelly Donuts",
        "Bavarian Cream-Filled Donuts",
        "Bear Claws"
    );
    rtv.doStore("runtimeFood", food);            (2)
    food = null;                                 (3)
    request(Event("No, we're out of @{runtimeFood}")); (4)
});

  1. A food type is selected using select().from() an is place into food, a regular variable.

  2. The value from food is stored into the runtimeFood runtime variable. This is requests an event that will later appear in the test scenario, like any other event.

  3. The local variable is set to null, reducing the state space

  4. The runtime variable is used via the @{} syntax.

Scenario space for the above code. Note how all traces converge on the "no food" message node - if the example was using regular program variables, that node would have been replaces by four nodes.

rtv do store

rtv.assertEq(expr1, expr2, errorMessage)

Requests the event returned by rtv.assertEqEvent. The assertion compares the string representations of expr1 and expr2 for equality.

expr1

Number or String. May contain @{} expressions.

expr2

Number or String. May contain @{} expressions.

errorMessage

Optional string. The message to display if the assertion fails.

Example

Checking that both the API and the Web UI agree on an account balance. Assume browser is a Selenium session, and api is a REST API session.

bthread("account-balance-correct", function(){
    browser.store("uiBalance", XPATH_TO_BALANCE);
    api.get("/balance", {
        callback: function(response){
            pvg.rtv.set("apiBalance", response.body);
        }
    });
    rtv.assertEq("@{uiBalance}", "@{apiBalance}", "Balances on API and UI do not match");
});

rtv.assertLt(expr1, expr2, errorMessage)

Requests the event returned by rtv.assertLtEvent. Both expressions are converted to numbers before checking that expr1 < expr2.

expr1

Number or String. May contain @{} expressions.

expr2

Number or String. May contain @{} expressions.

errorMessage

Optional string. The message to display if the assertion fails.

Example

Asserting that the user account balance is greater than 10.

bthread("account-balance-correct", function(){
    browser.store("balance", XPATH_TO_BALANCE);
    rtv.assertLt("10", "@{balance}", "User balance should be greater than 10.");
});

rtv.assertLe(expr1, expr2, errorMessage)

Requests the event returned by rtv.assertLeEvent. Both expressions are converted to numbers before checking that expr1 ⇐ expr2.

expr1

String that may contain @{} expressions.

expr2

String that may contain @{} expressions.

errorMessage

Optional string. The message to display if the assertion fails.

Example

Asserting that the user account balance is greater or equals to 10.

bthread("account-balance-correct", function(){
    browser.store("balance", XPATH_TO_BALANCE);
    rtv.assertLe("10", "@{balance}", "User balance should be greater than or equal to 10.");
});

rtv.assert(expr1, expr2, testFunction, alias, errorMessage)

A generalized way of asserting over runtime values. Requests the event returned by rtv.assertEvent. The assertion succeeds only when testFunction returns the boolean value true; it fails on false and produces an execution error for any other return value.

expr1

Value passed to testFunction. Strings may contain @{} expressions.

expr2

Value passed to testFunction. Strings may contain @{} expressions.

testFunction

Function. Takes two parameters (expr1 and expr2), returns a boolean (true/false). If it returns false, the assertion fails.

alias

Optional string used to identify this assertion in analysis graphs.

errorMessage

Optional string. The message to display if the assertion fails.

Example

Asserting that the sum of two runtime variables is even. Values were populated on different bthreads.

bthread("must be even", function(){
    rtv.assert("@{varA}", "@{varB}", function(a, b){ return (a+b)%2==0; } )
});

rtv.log.LEVEL(text)

Logs runtime values during test execution. Logging is available at four levels: fine, info, warn, and error. Messages logged to fine are only printed when running in --verbose mode. All messages are stored in the logs. Useful for inspecting calculations performed and values sampled from tested system during test execution.

The available requesting methods are rtv.log.fine(text), rtv.log.info(text), rtv.log.warn(text), and rtv.log.error(text). Each requests the event returned by its corresponding rtv.log.LEVELEvent(text) method.

text

String or number to log. Strings may contain @{} expressions, which are evaluated when the event executes.

Example

Logging at the various levels during execution:

// Performs an API call and stores the data
bthread("performApiCall", function(){
    session.post("/get-quote", {
        callback: function(r){
            let body = JSON.parse(r.body);
            pvg.rtv.set("quote", body.quote);      (1)
        }
    });
    rtv.log.info("Suggested quote is @{quote}");   (2)
    rtv.log.warn("Tax on quote is @{quote*0.17}"); (3)
});
1 Querying the system-under-test for a quote.
2 Logging the quote at the INFO level.
3 Logging an expression based on the quote at the WARN level.

rtv.run(processFunction)

Requests an event that performs general processing of runtime values. For example, it can extract numbers hidden in text paragraphs so they can be asserted later.

processFunction

Function. Receives an object that can read and write to the runtime values table, using methods get(varName) and set(varName, value).

Example 
browser.store("userStats", XPATH_TO_USER_STATS)                  (1)
rtv.run(function(values) {
    let userCount = extractUserCount(values.get("userStats")); (2)
    values.set("userCount", userCount);                      (3)
});
request(Event("User count is @{userCount}"));                (4)
1 Sample user statistics text from a web ui
2 Read that value, and call a function to extract user count
3 Store the clean number in a new runtime variable
4 Using the new variable value as usual.