Smashtest Generate tests fast

Smashtest is an open-source tool and language for rapidly generating tests.

Greatly speed up your web automated testing by writing tests in a tree-like format.

Trees represent how we think when we're testing. They allow us to list all the permutations that branch off from any given point.

npm i -g smashtest
Multiple browsers and devices
UI and API
Run tests in parallel
Human-readable steps
Awesome live reports
Run locally or in CI/CD

Screenshots

Live report
Preview in report
Debug mode
Error reporting
Console
Syntax highlighting

Sample test

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
Open Chrome
Open Firefox
Open Safari
Navigate to 'site.com'
Click 'Sign In'
Type {username:} into 'username box'
{username} is 'joe'
{username} is 'bob'
{username} is 'mary'
Verify success
{username} is 'baduser'
Verify error
represents
Test Case 1 Test Case 2 Test Case 3 ----------- ----------- ----------- Open Chrome Open Firefox Open Safari Navigate to 'site.com' Navigate to 'site.com' Navigate to 'site.com' Click 'Sign In' Click 'Sign In' Click 'Sign In' Type 'joe' into 'username box' Type 'joe' into 'username box' Type 'joe' into 'username box' Verify success Verify success Verify success Test Case 4 Test Case 5 Test Case 6 ----------- ----------- ----------- Open Chrome Open Firefox Open Safari Navigate to 'site.com' Navigate to 'site.com' Navigate to 'site.com' Click 'Sign In' Click 'Sign In' Click 'Sign In' Type 'bob' into 'username box' Type 'bob' into 'username box' Type 'bob' into 'username box' Verify success Verify success Verify success Test Case 7 Test Case 8 Test Case 9 ----------- ----------- ----------- Open Chrome Open Firefox Open Safari Navigate to 'site.com' Navigate to 'site.com' Navigate to 'site.com' Click 'Sign In' Click 'Sign In' Click 'Sign In' Type 'mary' into 'username box' Type 'mary' into 'username box' Type 'mary' into 'username box' Verify success Verify success Verify success Test Case 10 Test Case 11 Test Case 12 ------------ ------------ ------------ Open Chrome Open Firefox Open Safari Navigate to 'site.com' Navigate to 'site.com' Navigate to 'site.com' Click 'Sign In' Click 'Sign In' Click 'Sign In' Type 'baduser' into 'username box' Type 'baduser' into 'username box' Type 'baduser' into 'username box' Verify error Verify error Verify error
which represents
Test Case 1 Test Case 2 Test Case 3 ----------- ----------- ----------- let driver = await new Builder().forBrowser('chrome').build(); let driver = await new Builder().forBrowser('firefox').build(); let driver = await new Builder().forBrowser('safari').build(); await driver.get('http://site.com'); await driver.get('http://site.com'); await driver.get('http://site.com'); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); await signInButton.click(); await signInButton.click(); await signInButton.click(); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); await usernameBox.sendKeys('joe'); await usernameBox.sendKeys('joe'); await usernameBox.sendKeys('joe'); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); Test Case 4 Test Case 5 Test Case 6 ----------- ----------- ----------- let driver = await new Builder().forBrowser('chrome').build(); let driver = await new Builder().forBrowser('firefox').build(); let driver = await new Builder().forBrowser('safari').build(); await driver.get('http://site.com'); await driver.get('http://site.com'); await driver.get('http://site.com'); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); await signInButton.click(); await signInButton.click(); await signInButton.click(); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); await usernameBox.sendKeys('bob'); await usernameBox.sendKeys('bob'); await usernameBox.sendKeys('bob'); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); Test Case 7 Test Case 8 Test Case 9 ----------- ----------- ----------- let driver = await new Builder().forBrowser('chrome').build(); let driver = await new Builder().forBrowser('firefox').build(); let driver = await new Builder().forBrowser('safari').build(); await driver.get('http://site.com'); await driver.get('http://site.com'); await driver.get('http://site.com'); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); await signInButton.click(); await signInButton.click(); await signInButton.click(); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); await usernameBox.sendKeys('mary'); await usernameBox.sendKeys('mary'); await usernameBox.sendKeys('mary'); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); Test Case 10 Test Case 11 Test Case 12 ------------ ------------ ------------ let driver = await new Builder().forBrowser('chrome').build(); let driver = await new Builder().forBrowser('firefox').build(); let driver = await new Builder().forBrowser('safari').build(); await driver.get('http://site.com'); await driver.get('http://site.com'); await driver.get('http://site.com'); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); await signInButton.click(); await signInButton.click(); await signInButton.click(); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); await usernameBox.sendKeys('baduser'); await usernameBox.sendKeys('baduser'); await usernameBox.sendKeys('baduser'); await driver.wait(until.elementLocated(By.id('#error-element')), 10000); await driver.wait(until.elementLocated(By.id('#error-element')), 10000); await driver.wait(until.elementLocated(By.id('#error-element')), 10000);

Getting Started

Start by installing Smashtest, then writing your first .smash file

Setup

1. Install NodeJS

Make sure you have NodeJS installed. We recommend the LTS version. Use node -v to check.

2. Install Selenium Webdriver (if you're doing web UI testing)

  1. Make sure all of the browsers you want to automate are installed
  2. Make sure you have Java installed. Use java -showversion to check.
  3. Choose one of the following options:

Option 1: Webdriver manager

Webdriver managers take care of the installation process for you, but usually require a second console to be open during test runs, require an additional command-line flag to be passed in, and might not have support for less-popular browsers, such as Safari. You have to run a single update command when a browser has a major release.

  • Mac users, consider using webdriver-manager, which supports Chrome and Firefox:

    1. In a new console, run npm install -g webdriver-manager (if that gives you permissions errors, put sudo before npm).
    2. Run webdriver-manager update to download the latest versions of everything.
    3. Run webdriver-manager start. This must always be running when executing tests.
  • Windows users, consider using selenium-standalone, which supports Chrome, Firefox, IE, and Edge:

    1. In a new console, run npm install -g selenium-standalone (if that gives you permissions errors, put sudo before npm).
    2. Run selenium-standalone install to download the latest versions of everything.
    3. Run selenium-standalone start. This must always be running when executing tests.

Whenever running Smashtest, always run it as smashtest --test-server=http://localhost:4444/wd/hub (or whatever the port is), or include that flag in the config file.

If you upgrade your browser, especially Chrome, make sure you run the update/install command (step 2) to make sure your drivers are in sync with the browser versions you have installed.

Option 2: Manual install

This option allows you to run everything from just one console, handles any browser, and doesn't require you to pass in additional command-line flags, but the install takes longer, and you have to do a manual update when a browser has a major release.

  1. You'll need to download individual executables (called "drivers") for each browser you want to automate:

    ChromeFirefoxEdgeIE

    Note: Safari 10+ on MacOS comes pre-installed with SafariDriver

  2. Each driver executable should be placed in your system's PATH

    • On MacOS that's usually /usr/local/bin (run echo $PATH to find out)
    • On Windows that's usually C:\Program Files (run echo %PATH% to find out)
  3. Download the latest version of selenium standalone. For example, click the 3.9 folder, then download selenium-server-standalone-3.9.1.jar.

  4. Set the SELENIUM_SERVER_JAR environment variable to the absolute path of the jar file you just downloaded.

    • On MacOS add the line export SELENIUM_SERVER_JAR=[path here] to the file ~/.bash_profile, then restart your console
    • On Windows run setx -m SELENIUM_SERVER_JAR "[path here]" (you may have to start a second command prompt as administrator), then restart your original command prompt

If you upgrade your browser, especially Chrome, make sure you re-download and install the correct driver versions.

Option 3: Cloud service or Grid

Alternatively, you can run Smashtest with a Selenium Grid. This is an advanced, distributed configuration.

  • To use a grid from a cloud service (Sauce Labs, BrowserStack, etc.):

    1. Set up the capabilities in your test. See your cloud service documentation for details.
    2. Set --test-server to your cloud service's url, or include that flag in the config file.
  • To install a grid locally:

    1. Follow steps 1-4 under Option 2 (but do not set the SELENIUM_SERVER_JAR variable)
    2. In a console, run java -jar selenium-server-standalone-[version].jar -role hub and keep it running
    3. In another console, run java -jar selenium-server-standalone-[version].jar -role node -hub http://localhost:4444/grid/register and keep it running
    4. Whenever running Smashtest, always run it as smashtest --test-server=http://localhost:4444/wd/hub (or whatever the port is), or include that flag in the config file

    See the grid page for more information.

3. Install Smashtest

From a console, run npm install -g smashtest

(if that gives you permissions errors, put sudo before npm)

4. Install a grammar (highly recommended)

A grammar will highlight your code, making your .smash files pretty and much easier to read. They are currently available on the Atom and VSCode editors.

Atom

  1. Download Atom
  2. Install the Smashtest package
  3. Set configuration for smash files
    1. Ctrl/Cmd + Shift + P
    2. Type "open config" and hit enter (this will open your config.cson file)
    3. Add the following to the bottom of the file:
        1
        2
        3
        4
        5
      ".smash.source":
      editor:
      autoIndentOnPaste: false
      commentStart: "//"
      tabLength: 4

VSCode

Writing your first test

Here's what to do

  1. Create a new text file called "helloworld.smash"
  2. Put the following into the file:
      1
      2
      3
      4
      5
    Open Chrome
    Navigate to 'google.com'
    Type 'hello world[enter]' into 'textbox'
  3. Open a console and cd to that file's directory
  4. Run smashtest
  5. Watch test run
  6. Open live report while test is running (see console output for path)

So what did I just do?

Smashtest comes with a lot of built-in steps, a few of which you just used.

Think of steps as being arranged like comments on a website. Just like comments are indented to the comment they're replying to, steps are indented to the step they come after. In this example, there's only one line of steps, hence there's only one test. We call tests branches.

So let's add a branch!

  1
  2
  3
  4
  5
  6
  7
Open Chrome
Navigate to 'google.com'
Type 'hello world[enter]' into 'textbox'
Type 'hello universe[enter]' into 'textbox'

Notice line 7. It's at the same indent level as line 5. That means both lines will follow line 3.

Therefore, this file will have two branches:

  1. Open Chrome, Navigate to 'google.com', Type 'hello world[enter]' into 'textbox'
  2. Open Chrome, Navigate to 'google.com', Type 'hello universe[enter]' into 'textbox'

Try running it yourself. Then, give yourself a pat on the back!

Debug and learn

Try this debugging technique...

  1
  2
  3
  4
  5
  6
  7
Open Chrome
Navigate to 'google.com'
~ Type 'hello world[enter]' into 'textbox'
Type 'hello universe[enter]' into 'textbox'

Put a ~ in front of line 5.

That single branch will be isolated, the browser will run non-headless (you can see it), and the test will pause before that line. You can now use the console to try out steps, move to the next step, repeat a previous step, etc.

This mode is called the REPL. It's great for learning and debugging.

Development technique

Using ~ is actually a recommended test development technique:

  1. Write a step
  2. Put a ~ at the end of it
  3. Run smashtest, which runs the browser to that line
  4. Come up with all the permutations that can branch off from that point (using the browser as a guide)
  5. List them as steps, indented to the step with the ~
  6. Repeat

Examples

Feel free to play with this UI test example and this API test example.

Basic language syntax

Branches

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
Open Chrome // executed in both branches
Navigate to 'site.com' // executed in both branches
Click 'one' // branch 1 ends here
Click 'two' // branch 2 ends here
// produces branches:
// 1) open, nav, click 'one'
// 2) open, nav, click 'two'
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
Open Chrome
Navigate to 'google.com'
Do a search // this step ends branch 1
Navigate to 'pets.com'
Buy cat food // this step ends branch 2
Buy dog food // this step ends branch 3

Step blocks

  1
  2
  3
  4
  5
  6
  7
Open Chrome // this group of 5 steps is known as a "step block" (same indent, no blank lines in between)
Open Firefox
Open Safari
Open Edge
Open IE
// one empty line under a step block is mandatory
Navigate to 'site.com' // 5 separate branches end in this step

Sequential

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
..
Open Chrome
Nav to 'site.com'
Click 'button'
// is the same as
Open Chrome
Nav to 'site.com'
Click 'button'

Textual steps

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
This step is a function call // it executes an action
- This step is a textual step // it's just a piece of text to organize your tests
Look, I can put the "-" modifier at the end too! -
Navigate to 'site.com'
- Logged-in tests
// etc.
- Logged-out tests
// etc.

Code blocks

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
Open Chrome
Navigate to 'site.com'
Click the logo {
// this is a code block
// you can do anything js or nodejs supports
(await $('#logo')).click();
browser.driver; // webdriverjs's WebDriver object
} // must end at the same indent level as the starting line

Functions

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
Open Chrome
Navigate to 'site.com'
Click 'element' // all 3 steps are function calls to built-in ("packaged") functions
* Log In // this is a function declaration
Click 'log in box'
Type 'username' into 'username box'
Click 'ok'
Open Chrome
Navigate to 'site.com'
Log In // this is a function call (it will execute the 3 login steps)
Log Out // another function call
log out // steps are case insensitive
* Log Out {
// this is a code block
(await $('.logout-button')).click();
}

Variables

  1
  2
  3
  4
  5
  6
{username} = 'superman' // this sets the global variable 'superman'
{username} is 'superman' // same as above
{username} is "superman" // same as above
{username} is [superman] // same as above
Type '{username} is a handsome guy' into 'textbox'

ElementFinders

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
Open Chrome
Navigate to 'https://www.site.com'
On homepage {
// describe props, which are things on the page and/or the state of those things
props({
'message box': `.textbox, enabled`, // inside the `` is an ElementFinder,
// a special syntax for finding elements
'login button': `#login`,
'search results': `
#list
.result, 'one'
.result, 'two'
`,
'groovy': `'contains this groovy text'`,
'retro button': `selector '.button', groovy`
});
}
Type 'hello world' into 'message box' // using a prop makes it easier to read and refactor

Examples

Web UI

TodoMVC tests  •  what's being tested

REST API

OnWater API tests  •  what's being tested

How to read and run

main.smash is the main entry point for tests in both examples.

To run an example, clone the project, then run smashtest inside the example's directory.

Running Tests

Command line

Run smashtest [files] [options]

[files] can be a filename or a glob (e.g., tests/*.smash)
[options] are explained here.

All files are concatenated into one big piece of text, and everything at indent 0 (e.g., function declarations) is accessible to all the other files.

After branches are compiled, they are all run.

smashtest will run all .smash files in the current directory (but not sub-directories, unless --recursive is set).

Ordering of branches

Branches are run in order of frequency, high to low.

Otherwise, branches are shuffled randomly. Here's why:

  1. You can stop the run at any time and get a good sampling of different functionality
  2. Diversifies the browsers and devices used at any given time, so as to maximize the capabilities of a grid (such as selenium grid)

Disable with the --random=false flag.

Headless

For UI testing, browsers will run headless by default, except if you're debugging (in which case they will run visibly).

Safari, IE, and Edge don't support headless, so they always run visibly.

Override with the --headless=[true/false] flag.

Command-line options

Command-line flags vs. config file

Pass in options via command-line flags (e.g., smashtest test.smash --headless=false --max-parallel=7) or by setting them in a config file.

The config file must be in the same directory where smashtest is called from, must be valid json, and must be called smashtest.json:

  1
  2
  3
  4
{
"headless": false,
"max-parallel": 7
}

Command-line flags will override config file options when they conflict.

List of options

--debug=[hash]

Only run the branch with the given hash, in debug mode. Ignore $'s, ~'s, groups, and frequency.

--groups=[value] or --groups="group1,group2+group3"

Only run branches that are part of one of the groups covered by the expression.

Some group names are set automatically. For example, you can do --groups=chrome, --groups=firefox, --groups=safari, --groups=ie, --groups=edge to only run branches that use those browsers.

+ = AND, , = OR, where AND takes precendence. In other words, --groups="one,two+three,four" means run branches part of group one, or both two and three, or four.

--g:[name]=[value]

Sets a global variable before every branch.

--headless=[true/false]

Whether to run browsers as headless. Overrides default headless behavior (which is headless unless debugging).

--help or -?

Show a help prompt

--max-parallel=[N]

Do not run more than N branches simultaneously. 5 by default.

If you're hitting a selenium grid (i.e., with --test-server), set this high enough so as to max out the available slots (capabilities). Tests will block on Open [browser] if a capability isn't available yet.

--max-screenshots=[N]

Do not take more than N screenshots. No limit by default.

Screenshots taken but deleted after a branch passed do not count against N.

--min-frequency=[high/med/low]

Only run branches at or above this frequency. Set to "med" if omitted.

--no-debug

Fail if there are any $'s or ~'s. Useful to prevent debugging in CI.

--output-errors=[true/false]

Whether to output all errors to console. True by default.

Good for when you're developing tests and expect a few to fail (goes well with -s). Also good for unit tests that run quick and you don't want to switch to the report each time.

--p:[name]=[value]

Sets a persistent variable.

--random=[true/false]

Whether to randomize the order of branches. True by default.

--recursive

Scan the current directory and subdirectories for .smash files (when no filenames are passed in). Without this flag, only the current directory is scanned.

--repl or -r

Opens the REPL (drive Smashtest from command line)

--report-domain=[domain or domain:port]

Domain where the report server should run. This is where the report page gets its live updates.

Port indicates what port smashtest should run on. If omitted, an open port will be chosen, starting with port 9000.

Domain indicates the domain of the machine you're running smashtest on. When this option is omitted, localhost is chosen by default. Choose a domain other than localhost for when you're running tests in CI and want people to hit the report externally.

--report-history=[true/false]

Whether to keep a history of all reports by date/time. If true, will output each report and passed-data file to smashtest/reports/[datetime]/. Otherwise, will output each report and passed-data to smashtest/, possibly overriding the ones from the previous run.

--report-path="[absolute path]"

Sets a custom absolute path for the report and passed-data. Normally these files are outputted to smashtest/ (from the current directory), but if this flag is set, they will be outputted to [absolute path]/smashtest/.

--report-server=[true/false]

Whether to run a server during run for live report updates. Default is true.

--screenshots=[true/false]

Whether to take screenshots before and after each step. Default is false.

--skip-passed=[true/false/filename], -s, -a

Doesn't run branches that passed last time. Carries them and their passed state over into the new report.

Useful when you're fixing a breaking test and don't want to rerun the whole suite again, or when you only want to run new or updated tests.

  • --skip-passed=true
    don't run branches that passed last time (carry them over), and look to ./smashtest/passed-data for the data

  • --skip-passed=false
    run all branches that are supposed to run

  • --skip-passed=[filename]
    don't run branches that passed last time, and use the given file as the record

  • -s
    same as --skip-passed=true

  • -a
    same as --skip-passed=false

Consider copying smashtest/passed-data to a different filename from time-to-time in order to save it. Then restore it by copying that file back in. This is because every new run will overwrite whatever's in smashtest/passed-data.

--step-data=[all/fail/none]

Keep step data for all steps, only failed steps, or no steps. Set to 'all' by default. Includes screenshots. Helps reduce size of reports.

--test-server=[url]

Location of the test server (e.g., http://localhost:4444/wd/hub for selenium server).

--version or -v

Outputs the version of Smashtest.

More memory

If you need more memory, set the NODE_OPTIONS nodejs variable.

For example, in MacOS you'd enter export NODE_OPTIONS="--max_old_space_size=[max MB to use]" && smashtest [files]

Selective test running

-s

You can skip running branches that passed in the previous run by running smashtest -s or by using the --skip-passed=true flag.

Useful when you're fixing a breaking test and don't want to rerun the whole suite again, or when you only want to run new or updated tests.

Only modifier ($)

  1
  2
  3
  4
  5
  6
  7
  8
  9
Open Chrome
Navigate to 'google.com'
Do a search
$ Navigate to 'pets.com' // only runs branches that pass through this step
Buy cat food // i.e., the branch ending here
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
Open Chrome
$ Open Firefox
Open Safari
Desktop
$ Mobile
Navigate to 'google.com'
Do a search // only runs the Firefox mobile branch that ends here
  1
  2
  3
  4
  5
  6
  7
  8
Open Chrome
Open Firefox $
Open Safari
Desktop
Mobile $
Navigate to 'google.com' ~ // use to help isolate a branch for a debug

Groups

  1
  2
  3
  4
  5
  6
  7
  8
Open Chrome
Open Firefox
Navigate to 'google.com' #google #happy-path
Do a search
Navigate to 'pets.com' #pets #happy-path
Buy cat food

Only run Google branch: smashtest --groups=google

Only run branches in happy path OR pets: smashtest --groups="happy-path,pets"

Only run branches in happy path AND pets: smashtest --groups="happy-path+pets"

Only run Firefox branches: smashtest --groups=firefox (built-in group)

Only run branches in happy path AND pets, or firefox AND pets: smashtest --groups="happy-path+pets,firefox+pets"

Frequency

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
Open Chrome
Navigate to 'google.com'
Do something you want tested very often #high // good for quick smoke tests
Do something you want usually tested #med // your normal test suite
Do something you want usually tested // #med is the default freq of a branch if #high/med/low is omitted
Do something you want tested once in a while #low // good for long-running, low-risk, edge-casey stuff
This branch will be med #med #some-group // the later step controls the branch's freq
This branch will be low

Run low/med/high tests: smashtest --min-frequency=low

Run med/high tests: smashtest --min-frequency=med

Run high tests: smashtest --min-frequency=high

Branches are also run in order of frequency, from high to low (and are shuffled within their freq).

These are the reserved hashtags used by smashtest

#desktop, #mobile,

#chrome, #firefox, #safari, #ie, #edge,

#low, #med, #high

CI/CD integration

Test server

If you're running your tests from a test server, such as selenium grid, include smashtest --test-server=[url] with the test server's url. A selenium server, for example, runs on http://localhost:4444/wd/hub by default.

If you're using a cloud service, be sure to set your capabilities in your tests as well.

Report server

Reports are outputted to ./smashtest/report.html from the directory smashtest is run from. Make sure users who access the report have access to the contents of the ./smashtest directory.

To ensure the report can receive live updates during a run, set smashtest --report-domain=[domain:port] where the domain is of the box running smashtest, and port is the port where you want report apis to be accessible.

Flakiness

After running smashtest, run smashtest -s one or more times to rerun failed tests (mitigating environmental or selenium flakiness).

Or, consider running smashtest -s --screenshots=true, so that failed tests always get rerun with screenshots on every step.

Exit codes

The smashtest process exits with exit code 1 if at least one branch failed, 0 otherwise.

No debug

Run smashtest --no-debug to fail run if a ~, ~~, $, exists anywhere in your tests. This way, you're certain you're running the full suite (and nobody accidentally committed their local debug modifiers).

Reports

Location

The live report is outputted to smashtest/report.html from the directory where smashtest was run.

If --report-history=true is set, the report will be at smashtest/reports/smashtest-[timestamp]/report.html.

Failed branches

Similar failed branches are grouped together. This not only shortens the report, it also helps you debug. For example, if upon expanding a group you see that the same test fails regardless of browser, you know the browser isn't the culprit.

Colors

In the report, steps that execute code are colored as passed, failed, running, skipped, or not run yet, while non-executable textual steps and functions are colored in plain gray. Branches themselves have similar colors.

What's in a name?

We believe that tests shouldn't be named.

Tests have a tendency to look very similar to other tests, and it's actually quite common to have branches that only differ by one step. Names tend to just be a crude summary of one or two steps (e.g., "logged-in cart test 016"). Having to think of a name only slows you down.

Smashtest identifies branches by a unique hash, and just shows a list of steps in the report, collapsed by similarity to keep things clean.

If you really want to name a test, use textual steps.

Performance constraints

Due to performance constraints, reports are limited to 500 branches of each type (passed, failed, etc.), and currently running is limited to 20.

Language

Indents

Smash, the language that's run by smashtest, uses exactly 4 spaces for each indent (no tabs).

Each step is indented to the step it follows (like comments on a website).

Blank lines

Blank lines are generally ignored. Use them for stylistic organization, or to group similar steps.

The only time they matter is in a step block. A step block cannot have blank lines between members, and must have one blank line below (if it has children). You can also use blank lines to prevent a step block from forming.

  1
  2
  3
  4
  5
Log in as 'joe' // this is a simple step block
Log in as 'bob'
Log in as 'mary'
Do test stuff

Scope

All files passed to smashtest will be concatenated into one long piece of text at runtime. Everything at indent 0 (e.g., function declarations) will be accessible in all other files.

For example:

  1
  2
  3
  4
// file1.smash
// -----------
Open Chrome
Do test stuff // declared in file2.smash
  1
  2
  3
  4
  5
// file2.smash
// -----------
* Do test stuff
Navigate to 'site.com'
Click 'button'

Modifiers

Modifiers are symbols that come before or after the step:

  1
! - + This step is surrounded by modifiers ~ $ #med -s

Each modifier must be surrounded by spaces.

The only modifier which acts differently based on if it comes before vs. after the step is ~

Step blocks

Simple step blocks

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
Log in as 'joe' // this group of 3 steps is a step block
Log in as 'bob'
Log in as 'mary'
// one empty line under a step block is mandatory
Do test stuff
// produces branches:
// 1) log in as joe, do test stuff
// 2) log in as bob, do test stuff
// 3) log in as mary, do test stuff

Vertical list of 2 or more steps at the same indent, no blank lines in between. Must end in an empty line if it has children.

Multi-level step blocks

Anon

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
open chrome
nav to 'searchengine.com'
[
type 'hello world[enter]' into 'search box'
type 'hello world' into 'search box'
click 'search'
]
verify search results // called after each leaf in the bracketed branches
// produces branches:
// 1) open, nav, type w/ enter, verify
// 2) open, nav, type, click, verify

Named

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
open chrome
nav to 'searchengine.com'
enter search terms [ // same as the first example, but names the step block
type 'hello world[enter]' into 'search box'
type 'hello world' into 'search box'
click 'search'
]
verify search results

Think of square brackets as a "big parenthesis" around multiple steps.

A named step block will look like a function call in the report. They keep things neat.

Always put modifiers before the [.

With code blocks

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
// these 3 steps form a step block, even though they have code blocks
Click one {
(await $('#one')).click();
}
Click two {
(await $('#two')).click();
}
Click three {
(await $('#three')).click();
}
Verify action was completed

Sequential

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
..
Open Chrome
Nav to 'site.com'
Click 'button'
// is the same as
Open Chrome
Nav to 'site.com'
Click 'button'

More on the sequential modifier.

Textual steps (-)

  1
  2
  3
  4
  5
  6
  7
  8
  9
- Tests with a logged-in user // a textual step
Log in as 'bob'
Log in as 'mary'
// etc.
- Tests with invalid users // a textual step
Log in as ''
Log in as 'baduser'
// etc.

Textual steps serve to mark and organize. They don't execute anything.

They're gray in the report (to differentiate them from executable steps).

Uses

  • Can be a heading: grouping and describing the steps that come after and/or the whole branch
    • Alternatively, use functions or named step blocks to group multiple steps into one named task. They will appear as collapsible sections in the report.

  • Can be a comment you want to appear in the report
    • For example, to describe the state of the app or test at that moment in time (or a state that's beginning or ending)
    • A requirement, description, id, or name
    • Describe a manual step

  • Can be used to "comment out" an executable step (though it's recommended to use -s or // in this case)

Recommended test structure

Textual steps are great for dividing tests based on category. In this example, the app is divided and further sub-divided into functionalities, ending in function calls that are implemented in another file. Notice how the "When" steps correspond to the category, while the "Givens" and "Thens" vary from test to test.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
Given I am at my note-taking app
- Notes
- Creating
- with a normal string
Given no notes exist
Given notes exist
When a note is created
Then it is properly displayed
- with a whitespace-only string
When a note is created with an empty string
When a note is created with whitespace only
Then no note is created
// ...
- Updating
// ...
- Deleting
// ...
- Login
// ...
- Register
// ...
- Search
// ...

Functions (*, **)

Function calls

  1
  2
  3
  4
  5
  6
  7
Function call here
Function call with 'strings' and "strings" and [strings] as inputs
Function call with {variables} and {{variables}} as inputs
Function call with '{variables} inside strings' as inputs

Function declarations

Public

  1
  2
  3
  4
  5
* Function declaration here
Navigate to 'site.com'
Check 'checkbox'
Function declaration here // function call, runs the nav and check steps above

A function declaration is accessible to all steps under its parent (or all steps, if declared at indent 0).

With inputs

  1
  2
  3
  4
  5
* Function declaration that {{takes}} in {{inputs}}
Navigate to 'site.com/{{takes}}/{{inputs}}'
Check 'checkbox'
Function declaration that 'string input' in {var input} // function call

With brackets

  1
  2
  3
  4
  5
* Log In [ // optional brackets
Click 'login button'
Type 'username' into 'username box'
Click 'sign in button'
]

With code block

  1
  2
  3
  4
  5
* Log In {
(await $('.login-button')).click();
(await $('.username')).sendKeys('username');
(await $('.signin-button')).click();
}
  1
  2
  3
  4
  5
  6
  7
  8
  9
* Log In {
(await $('.login-button')).click();
}
Type 'username' into 'username box'
Click sign-in {
(await $('.signin-button')).click();
}

Multiple branches

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
* Nav to the cart page // has 2 branches, which represent 2 ways of doing this thing
Navigate to '/'
Click 'cart button'
Navigate to '/cart'
Nav to the cart page
Click 'checkout'
// produces branches:
// 1) nav to /, click cart, click checkout
// 2) nav to /cart, click checkout
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
* Choose a browser
Open Chrome
Open Firefox
* Choose a viewport
Desktop
Mobile
Choose a browser
Choose a viewport
Do a test
// produces branches:
// 1) open chrome, desktop, do a test
// 2) open firefox, desktop, do a test
// 3) open chrome, mobile, do a test
// 4) open firefox, mobile, do a test

Declarations inside declarations (calling in context)

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
* On Desktop // when this is called, all child function declarations are made accessible to future steps
* On Homepage
* Logout
// logout actions for desktop homepage
* On Cart page
* Logout
// logout actions for desktop cart page
* On Mobile
* On Homepage
* Logout
// logout actions for mobile homepage
* On Cart page
* Logout
// logout actions for mobile cart page
* On Desktop
Desktop // built-in step for desktop viewport
* On Mobile
Mobile // built-in step for mobile viewport
..
Open Chrome
On Desktop
Navigate to '/cart'
On Cart page
Logout // executes the logout for the desktop cart page

Gherkin

  1
  2
  3
  4
  5
  6
  7
* I log in // matches all 4 function calls below
// etc.
Given I log in // if an exact match cannot be found, gherkin (given/when/then/and) is stripped
When I log in
Then I log in
And I log in

{var} = F

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
* Choose a username
{x} = 'bob'
{x} = 'joe'
{x} = 'mary'
{username} = Choose a username
Type {username} into 'username box'
// produces branches:
// 1) type 'bob'
// 2) type 'joe'
// 3) type 'mary'

Private

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
* On cart page
** Private function // not made accessible after a call to "On cart page"
// etc.
Private function // call is ok
Something
Private function // call is ok
On cart page
Private function // compile-time error

Hooks

  1
  2
  3
*** Before Every Branch {
// stuff to do before every branch begins
}

More on hooks, and a full list of supported ones.

Patterns

Encapsulating and refactoring

  1
  2
  3
  4
  5
  6
  7
  8
* Order dinner // multiple branches for different ways of accomplishing the same thing
- Variant 1
Add beans to meal
Add rice to meal
- Variant 2
Add rice to meal
Add beans to meal

Organizing

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
// main-tests.smash - bird's eye view of all tests helps ensure we have full coverage
// ----------------
- Test app
- Homepage tests
Display homepage test
- Cart tests
Empty cart test
Full cart test
- Search tests
Empty search test
Base case search test
  1
  2
  3
  4
  5
  6
  7
// cart-tests.smash
// ----------------
* Empty cart test
// etc.
* Full cart test
// etc.

Dividing a single declaration into multiple files

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
// logout.smash
// ------------
* On Desktop // this func declaration split into multiple files (to keep logout together)
* On Homepage
* Logout
// etc.
* On Mobile
* On Homepage
* Logout
// etc.
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
// search.smash
// ------------
* On Desktop // this func declaration split into multiple files (to keep search together)
* On Homepage
* Do a search
// etc.
* On Mobile
* On Homepage
* Do a search
// etc.

"On" pattern

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
// Call a function starting with "On" to indicate that this is the current state
// (and not an action to be performed, such as navigating there)
// e.g., "On [page]" can do verifications and set up props/functions for that page
* On cart page { // call when on the cart page
// set up ElementFinder props for everything on this page
props({
'list of items': `
#list
.item
.item
.item
`,
'checkout button': `#checkout`
});
}
// do some initial page verifications
Verify at page '/cart'
Verify 'list of items' is visible
// expose cart-related functions
* Add item to cart
// etc.
Open Chrome
Nav to '/cart'
On cart page
Add item to cart

Enforce permutations

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
// this pattern ensures that all permutations of these function calls are implemented below
On Desktop
On Mobile
Logged In
Logged Out
Verify something
// compile-time error if we're missing a permutation here:
* On Desktop
* Logged In
* Verify something
// etc.
* Logged Out
* Verify something
// etc.
* On Mobile
* Logged In
* Verify something
// etc.
* Logged Out
* Verify something
// etc.
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
* Try every string permutation // calling this function ensures that all 3 permutations
String is empty // are implemented directly below, so you don't forget
String is whitespace
String is normal
// in another file...
Type {string:} into 'textbox'
Try every string permutation // error if any of the 3 are missing
* String is empty
{string} = ''
// ...
* String is whitespace
{string} = ' '
// ...
* String is normal
{string} = 'normal'
// ...

Rules for matching calls to declarations

Simple case

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
Open Chrome // 4) looks among children of this step, finds line 14
Navigate to '/page1' // 3) looks among children of this step, finds nothing
Login // 2) looks among children of this step, finds nothing
My function // 1) function call *** START HERE ***
Click 'something'
Click 'something else'
Click 'something else'
Navigate to '/page2'
Login
* My function // the one that's matched ("overrides" line 17)
// etc.
* My function
// etc.

For every function call, searches for a matching function declaration among the children of that call's parent. If nothing is found, searches among the grandparent's children, then the great-grandparent's children, etc. until it searches among all declarations at indent 0 before erroring out.

Function calls and declarations are case insensitive, leading and trailing whitespace is ignored, and whitespace in the middle is always treated as a single space.

Don't forget to escape chars!

Since 'strings' and {vars} designate inputs in a function call, always use \', \", \[, \], \{, \} when using those actual characters inside a function call's text, to prevent an input from forming:

  1
  2
Clear user\'s credentials
- Textual step's text // not necessary for textual steps

Matching multiple declarations

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
- Matching multiple function declarations under the same parent
* F
A
* F
B
F // matches both line 2 and line 5
// produces branches:
// 1) F, A
// 2) F, B

Making vars available below

  1
  2
  3
  4
  5
* F
{name} = 'bob' // this variable will be accessible after a function call to F
F
Type {name} into 'textbox' // will type 'bob'

Making funcs available below

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
* F
* A
B
* G
* A
C
* A
D
F // makes public function A at line 2 available below
A // B is run here
F // makes public function A at line 2 available below
G // makes public function A at line 6 available below
A // C is run here

Equivalents

  1
  2
  3
  4
  5
  6
  7
// file1.smash
// -----------
* A
* B
- 1
* B
- 2
  1
  2
  3
  4
  5
// file2.smash
// -----------
* A
* B
- 3

is equivalent to

  1
  2
  3
  4
  5
* A
* B
- 1
- 2
- 3

Calls to itself

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
* Nav to homepage
Nav to '/'
- Some test
Open Chrome
Nav to homepage // calls line 8
* Nav to homepage // this "intercepts" navs to homepage to do security stuff
Do security checks
Nav to homepage // ignores line 8 because recursion not allowed, so calls line 1

Variables

Using

  1
  2
  3
  4
  5
// In a function call
Buy tickets for {num adults} adults and {{num children}} children
// In a string literal
Nav to '{host}/path/name' // {host} is replaced with its value

Setting

{var} = 'str'

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
{variable} = 'string' // everything in Smashtest is a string
{variable} = "string"
{variable} = [string]
{variable} is 'string' // same as =
{variable}='11', {variable}='22', {variable}='33'
{variable}='{host}/path/name'
{variable}='{var2}' // cloned a var
{ Variable Name With Caps and Whitespace } = 'string'

{var} = Func with code block

  1
  2
  3
  4
  5
{variable} = Get hello // variable is set to "hello world!"
* Get hello {
return "hello " + "world!"; // any kind of js value will work (not just string)
}
  1
  2
  3
{variable} = Get goodbye { // variable is set to "goodbye!"
return "goodbye!";
}

{var} = Func with branches

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
* A bad username
{x} = '00' // you can use any variable name, not just {x}
{x} = 'baduser'
{x} = '[none]'
{x} = ''
{x} = ' '
{username} = A bad username
Type {username} into {textbox}
// produces branches:
// 1) type '00'
// 2) type 'baduser'
// 3) type '[none]'
// 4) type ''
// 5) type ' '

Variable names are case sensitive (unlike step names), leading and trailing whitespace is ignored, and whitespace in the middle is always treated as a single space.

They're case sensitive because they're converted to js vars in code blocks (and js is case-sensitive).

Escape sequences

You're not allowed to have a \ in a variable name, and you're not allowed to have a \0, \x, \u, or \c inside a 'string literal'. To get around this, use:

  1
  2
  3
{variable} = special char string {
return "\u2665 \cJ";
}

Types

{Global}

Global variables are accessible to all steps for the rest of the branch (inside and outside function calls).

  1
  2
  3
  4
  5
{variable} = 'string' // global variable
F
* F
Type {variable} into 'textbox' // accessible here
  1
  2
  3
  4
  5
F
Type {variable} into 'textbox' // accessible here
* F
{variable} = 'string'

{{Local}}

Local variables are only accessible inside the current function call.

  1
  2
  3
  4
  5
  6
{{variable}} = 'string' // local variable
F
Type {{variable}} into 'textbox' // accessible here
* F
Type {{variable}} into 'textbox' // NOT accessible here
  1
  2
  3
  4
  5
  6
  7
F
Type {{variable}} into 'textbox' // NOT accessible here
* F
{{variable}} = 'string'
Type {{variable}} into 'textbox' // accessible here
// goes out of scope here

Persistent

Persistent variables exist for the lifetime of the whole suite run. They can only be get and set inside code blocks. They are usually used for internal stuff, like storing functions and libraries.

Inside a code block

Getting

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
{variable} = 'something'
Get a variable {
let v = variable; // just use it as a js variable
// for this to work, variable name must be a single word,
// no chars other than A-Z, a-z, 0-9, - _ .
// and not have the same name as a js keyword
// when different types of vars have the same name,
// local takes precedence over global, which takes precedence over persistent
}
Get a local variable {
let v = l('variable name');
}
Get a global variable {
let v = g('variable name');
}
Get a persistent variable {
let v = p('variable name');
}

Setting

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
Set a local variable {
l('variable name', 'new value'); // any kind of js value will work (not just string)
}
Set a global variable {
g('variable name', 'new value');
}
Set a persistent variable {
p('variable name', 'new value');
}

Lookahead (:)

{var:} will get the value of the variable when it's set later in the branch. This allows you to refactor common steps higher up into the tree.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
Type {username:} into 'login box' // ignores current value of {username}, looks to the first
// {username}='str' line further in the branch
{username} = 'bob'
{username} = 'mary'
{username} = 'vishal'
Verify success
{username} = 'baduser'
{username} = '[none]'
{username} = ''
Verify error
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
Choose {adults:} and {children:} from reservations panel
{adults}='[none]'
{adults}='0'
{children}='[none]'
{children}='0'
{children}='1'
{children}='8'
Verify error
{adults}='1'
{adults}='8'
{children}='[none]'
{children}='0'
{children}='1'
{children}='8'
Verify success
Setting a lookahead var's value

You cannot set a lookahead var's value inside a code block, only in a {var}='something' or a {var}=Function call (but the function call has to be sync - i.e., no awaits and an immediate return).

Code blocks

Types

  1
  2
  3
  4
  5
  6
  7
* Function name {
// this is a code block
// you can do anything js or nodejs supports
// only lines between the "{" and "}" lines are part of the code block
} // must end at the same indent level as the starting line
  1
  2
  3
  4
Step name {
// this step is implemented in here
// kind of like a one-time function call
}

Modifiers

  1
  2
  3
  4
  5
  6
// Modifiers can come before the name, or after the name but before the "{"
.. + * Function name $ ! {
}
.. + Step name $ ! {
}

Await

  1
  2
  3
  4
// Code blocks are async, meaning you can use the 'await' keyword
Verify success {
await $('#success');
}
Don't forget await!

Async js function calls should always be preceded with await. If errors are thrown but aren't captured inside a step, or if things just seem wonky, it's probably because you forgot an await.

Prev

  1
  2
  3
  4
  5
  6
  7
// Pass a value from one code block to the next via 'return' and 'prev'
Step one {
return "hello";
}
Step two {
expect(prev).to.equal("hello"); // chai's expect and assert are available by default
}
  1
  2
  3
  4
  5
  6
  7
  8
  9
Step one {
return "hello";
}
Some intermediate step
Step two {
expect(prev).to.equal("hello");
}

Variables

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
Settings vars {
l('var1', 'new value'); // local variable (any value will work, not just strings)
g('var2', 'new value'); // global variable
p('var3', 'new value'); // persistent variable
}
Getting vars {
let v = var1; // works if variable name is a single word with no special chars
let v = l('var1'); // local variable
let v = g('var2'); // global variable
let v = p('var3'); // persistent variable
}
Use l()/g()/p() to set variables

Don't set a {variable} with variable = 'value'; as this will not persist past the end of the code block.

Be mindful when using 'let'

Don't use 'let' to declare a new variable with the same name as an existing {variable}, or you'll get a runtime error (double-declaration). You can use 'var' to get around this.

Timeouts

The default timeout for a step is 60 secs. If it hasn't completed by then, it will fail with a timeout error.

Steps generally have their own, more stringent timeouts as well. These are usually implemented by an (async) function called within the step that has its own timeout. We made the default timeout really high since we wanted to leave the more "realistic" timeout to each step's individual discretion.

The default timeout can be changed via setStepTimeout(secs) for all steps after the current one in the current branch.

Errors

Normal error

  1
  2
  3
Failing step {
throw new Error("oops"); // this step and branch will fail and end immediately
}

Error.continue

  1
  2
  3
  4
  5
Failing step {
let e = new Error("verify failed, but let's try the next verify anyway");
e.continue = true; // this error will fail the step and branch,
throw e; // but the branch will continue running
}

Error.fatal

  1
  2
  3
  4
  5
Failing step {
let e = new Error("something really bad");
e.fatal = true; // this error will end the whole test suite execution immediately
throw e;
}

Stack traces

A stack trace will contain something like this, where [LINE NUMBER] represents the line in the .smash file where the error occurred:

at CodeBlock_for_[NAME OF CODE BLOCK FUNCTION] (eval at ...), <anonymous>:[LINE NUMBER]:[COL NUMBER]

Implement complex functions in their own js files to generate more traditional stack traces:

  1
  2
  3
  4
  5
  6
// test.smash
// ----------
Step {
const yf = i('yf', './yourfile.js');
yf.something();
}
  1
  2
  3
  4
  5
  6
// yourfile.js
// ----------
function something() {
// etc.
}
module.exports.something = something;

Code reference

These js functions and objects are accessible within any code block.

c()

  1
  2
c('print to console'); // same as console.log(), but prints it out more neatly
// (and clear of the progress bar)

dir()

  1
dir(); // returns the absolute directory of the file where this step is

g()

  1
  2
g('variable', 'new value'); // set global variable
g('variable'); // get global variable

getGlobal()

  1
getGlobal('variable'); // get global variable

getLocal()

  1
getLocal('variable'); // get local variable

getPersistent()

  1
getPersistent('variable'); // get persistent variable

i()

  1
  2
  3
  4
  5
  6
  7
  8
  9
const packageName = i('package-name'); // require()'s (imports) the given nodejs package,
// sets persistent var 'packageName' to it (auto-camelCased),
// and returns it
const myPkg = i('myPkg', 'package-name'); // same, but sets the name of the persistent var
i('myPkg', './path/to/file.js'); // works with js files too
i('myPkg', '../path/to/file.js');
i('myPkg', '/Users/Shared/tests/file.js');

l()

  1
  2
l('variable', 'new value'); // set local variable
l('variable'); // get local variable

log()

  1
log('text to log'); // logs a piece of text to the report for this step

p()

  1
  2
p('variable', 'new value'); // set persistent variable
p('variable'); // get persistent variable

runInstance

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
runInstance; // represents the test runner "thread" that's
// running this step and branch (see RunInstance)
runInstance.runner; // represents the test runner (see Runner)
runInstance.tree; // represents the whole tree (see Tree)
runInstance.currBranch; // represents the current branch (see Branch)
runInstance.currStep; // represents the current step (see Step)
// These objects can be used to dynamically view, create, and/or edit tests at runtime

setGlobal()

  1
setGlobal('variable', 'new value'); // set global variable

setLocal()

  1
setLocal('variable', 'new value'); // set local variable

setPersistent()

  1
setPersistent('variable', 'new value'); // set persistent variable

setStepTimeout()

  1
setStepTimeout(30); // sets step timeout to 30 secs for all steps in this branch after this one

Sequential (..)

.. above a step block

Simple case

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
Nav to '/page'
.. // makes the whole step block run sequentially
Type '1111' into 'textbox'
Type '2222' into 'textbox'
Type '3333' into 'textbox'
Verify success
// produces 1 branch:
// 1) nav, type 1111, type 2222, type 3333, verify success

Function calls as step block members

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
// Note: Acts differently from function calls under a .. step.
// If a function call has multiple branches, multiple branches will be generated:
* Go to cart
// two different ways of getting to the cart
Nav to '/cart'
Click 'cart icon'
..
Go to cart
Add peanuts to cart
Verify peanuts added
// produces branches:
// 1) nav to /cart, add peanuts, verify peanuts
// 2) click cart icon, add peanuts, verify peanuts

.. on a step

Simple case

  1
  2
  3
  4
  5
  6
  7
  8
  9
Sequential test .. // flatten branches at or below me into one sequential branch
One
Two
Three
Four
Five
// produces 1 branch:
// 1) sequential test, one, two, three, four, five

With a step block

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
Nav to '/page' ..
Type '1111' into 'textbox'
Type '2222' into 'textbox'
Type '3333' into 'textbox'
Verify success
// produces 1 branch:
// 1) nav, type 1111, verify success, type 2222, verify success, type 3333, verify success

With a function call

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
* Type in
Type '1111' into 'textbox'
Type '2222' into 'textbox'
Type '3333' into 'textbox'
Nav to '/page' ..
Type in
Verify success
// produces 1 branch:
// 1) nav, type 1111, verify success, type 2222, verify success, type 3333, verify success
Nav to '/page'
Type in .. // all we did was move the .. down one line
Verify success
// produces the same branch as before

Inside a function declaration

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
* Open cart .. // the 3 steps here execute sequentially
Nav to '/'
Click 'cart icon'
Verify 'cart' is visible
Open cart // it's sequential inside, but not sequential out here
Do stuff
Do more stuff
// produces 1 branch:
// 1) open cart (nav to /, click cart icon, verify cart), do stuff, do more stuff

Non-parallel (!, !!)

!

  1
  2
  3
  4
  5
{username} is 'pete' ! // no two branches going through this step may execute simultaneously
Nav to '/' // useful for testing a stateful shared resource, like a test account
// etc.
Nav to '/page1'
// etc.

!!

  1
  2
  3
  4
  5
{username} is 'bob' !! // no two branches going through this step may execute simultaneously,
Nav to '/' // unless --test-server was set
// etc. // useful for things like safaridriver, which can't run more
Nav to '/page1' // than one instance locally, but can handle multiple instances
// etc. // in a selenium grid

Comments

Standard use

  1
  2
  3
// full-line comment
Step // comment at the end of a step

Comments ignore whole lines

  1
  2
  3
  4
  5
  6
  7
  8
  9
Open Chrome // this is still a valid step block
// Open Firefox // if whole line starts with //, it's ignored as if it weren't there
Open Safari
Do something
// produces branches:
// 1) open chrome, do something
// 2) open safari, do something

Inside code blocks

  1
  2
  3
  4
Code block step {
// normal js comments occur here
/* normal js comments occur here */
}

Groups and freq (#)

Groups

  1
  2
  3
  4
  5
  6
  7
  8
Open Chrome
Open Firefox
Navigate to 'google.com' #google #happy-path
Do a search
Navigate to 'pets.com' #pets #happy-path
Buy cat food

Only run Google branch: smashtest --groups=google

Only run branches in happy path OR pets: smashtest --groups="happy-path,pets"

Only run branches in happy path AND pets: smashtest --groups="happy-path+pets"

Only run Firefox branches: smashtest --groups=firefox (built-in group)

Only run branches in happy path AND pets, or firefox AND pets: smashtest --groups="happy-path+pets,firefox+pets"

Frequency

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
Open Chrome
Navigate to 'google.com'
Do something you want tested very often #high // good for quick smoke tests
Do something you want usually tested #med // your normal test suite
Do something you want usually tested // #med is the default freq of a branch if #high/med/low is omitted
Do something you want tested once in a while #low // good for long-running, low-risk, edge-casey stuff
This branch will be med #med #some-group // the later step controls the branch's freq
This branch will be low

Run low/med/high tests: smashtest --min-frequency=low

Run med/high tests: smashtest --min-frequency=med

Run high tests: smashtest --min-frequency=high

Branches are also run in order of frequency, from high to low (and are shuffled within their freq).

These are the reserved hashtags used by smashtest

#desktop, #mobile,

#chrome, #firefox, #safari, #ie, #edge,

#low, #med, #high

Conditional tests

It's best to avoid conditional tests. They should only be used to implement rare exceptions.

If step

  1
  2
  3
  4
  5
If A then B {
if(A) { // only does B if A is true
B();
}
}

If browser is...

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
- Test something
Do not allow Safari {
if(browser.params.name == 'safari') {
// pass and end the whole branch if the browser is safari, do nothing otherwise
runInstance.currBranch.markBranch('pass');
}
}
- This step only runs if the browser isn't safari

If viewport is...

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
- Test something
If mobile {
if(!runInstance.currBranch.groups.includes('mobile')) {
// pass and end the whole branch if the viewport isn't mobile, do nothing otherwise
runInstance.currBranch.markBranch('pass');
}
}
- This step only runs if the viewport is mobile

Skipping (-, -s, .s, $s)

Skip one step

-s (recommended)

  1
  2
  3
One // runs
Skipped step -s // doesn't run, marked skipped in report
Two // runs

-

  1
  2
  3
One // runs
Skipped step - // doesn't run, regular textual step in report
Two // runs

//

  1
  2
  3
One // runs
// Skipped step // doesn't run, not outputted to report
Two // will run, but needs to be dedented once

Skip all branches passing through a step

$s

  1
  2
  3
  4
  5
  6
  7
One // doesn't run
Two // doesn't run
Skipped step $s // skips any branch that passes through this step, still expands function calls (error if declaration not found)
Three // doesn't run
Four // doesn't run

Skip step and all steps below

.s

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
One // runs
Two // runs
Skipped step .s // doesn't run, still expands function calls (error if declaration not found)
Three // doesn't run
Four // doesn't run
// Also skips entire duplicate branches caused by .s
// Be careful, when .s is inside a function declaration it will skip steps after the function call as well

//

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
One // runs
Two // runs
// Skipped step // doesn't run
//
// Three // doesn't run
// Four // doesn't run
// Useful if you don't want function calls to expand,
// but won't remind you in the console/report that skipped steps exist

// on step block member

  1
  2
  3
  4
  5
One // runs
// Two // doesn't run, and doesn't run any steps below
Three // runs
Four // runs, except for "Two"

Collapsing (+, +?)

Collapse (+)

  1
  2
  3
  4
  5
  6
* Select best item from dropdown + // function calls here will be collapsed by default in the report
Click 'dropdown'
Scroll to 'best item'
Click 'best item'
Some big operation with lots of steps + // this function call will be collapsed by default in the report

If there's an error inside a +'ed function, or if the function is currently running, it will be uncollapsed automatically.

It's recommended to mark precondition steps with +, since their details aren't central to the test.

Hidden (+?)

  1
  2
  3
  4
  5
* Init +? // function calls here will be hidden in the report
Do some internal stuff
// this function call will be hidden in the report
Some internal thing somebody reading the report doesn't care about +?

If there's an error inside a +?'ed function, it will be visible in the report.

Only ($)

One $

  1
  2
  3
  4
  5
  6
  7
  8
  9
Open Chrome
Navigate to 'google.com'
Do a search
$ Navigate to 'pets.com' // only runs branches that pass through this step
Buy cat food // i.e., the branch ending here

Multiple $'s at different indents

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
Open Chrome
$ Open Firefox
Open Safari
Desktop
$ Mobile
Navigate to 'google.com'
Do a search // only runs the Firefox mobile branch that ends here

Multiple $'s with the same parent

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
Open Chrome
$ Open Firefox
$ Open Safari
Desktop
Mobile
Navigate to 'google.com'
Do a search // only runs the Firefox and Safari branches (4 of them)

On a function declaration

  1
  2
  3
  4
  5
  6
  7
  8
$ * Do a search // only runs branches that call this function
// etc.
Log in as 'bob'
Do a search
Log in as 'vishal'
Do a search

With ~'s

  1
  2
  3
  4
  5
  6
  7
  8
Open Chrome
Open Firefox $
Open Safari
Desktop
Mobile $
Navigate to 'google.com' ~ // use to help isolate a branch for a debug

Debug (~, ~~)

Debug modifier (~)

  1
  2
  3
Navigate to 'google.com'
~ Click 'button' // isolate this branch, run in REPL, and pause right before this step
Click 'other thing'
  1
  2
  3
Navigate to 'google.com'
Click 'button' ~ // isolate this branch, run in REPL, and pause right after this step
Click 'other thing'

A branch run in debug will also pause after a failing step.

If a ~ goes through multiple branches, the first one is chosen.

No report is generated and the list of previously passed branches (for -s) is retained.

Express debug modifier (~~)

  1
  2
  3
Navigate to 'google.com'
~~ Click 'button' // only run this branch, but no pausing and don't run in REPL
Click 'other thing'

Just like for ~, no report is generated and the list of previously passed branches (for -s) is retained.

Debug flag

You can also run smashtest --debug=[hash] to debug the branch with the given hash.

Be careful. If you change any step, the hash of its branch will change. The recommended technique is to place a $, ~~, or ~ on a line you change, then run that. You'll want to rerun all the branches that pass through that line anyway.

Hooks (***)

What's a hook?

A hook is a code block function that runs before or after a step, branch, or test suite.

They're not meant for testing or setup/teardown. They're for internal stuff, such as reporting, importing code, js function declarations, screenshots, and logging.

Only code blocks are allowed, and they cannot have children or modifiers. Hooks aren't listed in the report. If a hook fails, the step/branch it corresponds to will take the error and fail.

Types

Before Every Branch

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
Parent step
A
B
*** Before Every Branch {
// this code runs before every branch that goes through the parent step
// i.e.,
// 1) this code, parent step, A
// 2) this code, parent step, B
}
*** Before Every Branch {
// this code runs before every branch in existence
}

After Every Branch

  1
  2
  3
  4
  5
  6
  7
  8
  9
Parent step
*** After Every Branch {
// this code runs after every branch that goes through the parent step (whether it passes or fails)
}
*** After Every Branch {
// this code runs after every branch in existence (whether it passes or fails)
}

Before Every Step

  1
  2
  3
  4
  5
  6
  7
  8
  9
Parent step
*** Before Every Step {
// this code runs before every step of every branch that goes through the parent step
}
*** Before Every Step {
// this code runs before every step of every branch in existence
}

After Every Step

  1
  2
  3
  4
  5
  6
  7
  8
  9
Parent step
*** After Every Step {
// this code runs after every step of every branch that goes through the parent step
}
*** After Every Step {
// this code runs after every step of every branch in existence
}

Before Everything

  1
  2
  3
  4
*** Before Everything {
// this code runs before the whole test suite begins
// only valid at 0 indents
}

After Everything

  1
  2
  3
  4
*** After Everything {
// this code runs after the whole test suite ends
// only valid at 0 indents
}

Where should setup and teardown logic go?

Setup code should go in the actual test.

Teardown code should go in the same setup logic, such that the previous state is cleaned out prior to actual testing. It should go into a hook only if necessary.

  1
  2
  3
  4
  5
  6
  7
- My test
Setup
Testing
* Setup
Clean previous state
Set things up for new test

UI Testing

Smashtest is a general platform for testing which comes with several built-in packages. One of these packages handles web UI testing with selenium webdriver.

This section discusses the steps and js functions that this package makes available.

Also, check out this UI test example.

Browsers and devices

Open a browser

All UI steps MUST come after an "Open [browser]" step
  1
  2
  3
  4
  5
  6
  7
Open Chrome // run exclusively with --groups=chrome
Open Firefox // run exclusively with --groups=firefox
Open Safari // run exclusively with --groups=safari
Open IE // run exclusively with --groups=ie
Open Edge // run exclusively with --groups=edge
Open browser 'chrome' // use a string recognized by webdriver as a browser name

Set viewport size

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
// Run these exclusively with --groups=desktop
Desktop // sets browser to 1920 x 1080
Laptop // sets browser to 1024 x 768
Laptop L // sets browser to 1440 x 900
// Run these exclusively with --groups=mobile
Mobile // sets browser to 375 x 667
Mobile Portrait // sets browser to 375 x 667
Mobile Landscape // sets browser to 667 x 375
Mobile S // sets browser to 320 x 480
Mobile S Portrait // sets browser to 320 x 480
Mobile S Landscape // sets browser to 480 x 320
Mobile M // sets browser to 375 x 667
Mobile M Portrait // sets browser to 375 x 667
Mobile M Landscape // sets browser to 667 x 375
Mobile L // sets browser to 425 x 667
Mobile L Portrait // sets browser to 425 x 667
Mobile L Landscape // sets browser to 667 x 425
Tablet // sets browser to 768 x 1024
Tablet Portrait // sets browser to 768 x 1024
Tablet Landscape // sets browser to 1024 x 768

Set device type

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
// These steps set the viewport to the given device, and do device emulation in Chrome
BlackBerry Z30 // sets browser to 360 x 640
Blackberry PlayBook // sets browser to 600 x 1024
Galaxy Note 3 // sets browser to 360 x 640
Galaxy Note II // sets browser to 360 x 640
Galaxy S III // sets browser to 360 x 640
Galaxy S5 // sets browser to 360 x 640
Kindle Fire HDX // sets browser to 800 x 1280
LG Optimus L70 // sets browser to 384 x 640
Laptop with HiDPI screen // sets browser to 1440 x 900
Laptop with MDPI screen // sets browser to 1280 x 800
Laptop with touch // sets browser to 1280 x 950
Microsoft Lumia 550 // sets browser to 640 x 360
Microsoft Lumia 950 // sets browser to 360 x 640
Nexus 4 // sets browser to 384 x 640
Nexus 5 // sets browser to 360 x 640
Nexus 5X // sets browser to 412 x 732
Nexus 6 // sets browser to 412 x 732
Nexus 6P // sets browser to 412 x 732
Nexus 7 // sets browser to 600 x 960
Nexus 10 // sets browser to 800 x 1280
Nokia Lumia 520 // sets browser to 320 x 533
Nokia N9 // sets browser to 480 x 854
Pixel 2 // sets browser to 411 x 731
Pixel 2 XL // sets browser to 411 x 823
iPhone 4 // sets browser to 320 x 480
iPhone 5/SE // sets browser to 320 x 568
iPhone 6/7/8 // sets browser to 375 x 667
iPhone 6/7/8 Plus // sets browser to 414 x 736
iPhone X // sets browser to 375 x 812
iPad // sets browser to 768 x 1024
iPad Mini // sets browser to 768 x 1024
iPad Pro // sets browser to 1024 x 1366

Usage examples

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
Open Chrome
Open Firefox
Open Safari
Desktop
- Desktop test 1
// etc.
- Desktop test 2
// etc.
Mobile
- Mobile test 1
// etc.
- Mobile test 2
// etc.
iPhone X
- iPhone X test
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
// Login function with different steps on desktop vs. mobile
// Run with `smashtest` or `smashtest *.smash`
// ----- main.smash -----
Open Chrome
On Desktop
On Mobile
// etc.
Login // the login that's called depends on desktop vs. mobile
// ----- viewports.smash -----
* On Desktop
Desktop // calls `Desktop` step exposed by `Open Chrome`
* On Mobile
Mobile // calls `Mobile` step exposed by `Open Chrome`
// ----- desktop.smash -----
* On Desktop
* Login
- Desktop implementation of Login
// ----- mobile.smash -----
* On Mobile
* Login
- Mobile implementation of Login

Capabilities

Sometimes you need to set custom browser capabilities and options, such as when you need to provide a username and password for a cloud service (BrowserStack, Sauce Labs, etc.)

Set custom capabilities

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
Open Chrome
Set custom capabilities {
g('browser capabilities', {
'name': 'foobar'
// This is the Capabilities object. Capabilities go here.
});
}
// Note: you can set this before or after the "Open [browser]" step

Set custom options

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
Open Chrome
Set custom options {
const chrome = i('selenium-webdriver/chrome');
let opts = new chrome.Options();
// Call functions on opts here
g('browser options', opts);
}
// Note: you can set this before or after the "Open [browser]" step

Browser steps

You must run an Open [browser] step before using any of the steps below.
{{element}} below can either be an ElementFinder string or an existing WebElement object.
All steps that involve mouse interaction (i.e., clicking) choose the first clickable element that matches {{element}}. If nothing found, they choose the first non-clickable element that matches {{element}}.

Interact

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
Click 'elementfinder'
Native click 'elementfinder' // same as click, but uses js click instead of webdriver click
// try this when webdriver click doesn't work
Double click 'elementfinder'
Hover over 'elementfinder'
Scroll to 'elementfinder'
Check 'elementfinder' // clicks the element, if it's currently unchecked
Uncheck 'elementfinder' // clicks the element, if it's currently checked

Set

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
Type 'text' into 'elementfinder'
Type 'text[enter]' into 'elementfinder' // see list of keys (case-insensitive)
Type '[none]' into 'elementfinder' // step does nothing
// good for including inaction when testing different inputs
Clear 'elementfinder' // clear a textbox, etc.
Set 'elementfinder' to 'value'
Set 'elementfinder' to '[none]' // step does nothing
// good for including inaction when testing different inputs
Select '6' from 'elementfinder' // selects an <option> from a <select>
// if an <option> with this exact value cannot be found,
// searches for an <option> that contains the value,
// trimmed and case-insensitive
Select '[none]' from 'elementfinder' // step does nothing
// good for including inaction when testing different inputs
Select element 'option elementfinder' from 'dropdown elementfinder' // selects an <option> from a <select>
{variable} = Value of 'elementfinder'

Navigate

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
Navigate to 'https://site.com/page'
Navigate to 'http://site.com/page'
Navigate to 'site.com/page' // defaults to http
Navigate to '/page' // uses domain browser is currently on
Nav to '/page' // shorthand for Navigate
Go Back
Go Forward
Refresh
{current url} = Current url // returns current absolute url

Window

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
Set dimensions to width='1024' height='768'
Maximize window
Open new tab // opens new tab ("window") and switches to it
Switch to window whose title contains 'hello'
Switch to window whose url contains '/page'
Switch to the '1st' window
Switch to the '4th' window
Switch to iframe 'elementfinder'
Switch to topmost iframe
{window title} = Window title // returns current window title

Alerts

  1
  2
  3
Accept alert // clicks ok in alert modal, error if no alert open
Dismiss alert // clicks cancel in alert modal, error if no alert open

Wait

Avoid this kind of wait. Instead, consider Verify or Wait Until.
  1
  2
  3
  4
Wait '2' secs // sleeps this long
Wait '2' seconds
Wait '1' sec
Wait '1' second

Cookies and storage

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
{cookie} = Get cookie 'name'
Verify cookie {
cookie; // object containing cookie info
cookie.value; // value of cookie
}
Set cookie 'name' to 'value'
Set cookie 'name' to 'value', expiring in '65' secs
Delete cookie 'name'
Delete all cookies
Clear local storage
Clear cookies and local storage

Print and log

  1
  2
  3
  4
  5
Log 'text to log to this step in report'
'elementfinder' // outputs found elements to browser's console
"elementfinder" // and number of found elements to regular console
[elementfinder] // useful when using REPL

Verify steps

You must run an Open [browser] step before using any of the steps below.

Verify

'Verify' steps wait up to 2 secs for the verify to pass before failing.
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
Verify 'elementfinder' is visible
Verify 'elementfinder' is not visible
Verify 'elementfinder' is 'state-elementfinder'
Verify every 'elementfinder' is 'state-elementfinder'
Verify at page 'Page title' // passes if current page title (case-insensitive)
Verify at page 'site.com/page' // or url contains this text
Verify at page '/page'
Verify at page 'Page .*' // passes if current page title
Verify at page '(.*)page' // or url matches this regex
Verify at page '^http(.*)page.+$'
Verify cookie 'name' contains 'value'
Verify alert contains 'hello' // passes if alert is open and contains this text
// Note: this step doesn't wait up to 2 secs - it's immediate

Wait until

'Wait until' steps wait up to 15 secs, or the specified amount, for the verify to pass before failing.
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
Wait until 'elementfinder' is visible
Wait until 'elementfinder' is visible (up to '30' secs)
Wait until 'elementfinder' is not visible
Wait until 'elementfinder' is not visible (up to '30' secs)
Wait until 'elementfinder' is 'state-elementfinder'
Wait until 'elementfinder' is 'state-elementfinder' (up to '30' secs)
Wait until every 'elementfinder' is 'state-elementfinder'
Wait until every 'elementfinder' is 'state-elementfinder' (up to '30' secs)
Wait until at page 'Page title' // passes if current page title (case-insensitive)
Wait until at page 'site.com/page' // or url contains this text
Wait until at page '/page'
Wait until at page 'Page .*' // passes if current page title
Wait until at page '(.*)page' // or url matches this regex
Wait until at page '^http(.*)page.+$'
Wait until at page 'Page title' (up to '30' secs)
Wait until at page 'site.com/page' (up to '30' secs)
Wait until at page '/page' (up to '30' secs)
Wait until at page 'Page .*' (up to '30' secs)
Wait until at page '(.*)page' (up to '30' secs)
Wait until at page '^http(.*)page.+$' (up to '30' secs)
Wait until cookie 'name' contains 'value'
Wait until cookie 'name' contains 'value' (up to '30' secs)

Assert

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
Verify {variable} equals 'value'
Verify {variable} is 'value'
Verify {variable} == 'value'
Verify {variable} is greater than 'value'
Verify {variable} > 'value'
Verify {variable} is greater than or equal to 'value'
Verify {variable} >= 'value'
Verify {variable} is less than 'value'
Verify {variable} < 'value'
Verify {variable} is less than or equal to 'value'
Verify {variable} <= 'value'

Network conditions and throttling

Chrome only

Since Chrome is the only browser that supports emulating network conditions, this step only works in that browser. In all other browsers, this step quietly does nothing.

See conditional tests for ways of excluding other browsers, if it's cleaner that way.

You must run an Open [browser] step before using the step below.
  1
  2
  3
  4
  5
// Makes the browser emulate the given network conditions
// latency is additional latency in ms
// max download and upload speeds are in bytes/sec
Set network conditions to offline='true' latency='200' max-download-speed='300000' max-upload-speed='400000'

Mocking time and geolocation

You must run an Open [browser] step before using any of the steps below.

Time

  1
  2
  3
  4
  5
  6
  7
  8
  9
// Makes the browser think the date and time is the one that's given (hijacks js Date)
// Takes any string the js Date object can interpret
Mock time to '4/1/2003'
Mock time to '2011 Aug 19 4:45 pm'
Mock time to '2020-09-02 19:19:45'
// Where {date} contains a js Date object
Mock time to {date}

Geolocation

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
// Makes the browser think the user's current location is the one that's given
Mock location to latitude='28.538336' longitude='-81.379234' // that's Orlando, FL, USA
// Current pre-defined locations (case-insensitive)
Mock location to 'Berlin'
Mock location to 'London'
Mock location to 'Moscow'
Mock location to 'New York'
Mock location to 'Mumbai'
Mock location to 'San Francisco'
Mock location to 'Seattle'
Mock location to 'Shanghai'
Mock location to 'São Paulo'
Mock location to 'Sao Paulo'
Mock location to 'Tokyo'

Stop

  1
  2
// Stops all time, geolocation, and http mocks and restores originals
Stop all mocks

Mocking APIs

You must run an Open [browser] step before using any of the js functions listed below.
The following js functions work with both GET and POST

String response

  1
  2
  3
  4
  5
  6
Mock an endpoint {
await mockHttp('GET', '/endpoint', 'canned response');
}
// An XHR GET from the browser to /endpoint will always get a
// 200 with 'canned response' body

JSON response

  1
  2
  3
  4
  5
  6
Mock an endpoint with a JSON response {
await mockHttp('GET', '/endpoint', {key: 'val'});
}
// An XHR GET from the browser to /endpoint will always get a
// 200 with json body '{"key":"val"}'

Detailed response

  1
  2
  3
  4
  5
  6
  7
  8
Mock an endpoint and specify the response status code, http headers, and body {
await mockHttp('GET', '/endpoint',
[201, {'Content-Type': 'text/plain'}, 'canned response']
);
}
// An XHR GET from the browser to /endpoint will always get a
// 201 with the given http headers and 'canned response' body

Function response

  1
  2
  3
  4
  5
  6
  7
  8
Mock an endpoint with a function {
await mockHttp('GET', '/endpoint', function(xhr) {
xhr.respond(201, {'Content-Type': 'text/plain'}, 'canned response');
});
}
// An XHR GET from the browser to /endpoint will always get a
// 201 with the given http headers and 'canned response' body

Regex endpoint

  1
  2
  3
  4
  5
  6
Mock every endpoint that matches a regex {
await mockHttp('GET', /\/end.*/, 'canned response');
}
// An XHR GET from the browser to any matching endpoint will always get a
// 200 with 'canned response' body

Stop mocks

  1
  2
  3
Stop all http mocks and restore original endpoints {
await mockHttpStop();
}

Configure

  1
  2
  3
  4
Configure the mock server {
await mockHttpConfigure({autoRespond: false});
// See "Fake server options" at sinon's page for a list of configuration options
}

Check out sinon's fake xhr and server (the underlying library) for more details on what you can do.

ElementFinders

What's an ElementFinder?

An ElementFinder (or EF) is a string that matches elements on a page.

They contain one or more lines, where each line is a comma-separated list of props. A prop (or property) describes an element, or the state of an element.

  1
  2
  3
  4
  5
Click 'login button' // login button is an EF consisting of one prop, login button
Click ['follow', next to 'bob'] // 'follow', next to 'bob' is an EF
// 'follow' and next to 'bob' are both props
// remember that [brackets] delimit strings, like "quotes" or 'quotes'

Alternatively, a list of props may be separated by spaces, but only if each item is either 'text', an ord (see below), or a prop that's already been defined (multiple words are ok, but no 'inputs').

  1
  2
  3
  4
  5
  6
Click ['Login' button] // 'Login' button is an EF, 'Login' and button are both props
Click [4th 'Login' button] // 4th (the ord prop), must come first in space-separated EFs
Click 'red button' // if red button is not already defined, it will try to
// interpret this as a list of two props, red and button
Use space-separated inside steps

You should use space-separated EFs in steps (such as Click) because they sound natural and are easier to read. For example, Click [red 'login' button] sounds better than Click [button, red, 'login'].

The $() function is used to find an element given an EF. Since it throws an error if nothing is found in time, $() can also be used to verify the existence (and visibility) of an element.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
Verify login box {
// Verifies the existence of at least one visible element
// that matches the EF with prop login box
await $(`login box`);
}
Get login box {
// Sets elem to the first visible WebElement
// that matches the EF with prop login box
let elem = await $(`login box`);
}
Verify focused login box {
// Verifies the existence of at least one visible element
// that matches the EF with these 4 props
await $(`login box, focused, 'text inside', .selector`);
// Similar example using props separated by either spaces or commas
// css selectors and props with input have to be separated with commas
await $(`focused 'text inside' login box, .selector`);
}
// See code reference for details on $()

Likewise, $$() finds multiple matching elements, given an EF.

To play around with EFs, run smashtest -r, open a browser, nav to a page, and type in various quoted 'EFs' (or, you can run a test in debug). The browser console (DevTools) will contain logs for every EF match.

Props

Use defined props whenever possible

Although selectors can be valid props on their own, avoid steps like Click '#some-elem'. Instead, define a prop and use it in the step, e.g., Click 'login box'. Your tests will be easier to read and refactorable.

Setting

Props are defined with the props() function. More on that on the code reference page.

  1
  2
  3
  4
  5
  6
  7
  8
On homepage {
// Define props for all the elements on the homepage
// Format: 'prop name': `EF` or function
props({
'login box': `.msgbox, enabled, 2nd`,
'about link': `selector "a[name='about']"`
});
}

Matching rules

Starting with all elements in the DOM, as props are applied (left to right), each one narrows down the list of elements. The elements that remain at the end are the ones that match the EF.

Whenever a prop is encountered, it is interpreted according to the first rule it matches in the following list:

  1. 'text'
    • Matches elements where the given text is contained in its innerText, value, placeholder, associated label's innerText, or currently selected <option>'s innerText (for selects).
    • Case insensitive, leading and trailing whitespace is ignored, and all whitespace is treated as a single space (both the 'text' and text in the DOM).

  2. 1st, 2nd, 3rd, etc. (ord)
    • E.g., 4th = take the elements currently matched and choose only the 4th one.
    • In a comma-separated prop list, you will usually list an ord last, since it narrows you down to just one element.
    • In a space-separated prop list, the ord must come first (it's more natural-sounding that way), but will be applied last.

  3. defined prop
    • These are the definitions set by props()
    • They may take an input string, such as propname 'input string'
      • You must use commas to separate these from other props
    • There are several props that come pre-defined

  4. css selector
    • If nothing else matches, a prop is interpreted to be a css selector
    • You must use commas to separate these from other props

If an EF fails to match an element, it will throw an error containing the full EF, prettified, with a --> explanation next to the lines that didn't match.

Mind your selectors!

If you use a tagname selector and it actually matches the name of a defined prop, the defined prop will be used. To be safe, use selector 'tagname' as your prop.

The same is true for selectors containing commas (ORs). Always use selector '.selector1, .selector2' since commas separate props by default.

Implicit visible prop

By default, all EFs get an implicit visible prop (meaning, only match visible elements). The exception is when you explicitly use a visible, not visible, or any visibility prop. More on those here.

Not

Start a prop with "not" to find elements that don't match that prop. E.g., not fuzzy or not .selector,

Counters

Match multiple elements by preceding a line with a counter.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
await $$(`login box`); // match 1 or more login boxes
await $$(`1 x login box`); // match exactly 1 login box
await $$(`3 x login box`); // match exactly 3 login boxes
await $$(`0+ x login box`); // match 0 or more login boxes
await $$(`1+ x login box`); // match 1 or more login boxes
await $$(`2+ x login box`); // match 2 or more login boxes
await $$(`2- x login box`); // match 2 or more login boxes
await $$(`2-5 x login box`); // match between 2 and 5 login boxes, inclusive

Child elements

Simple example

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
// Matches one or more .list elements that contain these 3 children, in that order:
await $$(`
.list
.item1
.item2
.item3
`);
// This is a subset matching, meaning that other elements
// can exist in and around the 3 children inside .list
// The top parent (.list) can start at any indentation that's a multiple of 4

Multi-level with counters

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
await $$(`
1+ x .list // matches 1 or more .list elements that contain these children:
4 x .item // 4 .item's
button[name=q] // 1 button with attribute name set to "q"
// blank lines are ok
.section // 1 .section that contains these children:
#textbox // 1 #textbox
enabled login box // 1 login box that's enabled
button, contains 'click me' // 1 button that contains 'click me'
`);
// Note that // comments are allowed inside EFs

Any order

  1
  2
  3
  4
  5
  6
  7
await $$(`
#list
any order // the 3 children can be in any order
.item, "NYC"
.item, "Tokyo"
.item, "Paris"
`);

Matching children

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
// A [] around a line will match and return those elements,
// as opposed to the top parent
// This EF will match 4 elements:
// a 'Tokyo' item and the 3 items that follow
await $$(`
#list
.item, 'NYC'
[.item, 'Tokyo']
[3 x .item]
`);
Mind your []'s!

To match the selector [attr="something"], use the prop selector '[attr="something"]'. Otherwise, the []'s will be interpreted as a matching operator.

Implicit body

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
await $$(`
#one // multiple lines at the top indent
#two
`);
// implicitly equates to
await $$(`
body
#one
#two
`);

Element array

Do stricter validations with element arrays:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
// An element array is a line that starts with a *
// It will match as many elements as it can, and verify that
// all children listed underneath map 1-to-1 with each element matched.
// The ordering and number must be the same. If not, an error occurs.
await $$(`
#list
* .item // this is an element array
.item, 'NYC'
.item, 'Tokyo'
2 x .item
`);
// There must be exactly 4 items inside of #list:
// a 'NYC', a 'Tokyo', and 2 more items of any kind
// in that exact order. Otherwise you get an error.
// Note: You can include the `any order` keyword, as shown before,
// to allow any ordering of the children

Default ElementFinder props

Smashtest defines these props by default. Remember, you can put not in front of any of them.

  • visible = matches if an element is visible to the user (width and height > 0, no hidden styles, opacity > 0, opacity of all ancestors > 0)
    • This prop is implicitly applied to all EFs, unless 'any visibility', 'visible', or 'not visible' are already in that EF
  • any visibility = matches elements regardless of visibility, disables implicit 'visible' prop
    • If you define a new prop as 'new prop': `#some-invisible-elem, any visibility`, you must use that new prop as Click 'new prop, any visibility'
    • Likewise, if you define another prop as 'even newer prop': `new prop, any visibility`, you must use that prop as Click 'even newer prop, any visibility', applying 'any visibility' all the way up the chain
  • enabled = matches if 'disabled' attribute isn't present
  • disabled = matches if 'disabled' attribute is present
  • checked = matches if element.checked is true
  • unchecked = matches if element.checked is false
  • selected = matches if element.selected is true (used for selects)
  • focused = matches if element currently has focus
  • element = matches any element
  • clickable = matches if element is of a clickable type (a, button, label, input, textarea, select, option) or has cursor:pointer style
  • page title 'title' = causes error if the page title isn't equal to the given string
  • page title contains 'title' = causes error if the page title doesn't contain the given string (case-insensitive)
  • page url 'url' = causes error if the page url isn't equal to the given string (relative or absolute)
  • page url contains 'url' = causes error if the page url doesn't contain the given string
  • next to 'text' = matches the one element that's closest in the DOM to the given text
    • Takes each elem and expands the container around it to its parent, parent's parent etc. until a container containing the text is found - matches that one element associated with that container (matches multiple elems if there's a tie)
  • value 'text' = matches if element.value is equal to the given text
  • contains 'text' = matches if the given text is contained in an element's innerText, value, placeholder, associated label's innerText, or currently selected <option>'s innerText (for selects).
    • Case insensitive, leading and trailing whitespace is ignored, and all other whitespace is treated as a single space (in both the 'text' and the DOM).
  • 'text' = same as contains 'text'
  • contains exact 'text' = matches if an element's innerText, value, placeholder, associated label's innerText, or currently selected <option>'s innerText (for selects) equals the given text exactly
  • innertext 'text' = matches if an element's innerText contains the given text
  • selector 'selector' = matches if an element matches the given css selector
  • xpath 'xpath' = matches if an element matches the given xpath
  • style 'name:value' = matches if an element has the style with the given name set to the given value
  • position 'N' = matches the one element from the pool of currently matched elements that's in position N (where the first index is 1)
    • Same as ords (1st, 2nd, etc.)
  • textbox = matches if an element is a textbox or textarea

Screenshots

When enabled, screenshots are taken before and after every step in branches that include an Open [browser] step. They're available in the report by clicking a step.

While good for debugging failures, screenshots can take up a lot of disk space and slow down test execution. Limit screenshots with these flags:

  • --screenshots=[true/false]
    • set to true to enable screenshots

  • --max-screenshots=[N]
    • doesn't take any more screenshots after N are taken
    • screenshots that are taken and later deleted due to --step-data don't count against N

  • --step-data=[all/fail/none]
    • set to 'fail' to only retain screenshots (and other step data) for failed branches
    • set to 'none' to not retain any screenshots (or other step data)

Code reference

These js functions and objects are accessible within any code block after an Open [browser] step.
Don't forget await!

Async js function calls should always be preceded with await. If errors are thrown but aren't captured inside a step, or if things just seem wonky, it's probably because you forgot an await.

Finding elements

$()

  1
  2
  3
  4
  5
  6
  7
  8
  9
// Finds the first matching WebElement
let elem = await $('elementfinder'); // waits up to 2 secs, otherwise throws error
// Can be used to verify existence of an ElementFinder (by throwing error if not found)
await $(`
#list
.item
.item
`);
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
/**
* Finds the first matching element. Waits up to timeout ms.
* Scrolls to the matching element, if found.
* @param {String or WebElement} element - An EF representing the EF to use. If set to a WebElement, returns that WebElement.
* @param {Boolean} [tryClickable] - If true, first try searching among clickable elements only. If no elements are found, searches among non-clickable elements.
* @param {WebElement} [parentElem] - If set, only searches at or within this parent element
* @param {Number} [timeout] - How many ms to wait before giving up (2000 ms if omitted)
* @param {Boolean} [isContinue] - If true, and if an error is thrown, that error's continue will be set to true
* @return {Promise} Promise that resolves to first WebDriver WebElement that was found
* @throws {Error} If a matching element wasn't found in time, or if an element array wasn't properly matched in time
*/
await $(element, tryClickable, parentElem, timeout, isContinue);

$$()

  1
  2
// Finds all the matching WebElements
let elems = await $$('elementfinder'); // waits up to 2 secs, otherwise throws error
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
/**
* Finds the matching elements. Waits up to timeout ms.
* See $() for param details
* If element is an EF and a counter isn't set on the top element, sets it to 1+
* @return {Promise} Promise that resolves to Array of WebDriver WebElements that were found
* @throws {Error} If matching elements weren't found in time, or if an element array wasn't properly matched in time
*/
await $$(element, parentElem, timeout, isContinue);
// Note: If you want this function to return an empty array and
// not throw an error, pass in an EF with counter `0+ x element`

not$()

  1
  2
// Ensures no matching elements exist
await not$('elementfinder'); // waits up to 2 secs, otherwise throws error
  1
  2
  3
  4
  5
  6
  7
/**
* Throws an error if the given element(s) don't disappear before the timeout
* See $() for param details
* @return {Promise} Promise that resolves if the given element(s) disappear before the timeout
* @throws {Error} If matching elements still found after timeout
*/
await not$(element, parentElem, timeout, isContinue);

str()

  1
  2
  3
  4
  5
  6
  7
// str() escapes strings for use in ElementFinders:
let s = "string\'\n";
await $(`
#list
.item
.item, contains '${str(s)}'
`);

ElementFinder props

props()

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
// Define props with ElementFinders or functions
// If a prop listed already exists, it is overridden
// Prop definitions exist for all future steps in the branch, until overridden
props({
// ElementFinder definitions:
'message box': `.msgbox, enabled`,
'search results': `
#list
.result, 'one'
.result, 'two'
`,
'groovy': `'contains this groovy text'`,
'retro button': `selector '.button', groovy`,
// Function definitions:
'fuzzy': function(elems, input) {
// This function will be injected into the browser
// elems is an array of DOM Elements
// input is the input string (e.g., fuzzy 'input here'), undefined if not set
// return an array of Elements from elem that match this prop
return elems;
}
});

propsAdd()

  1
  2
  3
  4
// Same as props(), but adds to a definition if it already exists
propsAdd({
'message box': `.msgbox`
});
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
Define a message box {
props({
'message box': `.msgbox`
});
}
Add to the definition of a message box {
propsAdd({
'message box': `#msgbox`
});
}
// Now, an element will be a 'message box' if it matches .msgbox OR #msgbox

propsClear()

  1
  2
// Clears the definitions of the props listed
propsClear(['message box', 'groovy']);

Executing JS in browser

The js executed inside the browser only has access to the args that were passed in and the vars/functions accessible inside the browser. All vars from the code block or test must be passed in as args.

executeScript()

  1
  2
  3
  4
await executeScript(function() {
// executes js in the browser
// see webdriverjs's executeScript()
});
  1
  2
  3
  4
  5
  6
  7
  8
  9
let arg1 = 'one';
let arg2 = 2;
let v = await executeScript(function(arg1, arg2) {
// arg1 and arg2 are accessible here
return arg1 + arg2;
}, arg1, arg2);
// v is "one2"

executeAsyncScript()

  1
  2
  3
  4
  5
  6
await executeAsyncScript(function(done) {
// executes js in the browser
// must call done() callback at the end
// see webdriverjs's executeAsyncScript()
done();
});
  1
  2
  3
  4
  5
  6
  7
  8
  9
let arg1 = 'one';
let arg2 = 2;
let v = await executeAsyncScript(function(arg1, arg2, done) {
// arg1 and arg2 are accessible here
done(arg1 + arg2);
}, arg1, arg2);
// v is "one2"

browser

browser

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
browser; // BrowserInstance object that represents the open browser
// Browser details
browser.params.name;
browser.params.version;
browser.params.platform;
browser.params.width;
browser.params.height;
browser.params.deviceEmulation;
browser.params.isHeadless;
browser.params.testServer;

browser.driver

  1
browser.driver; // webdriverjs WebDriver object that represents the open browser's driver

Mocking

mockTime()

Try to use the Mock time to {{date}} step instead.
  1
  2
let date = new Date();
await mockTime(date); // set the browser's date to this one
  1
  2
  3
  4
  5
  6
/**
* Mock's the current page's Date object to simulate the given time. Time will run forward normally.
* See sinon's fake timers for more details
* @param {Date} time - The time to set the browser to
*/
await mockTime(time);

mockLocation()

Try to use the Mock location steps instead.
  1
await mockLocation(28.538336, -81.379234); // sets the browser's location to the given latitude and longitude

mockHttp()

  1
  2
  3
// Responds with 200 'canned response' when an XHR GET in the browser tries to hit /endpoint on the current domain
// See mocking APIs for more details
await mockHttp('GET', '/endpoint', 'canned response');
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
/**
* Mocks the current page's XHR. Sends back the given response for any http requests to the given method/url from the current page.
* You can use multiple calls to this function to set up multiple routes. If a request doesn't match a route, it will get a 404 response.
* See sinon's fake xhr and server for more details
* @param {String} method - The HTTP method ('GET', 'POST', etc.)
* @param {String or RegExp} url - A url or a regex that matches urls
* @param response - A String representing the response body, or
* An Object representing the response body (it will be converted to JSON), or
* an array in the form [ [status code], { header1: "value1", etc. }, [response body string or object] ], or
* a function
* See server.respondWith() from sinon's documentation
*/
await mockHttp(method, url, response);

mockHttpConfigure()

  1
await mockHttpConfigure({autoRespond: false});
  1
  2
  3
  4
  5
  6
  7
/**
* Sets configs on the currently mocked XHR
* @param {Object} config - The options to set (key value pairs)
* See fake server options for details on what config options are available
* Fails silently if no mock is currently active
*/
await mockHttpConfigure(config);

mockTimeStop()

  1
await mockTimeStop(); // stops time mocks and restores original time

mockLocationStop()

  1
await mockLocationStop(); // stops geolocation mocks and restores original geolocation

mockHttpStop()

  1
await mockHttpStop(); // stops http mocks and restores original endpoints

mockStop()

Try to use the Stop all mocks step instead.
  1
await mockStop(); // stops all time, geolocation, and http mocks, restores originals

injectSinon()

  1
  2
  3
  4
await injectSinon(); // injects sinon js library into browser
// sinon becomes accessible in browser via global var 'sinon'
// automatically called when one of the mocking functions above invoked
// does nothing if sinon is already available inside browser

API Testing

One of Smashtest's built-in packages handles HTTP API testing. This section discusses the js functions that this package makes available.

Here are two examples of API testing:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
Book room for '0' nights
Book room for '1' nights
Book room for '2' nights
Verify success
Book room for '100' nights
Verify error
* Book room for {{n}} nights {
await api.get(`https://api.com/book/${n}`);
}
* Verify success {
response.verify({ statusCode: 200 });
}
* Verify error {
response.verify({ statusCode: { $min: 400, $max: 499 } });
}
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
Make a request {
await api.post({
url: `https://${host}/endpoint`,
headers: {
'content-type': 'application/json'
},
body: {
id: 123,
user: 'jerry'
},
timeout: 1500
});
}
Verify response {
response.verify({
statusCode: 200,
headers: {
'content-type': 'text/html; charset=utf-8'
},
error: null,
body: {
id: 123,
user: 'jerry',
results: [
'$anyOrder', // loose-matching of a JSON response body
{
cost: { $min: 10 },
items: 6,
name: { $contains: 'apples' }
},
{
cost: { $min: 15 },
items: 7,
name: { $typeof: 'string', $contains: 'berries' }
}
]
}
});
}

Also, check out this API test example.

Request

See request's documentation for details on the functions below

request()

  1
await request('https://api.com/endpoint'); // GET by default
  1
  2
  3
  4
  5
  6
  7
await request({
method: 'GET', // GET, POST, PUT, PATCH, DELETE, HEAD, or OPTIONS
url: 'https://api.com/endpoint',
timeout: 1500 // setting timeout (in ms) is recommended
// otherwise the default OS TCP timeout applies,
// which may be longer than the 60 sec step timeout
});

get()

  1
await get('https://api.com/endpoint');
  1
  2
  3
  4
  5
  6
  7
await get({
url: 'https://api.com/endpoint',
headers: {
'content-type': 'text/plain'
},
timeout: 1500
});

post()

  1
  2
  3
  4
  5
  6
  7
  8
await post({
url: 'https://api.com/endpoint',
headers: {
'content-type': 'text/plain'
},
body: `body goes here`,
timeout: 1500
});
  1
  2
  3
  4
  5
  6
  7
  8
  9
// JSON body
await post({
url: 'https://api.com/endpoint',
body: {
something: true
},
json: true, // converts body to json and sets content-type header
timeout: 1500
});

put()

  1
  2
  3
  4
  5
  6
  7
  8
await put({
url: 'https://api.com/endpoint',
headers: {
'content-type': 'text/plain'
},
body: `body goes here`,
timeout: 1500
});

patch()

  1
  2
  3
  4
  5
  6
  7
  8
await patch({
url: 'https://api.com/endpoint',
headers: {
'content-type': 'text/plain'
},
body: `body goes here`,
timeout: 1500
});

del()

  1
  2
  3
  4
  5
  6
  7
  8
await del({
url: 'https://api.com/endpoint',
headers: {
'content-type': 'text/plain'
},
body: `body goes here`,
timeout: 1500
});

head()

  1
await head('https://api.com/endpoint');

options()

  1
await options('https://api.com/endpoint');

api.defaults()

  1
api.defaults({proxy: 'http://localproxy.com'}); // see request's documentation for details

Cookies

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
// create cookies
let jar = api.jar();
let cookie1 = api.cookie('key1=value1');
let cookie2 = api.cookie('key2=value2');
let url = 'http://site.com';
jar.setCookie(cookie1, url);
jar.setCookie(cookie2, url);
// make a request that includes the cookies
await get({url: 'http://site.com/endpoint', jar: jar});

Response and verify

Simple example

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
Make a request {
await get('https://site.com/endpoint');
}
Verify the response {
// The global variable 'response' is automatically filled with the last response
// response.verify() checks that the actual response object matches
// the expected response object that's passed in
response.verify({
// you can list zero or more of these expected keys:
statusCode: 200, // expected status code
headers: { // expected headers
'content-type': 'application/json'
},
error: null, // expected error object from request library
body: { // expected body (js obj if body is json, string otherwise)
one: 'two'
},
rawBody: '{"one":"two"}', // expected raw response body
responseObj: {} // expected response object from request library
});
// To access the response details:
response.statusCode; // status code
response.headers; // js obj where keys are header names
response.error; // error obj from request library
response.body; // js obj if body is json, string otherwise
response.rawBody; // string containing unparsed body data
response.responseObj; // response obj from request library
}
Test development pattern

A good pattern for developing API tests is to make a request step followed by a response step that just does c(response); (similar to console.log(response))

Then, manually verify the response in the console and copy it into a response.verify(); in the response step.

Matching in response.verify()

Basic rules

  • A value in the expected object must == the corresponding value in the actual object, or an error will occur.

  • Expected arrays are exact match by default.
    • E.g., expected [ A, B, C ] means actual must be [ A, B, C ] exactly.

  • Expected objects are subset matching by default.
    • E.g., expected { one: 1 } will match { one: 1, two: 2 } but not { one: 3 } or { two: 2 }.

  • If matching fails, an error is thrown containing the entire actual object, prettified, with --> explanation next to each line that didn't match.

Special matching

Replace values in the expected object with { $key: value } for special loose matching, as described below:

  • { $typeof: "type" }
    • Makes sure the corresponding value in the actual object is of this type (uses js's typeof).
    • You can use "array" to match arrays.

  • { $regex: /regex/ } or { $regex: "regex" }
    • Makes sure the corresponding value in the actual object is a string that matches this regex.

  • { $contains: "string" }
    • Makes sure the corresponding value in the actual object is a string that contains this string.

  • { $max: N }
    • Makes sure the corresponding value in the actual object is a number that isn't greater than N.

  • { $min: N }
    • Makes sure the corresponding value in the actual object is a number that isn't less than N.

  • { $code: (actual) => { return actual == 'something'; } }
    { $code: "return actual == 'something'"}
    { $code: "actual == 'something'"}
    • Makes sure the corresponding value in the actual object causes this code to evaluate to true or return true.

  • { $length: N }
    • Makes sure the corresponding value in the actual object is an array, string, or object (with a length property set) whose length is N.

  • { $maxLength: N }
    • Makes sure the corresponding value in the actual object is an array, string, or object (with a length property set) whose length isn't greater than N.

  • { $minLength: N }
    • Makes sure the corresponding value in the actual object is an array, string, or object (with a length property set) whose length isn't less than N.

  • { $exact: true, a: A, b: B }
    • Makes sure the corresponding value in the actual object is an object that matches exactly (every expected key exists and has the expected value, and no other keys exist).
    • E.g., { $exact: true, one: 1 } will match { one: 1 } but not { one: 1, two: 2 }, { one: 3 }, or { two: 2 }.

  • { $every: A }
    • Makes sure the corresponding value in the actual object is an array where every item matches A.
    • E.g., { $every: 'Q' } would match [ 'Q', 'Q', 'Q' ].
    • E.g., { $every, { $contains: "foo" } } would match [ "foobar", "barfoo", "foo" ].

  • [ "$subset", A, B, C ]
    • Makes sure the corresponding value in the actual object is an array that contains A, B, and C in any order (and it could have more items).

  • [ "$anyOrder", A, B, C ]
    • Makes sure the corresponding value in the actual object is an array that contains A, B, and C (and nothing else) in any order.
    • You can use $subset and $anyOrder together such as [ "$subset", "$anyOrder", A, B, C ] to match an array containing A, B, and C (and potentially more items) in any order.

  • You can have multiple criteria in the same object.
    • E.g., { $typeof: "string", $length: 10, $regex: /[A-Z]+/ }
    • E.g., [ "$subset", "$anyOrder", A, B, C ]

  • When the expected value is an object, you can include both regular keys and $-keys in the { expected object }.

  • Undefineds
    • { one: undefined, two: 2 } will match { one: undefined, two: 2 } or { two: 2 }
    • To validate that one: undefined is actually there, use one: { $typeof: 'undefined' }
    • To validate that one: undefined is actually not there, use { $exact: true, two: 2 } with no "one" key

Comparer

Verify JS objects in general by using the built-in Comparer object. For example:

  1
  2
// Same functionality as response.verify(expectedObj);
Comparer.expect(actualObj).to.match(expectedObj);
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
/**
* Compares the actual object against the expected object
* @param {Object} actualObj - The object to check. Must not have circular references or multiple references to the same object inside. Could be an array.
* @param {Object} expectedObj - The object specifying criteria for actualObj to match
* @param {String} [errorStart] - String to mark the start of an error, '-->' with ANSI color codes if omitted
* @param {String} [errorEnd] - String to mark the end of an error, '' with ANSI color codes if omitted
* @param {String} [errorHeader] - String to put at the top of the entire error message, '' if omitted
* @param {Boolean} [jsonClone] - If true, compares using the rough clone method, aka JSON.stringify + JSON.parse (which handles multiple references to the same object inside actualObj, but also removes functions and undefineds, and converts them to null in arrays)
* @throws {Error} If actualObj doesn't match expectedObj
*/
Comparer.expect(actualObj, errorStart, errorEnd, errorHeader, jsonClone).to.match(expectedObj);

REPL

What's a REPL?

REPL stands for "read–eval–print loop". It's a way of driving Smashtest from the command-line by entering text commands.

$ smashtest -r ──────────────────────────────────────────────────────────────────────────────────────────── Smashtest 1.6.0 enter step to run it, enter or x = exit > open chrome Start: open chrome End: open chrome passed (1.085 s) Start: Use browser browser.smash:22 End: Use browser passed (0 s) enter step to run it, enter or x = exit > nav to 'site.com'

Try it yourself. Run smashtest -r or smashtest --repl, or debug a file with the ~ modifier.

Command list

  • Entering a step to run it
    • One-line steps ok
    • Multiple lines ok if step has code block (first line is Step name { and last line is })
    • Just type in { to start an anonymous step with a code block
    • No function declarations, hooks, or step blocks allowed
    • When browser is open, a convenient step is [<ElementFinder here>]. It will print the elements found to the browser's console and the number of elements found to the console you're typing into.
  • Ctrl + C = exit
  • Enter key = run next step or exit (if no more steps ahead)
  • s = skip next step
  • p = repeat previous step
  • r = resume running
  • x = exit
  • .break = sometimes you get stuck, this gets you out
  • .clear = break, and also clear the local context
  • .editor = enter editor mode
  • .exit = exit
  • .help = print a help message
  • .load = load js from a file into this REPL session
  • .save = save all evaluated commands in this REPL session to a file

Packages

Make a package

Share your steps and functions by distributing a package:

  1. Implement functionality inside a js file.
  2. Distribute that file as an npm package via the npm registry.
  3. Distribute simple function declarations that users can download directly from you or copy into their .smash files:
      1
      2
      3
    * My awesome function {
    i('my_awesome_npm_package').myAwesomeFunction(); // i() is similar to require()
    }

Use a package

To install a package:

  1. npm install the package globally (via npm -g install), in the same directory as the .smash files that will use it, or in any directory above that.
  2. Copy the above function declaration into one of your files, or download a .smash file with that declaration already inside it. Make sure it runs with the other .smash files when you run your tests.
  3. Now you can use the step anywhere in your tests:
      1
    My awesome function

Promote

We'll help promote your package! Simply contact us.

Contact Us

Say hello! We're very friendly :)

Not Found