Smashtest • Test 10x Faster

Smashtest is an open-source tool and language for rapidly generating tests.

Greatly speed up your automated testing by writing tests in a tree-like format.

Trees represent how we think when we're testing. They allow us to list all the permutations that branch off from any given point.

npm i -g smashtest
Multiple browsers and devices
UI and API
Run tests in parallel
Human-readable steps
Awesome live reports
Run locally or in CI/CD

Screenshots

Live report
Preview in report
Debug mode
Error reporting
Console
Syntax highlighting

Sample test

Open Chrome Open Firefox Open Safari Navigate to STR('site.com') Click STR('Sign In') Type STR({username:}) into STR('username box') STR({username}) is STR('joe') STR({username}) is STR('bob') STR({username}) is STR('mary') Verify success STR({username}) is STR('baduser') Verify error
represents
Test Case 1 Test Case 2 Test Case 3 ----------- ----------- ----------- Open Chrome Open Firefox Open Safari Navigate to 'site.com' Navigate to 'site.com' Navigate to 'site.com' Click 'Sign In' Click 'Sign In' Click 'Sign In' Type 'joe' into 'username box' Type 'joe' into 'username box' Type 'joe' into 'username box' Verify success Verify success Verify success Test Case 4 Test Case 5 Test Case 6 ----------- ----------- ----------- Open Chrome Open Firefox Open Safari Navigate to 'site.com' Navigate to 'site.com' Navigate to 'site.com' Click 'Sign In' Click 'Sign In' Click 'Sign In' Type 'bob' into 'username box' Type 'bob' into 'username box' Type 'bob' into 'username box' Verify success Verify success Verify success Test Case 7 Test Case 8 Test Case 9 ----------- ----------- ----------- Open Chrome Open Firefox Open Safari Navigate to 'site.com' Navigate to 'site.com' Navigate to 'site.com' Click 'Sign In' Click 'Sign In' Click 'Sign In' Type 'mary' into 'username box' Type 'mary' into 'username box' Type 'mary' into 'username box' Verify success Verify success Verify success Test Case 10 Test Case 11 Test Case 12 ------------ ------------ ------------ Open Chrome Open Firefox Open Safari Navigate to 'site.com' Navigate to 'site.com' Navigate to 'site.com' Click 'Sign In' Click 'Sign In' Click 'Sign In' Type 'baduser' into 'username box' Type 'baduser' into 'username box' Type 'baduser' into 'username box' Verify error Verify error Verify error
which represents
Test Case 1 Test Case 2 Test Case 3 ----------- ----------- ----------- let driver = await new Builder().forBrowser('chrome').build(); let driver = await new Builder().forBrowser('firefox').build(); let driver = await new Builder().forBrowser('safari').build(); await driver.get('http://site.com'); await driver.get('http://site.com'); await driver.get('http://site.com'); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); await signInButton.click(); await signInButton.click(); await signInButton.click(); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); await usernameBox.sendKeys('joe'); await usernameBox.sendKeys('joe'); await usernameBox.sendKeys('joe'); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); Test Case 4 Test Case 5 Test Case 6 ----------- ----------- ----------- let driver = await new Builder().forBrowser('chrome').build(); let driver = await new Builder().forBrowser('firefox').build(); let driver = await new Builder().forBrowser('safari').build(); await driver.get('http://site.com'); await driver.get('http://site.com'); await driver.get('http://site.com'); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); await signInButton.click(); await signInButton.click(); await signInButton.click(); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); await usernameBox.sendKeys('bob'); await usernameBox.sendKeys('bob'); await usernameBox.sendKeys('bob'); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); Test Case 7 Test Case 8 Test Case 9 ----------- ----------- ----------- let driver = await new Builder().forBrowser('chrome').build(); let driver = await new Builder().forBrowser('firefox').build(); let driver = await new Builder().forBrowser('safari').build(); await driver.get('http://site.com'); await driver.get('http://site.com'); await driver.get('http://site.com'); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); await signInButton.click(); await signInButton.click(); await signInButton.click(); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); await usernameBox.sendKeys('mary'); await usernameBox.sendKeys('mary'); await usernameBox.sendKeys('mary'); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); await driver.wait(until.elementLocated(By.id('#success-element')), 10000); Test Case 10 Test Case 11 Test Case 12 ------------ ------------ ------------ let driver = await new Builder().forBrowser('chrome').build(); let driver = await new Builder().forBrowser('firefox').build(); let driver = await new Builder().forBrowser('safari').build(); await driver.get('http://site.com'); await driver.get('http://site.com'); await driver.get('http://site.com'); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); let signInButton = await driver.findElement(By.id('#sign-in')); await signInButton.click(); await signInButton.click(); await signInButton.click(); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); await driver.wait(until.elementLocated(By.id('#username-box')), 10000); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); let usernameBox = await driver.findElement(By.id('#username-box')); await usernameBox.sendKeys('baduser'); await usernameBox.sendKeys('baduser'); await usernameBox.sendKeys('baduser'); await driver.wait(until.elementLocated(By.id('#error-element')), 10000); await driver.wait(until.elementLocated(By.id('#error-element')), 10000); await driver.wait(until.elementLocated(By.id('#error-element')), 10000);

Getting Started

Start by installing Smashtest, then writing your first .smash file

Setup

1. Install NodeJS

Make sure you have NodeJS installed. We recommend the LTS version.

2. Install Selenium Webdriver (if you're doing web UI testing)

Make sure all of the browsers you want to automate are installed. Then, choose one of the following:

Option 1: Selenium Standalone

This option allows you to run everything from just one console and handles any browser, but the install takes longer, and you have to do a manual update when a browser has a major release.

  1. You'll need to download individual executables (called "drivers") for each browser you want to automate:

    ChromeFirefoxEdgeIE

    Note: Safari 10+ on MacOS comes pre-installed with SafariDriver

  2. Each driver executable should be placed in your system's PATH

    • On MacOS that's usually /usr/local/bin (run echo $PATH to find out)
    • On Windows that's usually C:\Program Files (run echo %PATH% to find out)
  3. Make sure you have Java installed

  4. Download the latest version of selenium standalone. For example, click the 3.9 folder, then download selenium-server-standalone-3.9.1.jar.

  5. Set the SELENIUM_SERVER_JAR environment variable to the absolute path of the jar file you just downloaded.

    • On MacOS add the line export SELENIUM_SERVER_JAR=[path here] to the file ~/.bash_profile, then restart your console
    • On Windows run setx -m SELENIUM_SERVER_JAR "[path here]" (you may have to start a second command prompt as administrator), then restart your original command prompt

If you upgrade your browser, especially Chrome, make sure you re-download and install the correct driver versions.

Option 2: A Webdriver manager

Webdriver managers take care of the installation process for you, but usually require a second console to be open during test runs, and might not have support for less-popular browsers, such as Safari. You have to run a single update command when a browser has a major release.

  • If you're only running Chrome and Firefox tests, consider using webdriver-manager:

    1. In a new console, run npm install -g webdriver-manager (if that gives you permissions errors, put sudo before npm).
    2. Run webdriver-manager update to download the latest versions of everything.
    3. Run webdriver-manager start. This must always be running when executing tests.

    Whenever running Smashtest, always run it as smashtest --test-server=http://localhost:4444/wd/hub (or whatever the port is), or include that flag in the config file.

    If you upgrade your browser, especially Chrome, make sure you run the update command (step 2) to make sure your drivers are in sync with the browser versions you have installed.

  • A similar tool is selenium-standalone, which supports Chrome, Firefox, IE, and Edge. Windows users may want to look into this.

Option 3: Selenium Grid

Alternatively, you can run Smashtest with a Selenium Grid. This is an advanced, distributed configuration.

To install a grid locally:

  1. Follow steps 1-4 under Option 1 (but do not set the SELENIUM_SERVER_JAR variable)
  2. In a console, run java -jar selenium-server-standalone-[version].jar -role hub and keep it running
  3. In another console, run java -jar selenium-server-standalone-[version].jar -role node -hub http://localhost:4444/grid/register and keep it running
  4. Whenever running Smashtest, always run it as smashtest --test-server=http://localhost:4444/wd/hub (or whatever the port is), or include that flag in the config file

See the grid page for more information.

You may also use a grid from a cloud service. If so, be sure to set up your capabilities in your test and set --test-server to point at your service.

3. Install Smashtest

From a console, run npm install -g smashtest

(if that gives you permissions errors, put sudo before npm)

4. Install a grammar (highly recommended)

Smashtest grammars are available on the Atom and VSCode editors. A grammar will highlight your code, making your .smash files pretty and much easier to read.

Atom

  1. Download Atom
  2. Install the Smashtest package
  3. Set configuration for smash files
    1. Ctrl/Cmd + Shift + P
    2. Type "open config" and hit enter (this will open your config.cson file)
    3. Add the following to the bottom of the file:
      ".smash.source": editor: autoIndentOnPaste: false commentStart: "//" tabLength: 4

VSCode

Writing your first test

Here's what to do

  1. Create a new text file called "helloworld.smash"
  2. Put the following into the file:
    Open Chrome Navigate to STR('google.com') Type STR('hello world[enter]') into STR('textbox')
  3. Open a console and cd to that file's directory
  4. Run smashtest
  5. Watch test run
  6. Open live report while test is running (see console output for path)

So what did I just do?

Smashtest comes with a lot of built-in steps, a few of which you just used.

Think of steps as being arranged like comments on a website. Just like comments are indented to the comment they're replying to, steps are indented to the step they come after. In this example, there's only one line of steps, hence there's only one test. We call tests branches.

So let's add a branch!

Open Chrome Navigate to STR('google.com') Type STR('hello world[enter]') into STR('textbox') Type STR('hello universe[enter]') into STR('textbox')

Notice line 7. It's at the same indent level as line 5. That means both lines will follow line 3.

Therefore, this file will have two branches:

  1. Open Chrome, Navigate to 'google.com', Type 'hello world[enter]' into 'textbox'
  2. Open Chrome, Navigate to 'google.com', Type 'hello universe[enter]' into 'textbox'

Try running it yourself. Then, give yourself a pat on the back!

Debug and learn

Try this debugging technique...

Open Chrome Navigate to STR('google.com') MOD(~ Type 'hello world[enter]' into 'textbox') Type STR('hello universe[enter]') into STR('textbox')

Put a MOD(~) in front of line 5.

That single branch will be isolated, the browser will run non-headless (you can see it), and the test will pause before that line. You can now use the console to try out steps, move to the next step, repeat a previous step, etc.

This mode is called the REPL. It's great for learning and debugging.

Development technique

Using MOD(~) is actually a recommended test development technique:

  1. Write a step
  2. Put a MOD(~) at the end of it
  3. Run smashtest, which runs the browser to that line
  4. Come up with all the permutations that can branch off from that point (using the browser as a guide)
  5. List them as steps, indented to the step with the MOD(~)
  6. Repeat

Examples

Feel free to play with this UI test example and this API test example.

Basic language syntax

Branches

Open Chrome COM(// executed in both branches) Navigate to STR('site.com') COM(// executed in both branches) Click STR('one') COM(// branch 1 ends here) Click STR('two') COM(// branch 2 ends here) COM(// produces branches:) COM(// 1CLOSEP open, nav, click 'one') COM(// 2CLOSEP open, nav, click 'two')

Step blocks

Open Chrome COM(// this group of 5 steps is known as a "step block" (same indent, no blank lines in betweenCLOSEP) Open Firefox Open Safari Open Edge Open IE COM(// one empty line under a step block is mandatory) Navigate to STR('site.com') COM(// 5 separate branches end in this step)

More branches

Open Chrome Navigate to STR('google.com') Do a search COM(// this step ends branch 1) Navigate to STR('pets.com') Buy cat food COM(// this step ends branch 2) Buy dog food COM(// this step ends branch 3)

Textual steps

This step is a function call COM(// it executes an action) MOD(-) This step is a textual step COM(// it's just a piece of text to organize your tests) Look, I can put the "-" modifier at the end too! MOD(-) Navigate to 'site.com' MOD(-) Logged-in tests COM(// etc.) MOD(-) Logged-out tests COM(// etc.)

Code blocks

Open Chrome Navigate to STR('site.com') Click the logo { COM(// this is a code block) COM(// you can do anything js or nodejs supports) (JSKEYWORD(await) JSFUNC($)(STR('#logo'))).JSFUNC(click)(); JSVARIABLE(browser).JSFUNC(driver); COM(// webdriverjs's WebDriver object) } COM(// must end at the same indent level as the starting line)

Functions

Open Chrome Navigate to STR('site.com') Click STR('element') COM(// all 3 steps are function calls to built-in ("packaged"CLOSEP functions) FD(* Log In) COM(// this is a function declaration) Click STR('log in box') Type STR('username') into STR('username box') Click STR('ok') Open Chrome Navigate to STR('site.com') Log In COM(// this is a function call (it will execute the 3 login stepsCLOSEP) Log Out COM(// another function call) log out COM(// steps are case insensitive) FD(* Log Out) { COM(// this is a code block) (JSKEYWORD(await) JSFUNC($)(STR('.logout-button'))).JSFUNC(click)(); }

Variables

STR({username}) = STR('superman') COM(// this sets the global variable 'superman') STR({username}) is STR('superman') COM(// same as above) STR({username}) is STR("superman") COM(// same as above) STR({username}) is STR([superman]) COM(// same as above) Type STR('{username} is a handsome guy') into STR('textbox')

Sequential

MOD(..) Open Chrome Nav to STR('site.com') Click STR('button') COM(// is the same as) Open Chrome Nav to STR('site.com') Click STR('button')

ElementFinders

Open Chrome Navigate to STR('https://www.site.com') On homepage { COM(// describe props, which are things on the page and/or the state of those things) JSFUNC(props)({ STR('message box'): STR(`.textbox, enabled`), COM(// inside the `` is an ElementFinder,) COM(// a special syntax for finding elements) STR('login button'): STR(`#login`), STR('search results'): STR(`) STR(#list) STR(.result, 'one') STR(.result, 'two') STR(`), STR('groovy'): STR(`'contains this groovy text'`), STR('retro button'): STR(`selector '.button', groovy`) }CLOSEP; } Type STR('hello world') into STR('message box') COM(// using a prop makes it easier to read and refactor)

Running Tests

Command line

Run smashtest [files] [options]

[files] can be a filename or a glob (e.g., tests/*.smash)
[options] are explained here.

All files are concatenated into one big piece of text, and everything at indent 0 (e.g., function declarations) is accessible to all the other files.

After branches are compiled, they are all run.

smashtest will run all .smash files in the current directory (but not sub-directories, unless --recursive is set).

Ordering of branches

Branches are run in order of frequency, high to low.

Otherwise, branches are shuffled randomly. Here's why:

  1. You can stop the run at any time and get a good sampling of different functionality
  2. Diversifies the browsers and devices used at any given time, so as to maximize the capabilities of a grid (such as selenium grid)

Disable with the --random=false flag.

Headless

For UI testing, browsers will run headless by default, except if you're debugging (in which case they will run visibly).

Safari, IE, and Edge don't support headless, so they always run visibly.

Override with the --headless=[true/false] flag.

Command-line options

Command-line flags vs. config file

Pass in options via command-line flags (e.g., smashtest test.smash --headless=false --max-parallel=7) or by setting them in a config file.

The config file must be in the same directory where smashtest is called from, must be valid json, and must be called smashtest.json:

{ JSVARIABLE("headless"): JSCONST(false), JSVARIABLE("max-parallel"): JSCONST(7) }

Command-line flags will override config file options when they conflict.

List of options

--debug=[hash]

Only run the branch with the given hash, in debug mode. Ignore $'s, ~'s, groups, and frequency.

--groups=[value] or --groups="group1,group2+group3"

Only run branches that are part of one of the groups covered by the expression.

Some group names are set automatically. For example, you can do --groups=chrome, --groups=firefox, --groups=safari, --groups=ie, --groups=edge to only run branches that use those browsers.

+ = AND, , = OR, where AND takes precendence. In other words, --groups="one,two+three,four" means run branches part of group one, or both two and three, or four.

--g:[name]=[value]

Sets a global variable before every branch.

--headless=[true/false]

Whether to run browsers as headless. Overrides default headless behavior (which is headless unless debugging).

--help or -?

Show a help prompt

--max-parallel=[N]

Do not run more than N branches simultaneously. 5 by default.

If you're hitting a selenium grid (i.e., with --test-server), set this high enough so as to max out the available slots (capabilities). Tests will block on Open [browser] if a capability isn't available yet.

--max-screenshots=[N]

Do not take more than N screenshots. No limit by default.

Screenshots taken but deleted after a branch passed do not count against N.

--min-frequency=[high/med/low]

Only run branches at or above this frequency. Set to "med" if omitted.

--no-debug

Fail if there are any $'s or ~'s. Useful to prevent debugging in CI.

--output-errors=[true/false]

Whether to output all errors to console. True by default.

Good for when you're developing tests and expect a few to fail (goes well with -s). Also good for unit tests that run quick and you don't want to switch to the report each time.

--p:[name]=[value]

Sets a persistent variable.

--random=[true/false]

Whether to randomize the order of branches. True by default.

--recursive

Scan the current directory and subdirectories for .smash files (when no filenames are passed in). Without this flag, only the current directory is scanned.

--repl or -r

Opens the REPL (drive Smashtest from command line)

--report-domain=[domain or domain:port]

Domain where the report server should run. This is where the report page gets its live updates.

Port indicates what port smashtest should run on. If omitted, an open port will be chosen, starting with port 9000.

Domain indicates the domain of the machine you're running smashtest on. When this option is omitted, localhost is chosen by default. Choose a domain other than localhost for when you're running tests in CI and want people to hit the report externally.

--report-history=[true/false]

Whether to keep a history of all reports by date/time. If true, will output each report and passed-data file to smashtest/reports/[datetime]/. Otherwise, will output each report and passed-data to smashtest/, possibly overriding the ones from the previous run.

--report-path="[absolute path]"

Sets a custom absolute path for the report and passed-data. Normally these files are outputted to smashtest/ (from the current directory), but if this flag is set, they will be outputted to [absolute path]/smashtest/.

--report-server=[true/false]

Whether to run a server during run for live report updates. Default is true.

--screenshots=[true/false]

Whether to take screenshots before and after each step. Default is false.

--skip-passed=[true/false/filename], -s, -a

Doesn't run branches that passed last time. Carries them and their passed state over into the new report.

Useful when you're fixing a breaking test and don't want to rerun the whole suite again, or when you only want to run new or updated tests.

  • --skip-passed=true
    don't run branches that passed last time (carry them over), and look to ./smashtest/passed-data for the data

  • --skip-passed=false
    run all branches that are supposed to run

  • --skip-passed=[filename]
    don't run branches that passed last time, and use the given file as the record

  • -s
    same as --skip-passed=true

  • -a
    same as --skip-passed=false

Consider copying smashtest/passed-data to a different filename from time-to-time in order to save it. Then restore it by copying that file back in. This is because every new run will overwrite whatever's in smashtest/passed-data.

--step-data=[all/fail/none]

Keep step data for all steps, only failed steps, or no steps. Set to 'all' by default. Includes screenshots. Helps reduce size of reports.

--test-server=[url]

Location of the test server (e.g., http://localhost:4444/wd/hub for selenium server).

--version or -v

Outputs the version of Smashtest.

More memory

If you need more memory, set the NODE_OPTIONS nodejs variable.

For example, in MacOS you'd enter export NODE_OPTIONS="--max_old_space_size=[max MB to use]" && smashtest [files]

Selective test running

-s

You can skip running branches that passed in the previous run by running smashtest -s or by using the --skip-passed=true flag.

Useful when you're fixing a breaking test and don't want to rerun the whole suite again, or when you only want to run new or updated tests.

Only modifier ($)

Open Chrome Navigate to STR('google.com') Do a search DEBUG($ Navigate to 'pets.com') COM(// only runs branches that pass through this step) Buy cat food COM(// i.e., the branch ending here)
Open Chrome DEBUG($ Open Firefox) Open Safari Desktop DEBUG($ Mobile) Navigate to STR('google.com') Do a search COM(// only runs the Firefox mobile branch that ends here)
Open Chrome DEBUG(Open Firefox $) Open Safari Desktop DEBUG(Mobile $) DEBUG(Navigate to 'google.com' ~) COM(// use to help isolate a branch for a debug)

Groups

Open Chrome Open Firefox Navigate to STR('google.com') MOD(#google) MOD(#happy-path) Do a search Navigate to STR('pets.com') MOD(#pets) MOD(#happy-path) Buy cat food

Only run Google branch: smashtest --groups=google

Only run branches in happy path OR pets: smashtest --groups="happy-path,pets"

Only run branches in happy path AND pets: smashtest --groups="happy-path+pets"

Only run Firefox branches: smashtest --groups=firefox (built-in group)

Only run branches in happy path AND pets, or firefox AND pets: smashtest --groups="happy-path+pets,firefox+pets"

Frequency

Open Chrome Navigate to STR('google.com') Do something you want tested very often MOD(#high) COM(// good for quick smoke tests) Do something you want usually tested MOD(#med) COM(// your normal test suite) Do something you want usually tested COM(// #med is the default freq of a branch if #high/med/low is omitted) Do something you want tested once in a while MOD(#low) COM(// good for long-running, low-risk, edge-casey stuff) This branch will be med MOD(#med) MOD(#some-group) COM(// the later step controls the branch's freq) This branch will be low

Run low/med/high tests: smashtest --min-frequency=low

Run med/high tests: smashtest --min-frequency=med

Run high tests: smashtest --min-frequency=high

Branches are also run in order of frequency, from high to low (and are shuffled within their freq).

These are the reserved hashtags used by smashtest

MOD(#desktop), MOD(#mobile),

MOD(#chrome), MOD(#firefox), MOD(#safari), MOD(#ie), MOD(#edge),

MOD(#low), MOD(#med), MOD(#high)

CI/CD integration

Test server

If you're running your tests from a test server, such as selenium grid, include smashtest --test-server=[url] with the test server's url. A selenium server, for example, runs on http://localhost:4444/wd/hub by default.

If you're using a cloud service, be sure to set your capabilities in your tests as well.

Report server

Reports are outputted to ./smashtest/report.html from the directory smashtest is run from. Make sure users who access the report have access to the contents of the ./smashtest directory.

To ensure the report can receive live updates during a run, set smashtest --report-domain=[domain:port] where the domain is of the box running smashtest, and port is the port where you want report apis to be accessible.

Flakiness

After running smashtest, run smashtest -s one or more times to rerun failed tests (mitigating environmental or selenium flakiness).

Or, consider running smashtest -s --screenshots=true, so that failed tests always get rerun with screenshots on every step.

Exit codes

The smashtest process exits with exit code 1 if at least one branch failed, 0 otherwise.

No debug

Run smashtest --no-debug to fail run if a MOD(~), MOD(~~), MOD($), exists anywhere in your tests. This way, you're certain you're running the full suite (and nobody accidentally committed their local debug modifiers).

Reports

Location

The live report is outputted to smashtest/report.html from the directory where smashtest was run.

If --report-history=true is set, the report will be at smashtest/reports/smashtest-[timestamp]/report.html.

Failed branches

Similar failed branches are grouped together. This not only shortens the report, it also helps you debug. For example, if upon expanding a group you see that the same test fails regardless of browser, you know the browser isn't the culprit.

Colors

In the report, steps that execute code are colored as passed, failed, running, skipped, or not run yet, while non-executable textual steps and functions are colored in plain gray. Branches themselves have similar colors.

What's in a name?

We believe that tests shouldn't be named.

Tests have a tendency to look very similar to other tests, and it's actually quite common to have branches that only differ by one step. Names tend to just be a crude summary of one or two steps (e.g., "logged-in cart test 016"). Having to think of a name only slows you down.

Smashtest identifies branches by a unique hash, and just shows a list of steps in the report, collapsed by similarity to keep things clean.

If you really want to name a test, use textual steps.

Performance constraints

Due to performance constraints, reports are limited to 500 branches of each type (passed, failed, etc.), and currently running is limited to 20.

Examples

Feel free to play with these test examples:

main.smash is the main entry point for tests in both examples. The comments at the top of that file contain valuble information.

To run an example, clone the project, then run smashtest inside the example's directory.

Language

Indents

Smash, the language that's run by smashtest, uses exactly 4 spaces for each indent (no tabs).

Each step is indented to the step it follows (like comments on a website).

Blank lines

Blank lines are generally ignored. Use them for stylistic organization, or to group similar steps.

The only time they matter is in a step block. A step block cannot have blank lines between members, and must have one blank line below (if it has children). You can also use blank lines to prevent a step block from forming.

Log in as STR('joe') COM(// this is a simple step block) Log in as STR('bob') Log in as STR('mary') Do test stuff

Scope

All files passed to smashtest will be concatenated into one long piece of text at runtime. Everything at indent 0 (e.g., function declarations) will be accessible in all other files.

For example:

COM(// file1.smash) COM(// -----------) Open Chrome Do test stuff COM(// declared in file2.smash)
COM(// file2.smash) COM(// -----------) FD(* Do test stuff) Navigate to STR('site.com') Click STR('button')

Modifiers

Modifiers are symbols that come before or after the step:

MOD(! - +) This step is surrounded by modifiers MOD(~ $ #med -s)

Each modifier must be surrounded by spaces.

The only modifier which acts differently based on if it comes before vs. after the step is MOD(~)

Step blocks

Simple step blocks

Log in as STR('joe') COM(// this group of 3 steps is a step block) Log in as STR('bob') Log in as STR('mary') COM(// one empty line under a step block is mandatory) Do test stuff COM(// produces branches:) COM(// 1CLOSEP log in as joe, do test stuff) COM(// 2CLOSEP log in as bob, do test stuff) COM(// 3CLOSEP log in as mary, do test stuff)

Vertical list of 2 or more steps at the same indent, no blank lines in between. Must end in an empty line if it has children.

Multi-level step blocks

Anon

open chrome nav to STR('searchengine.com') [ type STR('hello world[enter]') into STR('search box') type STR('hello world') into STR('search box') click STR('search') ] verify search results COM(// called after each leaf in the bracketed branches) COM(// produces branches:) COM(// 1CLOSEP open, nav, type w/ enter, verify) COM(// 2CLOSEP open, nav, type, click, verify)

Named

open chrome nav to STR('searchengine.com') enter search terms [ COM(// same as the first example, but names the step block) type STR('hello world[enter]') into STR('search box') type STR('hello world') into STR('search box') click STR('search') ] verify search results

Think of square brackets as a "big parenthesis" around multiple steps.

A named step block will look like a function call in the report. They keep things neat.

Always put modifiers before the [.

With code blocks

COM(// these 3 steps form a step block, even though they have code blocks) Click one { (JSKEYWORD(await) JSFUNC($)(STR('#one'))).JSFUNC(click)(); } Click two { (JSKEYWORD(await) JSFUNC($)(STR('#two'))).JSFUNC(click)(); } Click three { (JSKEYWORD(await) JSFUNC($)(STR('#three'))).JSFUNC(click)(); } Verify action was completed

Sequential

MOD(..) Open Chrome Nav to STR('site.com') Click STR('button') COM(// is the same as) Open Chrome Nav to STR('site.com') Click STR('button')

More on the sequential modifier.

Textual steps (-)

MOD(-) Tests with a logged-in user COM(// a textual step) Log in as STR('bob') Log in as STR('mary') COM(// etc.) MOD(-) Tests with invalid users COM(// a textual step) Log in as STR('') Log in as STR('baduser') COM(// etc.)

Textual steps serve to mark and organize. They don't execute anything.

They're gray in the report (to differentiate them from executable steps).

Uses

  • Can be a heading: grouping and describing the steps that come after and/or the whole branch
    • Alternatively, use functions or named step blocks to group multiple steps into one named task. They will appear as collapsible sections in the report.

  • Can be a comment you want to appear in the report
    • For example, to describe the state of the app or test at that moment in time (or a state that's beginning or ending)
    • A requirement, description, id, or name
    • Describe a manual step

  • Can be used to "comment out" an executable step (though it's recommended to use MOD(-s) or COM(//) in this case)

Recommended test structure

Textual steps are great for dividing tests based on category. In this example, the app is divided and further sub-divided into functionalities, ending in function calls that are implemented in another file. Notice how the "When" steps correspond to the category, while the "Givens" and "Thens" vary from test to test.

Given I am at my note-taking app MOD(-) Notes MOD(-) Creating MOD(-) with a normal string Given no notes exist Given notes exist When a note is created Then it is properly displayed MOD(-) with a whitespace-only string When a note is created with an empty string When a note is created with whitespace only Then no note is created COM(// ...) MOD(-) Updating COM(// ...) MOD(-) Deleting COM(// ...) MOD(-) Login COM(// ...) MOD(-) Register COM(// ...) MOD(-) Search COM(// ...)

Functions (*, **)

Function calls

Function call here Function call with STR('strings') and STR("strings") and STR([strings]) as inputs Function call with STR({variables}) and STR({{variables}}) as inputs Function call with STR('{variables} inside strings') as inputs

Function declarations

Public

FD(* Function declaration here) Navigate to STR('site.com') Check STR('checkbox') Function declaration here COM(// function call, runs the nav and check steps above)

A function declaration is accessible to all steps under its parent (or all steps, if declared at indent 0).

With inputs

FD(* Function declaration that STR({{takes}}) in STR({{inputs}})) Navigate to STR('site.com/{{takes}}/{{inputs}}') Check STR('checkbox') Function declaration that STR('string input') in STR({var input}) COM(// function call)

With brackets

FD(* Log In) [ COM(// optional brackets) Click STR('login button') Type STR('username') into STR('username box') Click STR('sign in button') ]

With code block

FD(* Log In) { (JSKEYWORD(await) JSFUNC($)(STR('.login-button'))).JSFUNC(click)(); (JSKEYWORD(await) JSFUNC($)(STR('.username'))).JSFUNC(sendKeys)(STR('username')); (JSKEYWORD(await) JSFUNC($)(STR('.signin-button'))).JSFUNC(click)(); }
FD(* Log In) { (JSKEYWORD(await) JSFUNC($)(STR('.login-button'))).JSFUNC(click)(); } Type STR('username') into STR('username box') Click sign-in { (JSKEYWORD(await) JSFUNC($)(STR('.signin-button'))).JSFUNC(click)(); }

Multiple branches

FD(* Nav to the cart page) COM(// has 2 branches, which represent 2 ways of doing this thing) Navigate to STR('/') Click STR('cart button') Navigate to STR('/cart') Nav to the cart page Click STR('checkout') COM(// produces branches:) COM(// 1CLOSEP nav to /, click cart, click checkout) COM(// 2CLOSEP nav to /cart, click checkout)
FD(* Choose a browser) Open Chrome Open Firefox FD(* Choose a viewport) Desktop Mobile Choose a browser Choose a viewport Do a test COM(// produces branches:) COM(// 1CLOSEP open chrome, desktop, do a test) COM(// 2CLOSEP open firefox, desktop, do a test) COM(// 3CLOSEP open chrome, mobile, do a test) COM(// 4CLOSEP open firefox, mobile, do a test)

Declarations inside declarations (calling in context)

FD(* On Desktop) COM(// when this is called, all child function declarations are made accessible to future steps) FD(* On Homepage) FD(* Logout) { COM(// logout actions for desktop homepage) } FD(* On Cart page) FD(* Logout) { COM(// logout actions for desktop cart page) } FD(* On Mobile) FD(* On Homepage) FD(* Logout) { COM(// logout actions for mobile homepage) } FD(* On Cart page) FD(* Logout) { COM(// logout actions for mobile cart page) } FD(* On Desktop) Desktop COM(// built-in step for desktop viewport) FD(* On Mobile) Mobile COM(// built-in step for mobile viewport) Open Chrome On Desktop Navigate to STR('/cart') On Cart page Logout COM(// executes the logout for the desktop cart page)

Gherkin

FD(* I log in) COM(// matches all 4 function calls below) COM(// etc.) Given I log in COM(// if an exact match cannot be found, gherkin (given/when/then/andCLOSEP is stripped) When I log in Then I log in And I log in

{var} = F

FD(* Choose a username) STR({x}) = STR('bob') STR({x}) = STR('joe') STR({x}) = STR('mary') STR({username}) = Choose a username Type STR({username}) into STR('username box') COM(// produces branches:) COM(// 1CLOSEP type 'bob') COM(// 2CLOSEP type 'joe') COM(// 3CLOSEP type 'mary')

Private

FD(* On cart page) FD(** Private function) COM(// not made accessible after a call to "On cart page") COM(// etc.) Private function COM(// call is ok) Something Private function COM(// call is ok) On cart page COM(Private function) COM(// compile-time error)

Hooks

FD(*** Before Every Branch) { COM(// stuff to do before every branch begins) }

More on hooks, and a full list of supported ones.

Patterns

Encapsulating and refactoring

FD(* Order dinner) COM(// multiple branches for different ways of accomplishing the same thing) MOD(-) Variant 1 Add beans to meal Add rice to meal MOD(-) Variant 2 Add rice to meal Add beans to meal

Organizing

COM(// main-tests.smash - bird's eye view of all tests helps ensure we have full coverage) COM(// ----------------) MOD(-) Test app MOD(-) Homepage tests Display homepage test MOD(-) Cart tests Empty cart test Full cart test MOD(-) Search tests Empty search test Base case search test
COM(// cart-tests.smash) COM(// ----------------) FD(* Empty cart test) COM(// etc.) FD(* Full cart test) COM(// etc.)

Dividing a single declaration into multiple files

COM(// logout.smash) COM(// ------------) FD(* On Desktop) COM(// this func declaration split into multiple files (to keep logout togetherCLOSEP) FD(* On Homepage) FD(* Logout) COM(// etc.) FD(* On Mobile) FD(* On Homepage) FD(* Logout) COM(// etc.)
COM(// search.smash) COM(// ------------) FD(* On Desktop) COM(// this func declaration split into multiple files (to keep search togetherCLOSEP) FD(* On Homepage) FD(* Do a search) COM(// etc.) FD(* On Mobile) FD(* On Homepage) FD(* Do a search) COM(// etc.)

"On" pattern

COM(// Call a function starting with "On" to indicate that this is the current state) COM(// (and not an action to be performed, such as navigating thereCLOSEP) COM(// e.g., "On [page]" can do verifications and set up props/functions for that page) FD(* On cart page) { COM(// call when on the cart page) COM(// set up ElementFinder props for everything on this page) JSFUNC(props)({ STR('list of items'): STR(`) STR(#list) STR(.item) STR(.item) STR(.item) STR(`), STR('checkout button'): STR(`#checkout`) }); } COM(// do some initial page verifications) Verify at page STR('/cart') Verify STR('list of items') is visible COM(// expose cart-related functions) FD(* Add item to cart) COM(// etc.) Open Chrome Nav to STR('/cart') On cart page Add item to cart

Enforce permutations

COM(// this pattern ensures that all permutations of these function calls are implemented below) On Desktop On Mobile Logged In Logged Out Verify something COM(// compile-time error if we're missing a permutation here:) FD(* On Desktop) FD(* Logged In) FD(* Verify something) COM(// etc.) FD(* Logged Out) FD(* Verify something) COM(// etc.) FD(* On Mobile) FD(* Logged In) FD(* Verify something) COM(// etc.) FD(* Logged Out) FD(* Verify something) COM(// etc.)
FD(* Try every string permutation) COM(// calling this function ensures that all 3 permutations) String is empty COM(// are implemented directly below, so you don't forget) String is whitespace String is normal COM(// in another file...) Type STR({string)LC(:)STR(}) into STR('textbox') Try every string permutation COM(// error if any of the 3 are missing) FD(* String is empty) STR({string}) = STR('') COM(// ...) FD(* String is whitespace) STR({string}) = STR(' ') COM(// ...) FD(* String is normal) STR({string}) = STR('normal') COM(// ...)

Rules for matching calls to declarations

Simple case

Open Chrome COM(// 4CLOSEP looks among children of this step, finds line 14) Navigate to STR('/page1') COM(// 3CLOSEP looks among children of this step, finds nothing) Login COM(// 2CLOSEP looks among children of this step, finds nothing) My function COM(// 1CLOSEP function call *** START HERE ***) Click STR('something') Click STR('something else') Click STR('something else') Navigate to STR('/page2') Login FD(* My function) COM(// the one that's matched ("overrides" line 17CLOSEP) COM(// etc.) FD(* My function) COM(// etc.)

For every function call, searches for a matching function declaration among the children of that call's parent. If nothing is found, searches among the grandparent's children, then the great-grandparent's children, etc. until it searches among all declarations at indent 0 before erroring out.

Function calls and declarations are case insensitive, leading and trailing whitespace is ignored, and whitespace in the middle is always treated as a single space.

Don't forget to escape chars!

Since 'strings' and {vars} designate inputs in a function call, always use \', \", \[, \], \{, \} when using those actual characters inside a function call's text, to prevent an input from forming:

Clear userEC(\')s credentials MOD(-) Textual step's text COM(// not necessary for textual steps)

Matching multiple declarations

MOD(-) Matching multiple function declarations under the same parent FD(* F) A FD(* F) B F COM(// matches both line 2 and line 5) COM(// produces branches:) COM(// 1CLOSEP F, A) COM(// 2CLOSEP F, B)

Making vars available below

FD(* F) STR({name}) = STR('bob') COM(// this variable will be accessible after a function call to F) F Type STR({name}) into STR('textbox') COM(// will type 'bob')

Making funcs available below

FD(* F) FD(* A) B FD(* G) FD(* A) C FD(* A) D F COM(// makes public function A at line 2 available below) A COM(// B is run here) F COM(// makes public function A at line 2 available below) G COM(// makes public function A at line 6 available below) A COM(// C is run here)

Equivalents

COM(// file1.smash) COM(// -----------) FD(* A) FD(* B) MOD(-) 1 FD(* B) MOD(-) 2
COM(// file2.smash) COM(// -----------) FD(* A) FD(* B) MOD(-) 3

is equivalent to

FD(* A) FD(* B) MOD(-) 1 MOD(-) 2 MOD(-) 3

Calls to itself

FD(* Nav to homepage) Nav to STR('/') MOD(-) Some test Open Chrome Nav to homepage COM(// calls line 8) FD(* Nav to homepage) COM(// this "intercepts" navs to homepage to do security stuff) Do security checks Nav to homepage COM(// ignores line 8 because recursion not allowed, so calls line 1)

Variables

Using

COM(// In a function call) Buy tickets for STR({num adults}) adults and STR({{num children}}) children COM(// In a string literal) Nav to STR('{host}/path/name') COM(// {host} is replaced with its value)

Setting

{var} = 'str'

STR({variable}) = STR('string') COM(// everything in Smashtest is a string) STR({variable}) = STR("string") STR({variable}) = STR([string]) STR({variable}) is STR('string') COM(// same as =) STR({variable})=STR('11'), STR({variable})=STR('22'), STR({variable})=STR('33') STR({variable})=STR('{host}/path/name') STR({variable})=STR('{var2}') COM(// cloned a var) STR({ Variable Name With Caps and Whitespace }) = STR('string')

{var} = Func with code block

STR({variable}) = Get hello COM(// variable is set to "hello world!") FD(* Get hello) { JSKEYWORD(return) STR("hello ") + STR("world!"); COM(// any kind of js value will work (not just stringCLOSEP) }
STR({variable}) = Get goodbye { COM(// variable is set to "goodbye!") JSKEYWORD(return) STR("goodbye!"); }

{var} = Func with branches

FD(* A bad username) STR({x}) = STR('00') COM(// you can use any variable name, not just {x}) STR({x}) = STR('baduser') STR({x}) = STR('[none]') STR({x}) = STR('') STR({x}) = STR(' ') STR({username}) = A bad username Type STR({username}) into STR({textbox}) COM(// produces branches:) COM(// 1CLOSEP type '00') COM(// 2CLOSEP type 'baduser') COM(// 3CLOSEP type '[none]') COM(// 4CLOSEP type '') COM(// 5CLOSEP type ' ')

Variable names are case sensitive (unlike step names), leading and trailing whitespace is ignored, and whitespace in the middle is always treated as a single space.

They're case sensitive because they're converted to js vars in code blocks (and js is case-sensitive).

Escape sequences

You're not allowed to have a \ in a variable name, and you're not allowed to have a \0, \x, \u, or \c inside a 'string literal'. To get around this, use:

STR({variable}) = special char string { JSKEYWORD(return) STR("\u2665 \cJ"); }

Types

{Global}

Global variables are accessible to all steps for the rest of the branch (inside and outside function calls).

STR({variable}) = STR('string') COM(// global variable) F FD(* F) Type STR({variable}) into STR('textbox') COM(// accessible here)
F Type STR({variable}) into STR('textbox') COM(// accessible here) FD(* F) STR({variable}) = STR('string')

{{Local}}

Local variables are only accessible inside the current function call.

STR({{variable}}) = STR('string') COM(// local variable) F Type STR({{variable}}) into STR('textbox') COM(// accessible here) FD(* F) COM(Type {{variable}} into 'textbox') COM(// NOT accessible here)
F COM(Type {{variable}} into 'textbox') COM(// NOT accessible here) FD(* F) STR({{variable}}) = STR('string') Type STR({{variable}}) into STR('textbox') COM(// accessible here) COM(// goes out of scope here)

Persistent

Persistent variables exist for the lifetime of the whole suite run. They can only be get and set inside code blocks. They are usually used for internal stuff, like storing functions and libraries.

Inside a code block

Getting

STR({variable}) = STR('something') Get a variable { JSKEYWORD(let) JSVARIABLE(v) = JSVARIABLE(variable); COM(// just use it as a js variable) COM(// for this to work, variable name must be a single word,) COM(// no chars other than A-Z, a-z, 0-9, - _ .) COM(// and not have the same name as a js keyword) COM(// when different types of vars have the same name,) COM(// local takes precedence over global, which takes precedence over persistent) } Get a local variable { JSKEYWORD(let) JSVARIABLE(v) = JSFUNC(l)(STR('variable name')); } Get a global variable { JSKEYWORD(let) JSVARIABLE(v) = JSFUNC(g)(STR('variable name')); } Get a persistent variable { JSKEYWORD(let) JSVARIABLE(v) = JSFUNC(p)(STR('variable name')); }

Setting

Set a local variable { JSFUNC(l)(STR('variable name'), STR('new value')); COM(// any kind of js value will work (not just stringCLOSEP) } Set a global variable { JSFUNC(g)(STR('variable name'), STR('new value')); } Set a persistent variable { JSFUNC(p)(STR('variable name'), STR('new value')); }

Lookahead (:)

STR({var)LC(:)STR(}) will get the value of the variable when it's set later in the branch. This allows you to refactor common steps higher up into the tree.

Type STR({username)LC(:)STR(}) into STR('login box') COM(// ignores current value of {username}, looks to the first) COM(// {username}='str' line further in the branch) STR({username}) = STR('bob') STR({username}) = STR('mary') STR({username}) = STR('vishal') Verify success STR({username}) = STR('baduser') STR({username}) = STR('[none]') STR({username}) = STR('') Verify error
Choose STR({adults)LC(:)STR(}) and STR({children)LC(:)STR(}) from reservations panel STR({adults})=STR('[none]') STR({adults})=STR('0') STR({children})=STR('[none]') STR({children})=STR('0') STR({children})=STR('1') STR({children})=STR('8') Verify error STR({adults})=STR('1') STR({adults})=STR('8') STR({children})=STR('[none]') STR({children})=STR('0') STR({children})=STR('1') STR({children})=STR('8') Verify success
Setting a lookahead var's value

You cannot set a lookahead var's value inside a code block, only in a STR({var})=STR('something') or a STR({var})=Function call (but the function call has to be sync - i.e., no awaits and an immediate return).

Code blocks

Types

FD(* Function name) { COM(// this is a code block) COM(// you can do anything js or nodejs supports) COM(// only lines between the "{" and "}" lines are part of the code block) } COM(// must end at the same indent level as the starting line)
Step name { COM(// this step is implemented in here) COM(// kind of like a one-time function call) }

Modifiers

COM(// Modifiers can come before the name, or after the name but before the "{") MOD(..) MOD(+) FD(* Function name) MOD($) MOD(!) { } MOD(..) MOD(+) Step name MOD($) MOD(!) { }

Await

COM(// Code blocks are async, meaning you can use the 'await' keyword) Verify success { JSKEYWORD(await) JSFUNC($)(STR('#success')); }
Don't forget await!

Async js function calls should always be preceded with await. If errors are thrown but aren't captured inside a step, or if things just seem wonky, it's probably because you forgot an await.

Prev

COM(// Pass a value from one code block to the next via 'return' and 'prev') Step one { JSKEYWORD(return) STR("hello"); } Step two { JSFUNC(expect)(JSVARIABLE(prev)).JSVARIABLE(to).JSVARIABLE(equal)(STR("hello")); COM(// chai's expect and assert are available by default) }
Step one { JSKEYWORD(return) STR("hello"); } Some intermediate step Step two { JSFUNC(expect)(JSVARIABLE(prev)).JSVARIABLE(to).JSVARIABLE(equal)(STR("hello")); }

Variables

Settings vars { JSFUNC(l)(STR('var1'), STR('new value')); COM(// local variable (any value will work, not just stringsCLOSEP) JSFUNC(g)(STR('var2'), STR('new value')); COM(// global variable) JSFUNC(p)(STR('var3'), STR('new value')); COM(// persistent variable) } Getting vars { JSKEYWORD(let) JSVARIABLE(v) = JSVARIABLE(var1); COM(// works if variable name is a single word with no special chars) JSKEYWORD(let) JSVARIABLE(v) = JSFUNC(l)(STR('var1')); COM(// local variable) JSKEYWORD(let) JSVARIABLE(v) = JSFUNC(g)(STR('var2')); COM(// global variable) JSKEYWORD(let) JSVARIABLE(v) = JSFUNC(p)(STR('var3')); COM(// persistent variable) }
Use l()/g()/p() to set variables

Don't set a {variable} with JSVARIABLE(variable) = STR('value'); as this will not persist past the end of the code block.

Be mindful when using 'let'

Don't use 'let' to declare a new variable with the same name as an existing {variable}, or you'll get a runtime error (double-declaration). You can use 'var' to get around this.

Timeouts

The default timeout for a step is 60 secs. If it hasn't completed by then, it will fail with a timeout error.

Steps generally have their own, more stringent timeouts as well. These are usually implemented by an (async) function called within the step that has its own timeout. We made the default timeout really high since we wanted to leave the more "realistic" timeout to each step's individual discretion.

The default timeout can be changed via JSFUNC(setStepTimeout)(secsCLOSEP for all steps after the current one in the current branch.

Errors

Normal error

Failing step { JSKEYWORD(throw new) JSCLASS(Error)(STR("oops")); COM(// this step and branch will fail and end immediately) }

Error.continue

Failing step { JSKEYWORD(let) e = JSKEYWORD(new) JSCLASS(Error)(STR("verify failed, but let's try the next verify anyway")); JSVARIABLE(e).JSVARIABLE(continue) = JSCONST(true); COM(// this error will fail the step and branch,) JSKEYWORD(throw) JSVARIABLE(e); COM(// but the branch will continue running) }

Error.fatal

Failing step { JSKEYWORD(let) e = JSKEYWORD(new) JSCLASS(Error)(STR("something really bad")); JSVARIABLE(e).JSVARIABLE(fatal) = JSCONST(true); COM(// this error will end the whole test suite execution immediately) JSKEYWORD(throw) JSVARIABLE(e); }

Stack traces

A stack trace will contain something like this, where [LINE NUMBER] represents the line in the .smash file where the error occurred:

at CodeBlock_for_[NAME OF CODE BLOCK FUNCTION] (eval at ...), <anonymous>:[LINE NUMBER]:[COL NUMBER]

Implement complex functions in their own js files to generate more traditional stack traces:

COM(// test.smash) COM(// ----------) Step { JSKEYWORD(const) JSVARIABLE(yf) = JSFUNC(i)(STR('yf'), STR('./yourfile.js')); JSVARIABLE(yf).JSFUNC(something)(); }
COM(// yourfile.js) COM(// ----------) JSKEYWORD(function) JSFUNC(something)() { COM(// etc.) } JSVARIABLE(module.exports.something) = JSVARIABLE(something);

Code reference

These js functions and objects are accessible within any code block.

c()

JSFUNC(c)(STR('print to console')); COM(// same as console.log(CLOSEP, but prints it out more neatly) COM(// (and clear of the progress barCLOSEP)

dir()

JSFUNC(dir)(); COM(// returns the absolute directory of the file where this step is)

g()

JSFUNC(g)(STR('variable'), STR('new value')); COM(// set global variable) JSFUNC(g)(STR('variable')); COM(// get global variable)

getGlobal()

JSFUNC(getGlobal)(STR('variable')); COM(// get global variable)

getLocal()

JSFUNC(getLocal)(STR('variable')); COM(// get local variable)

getPersistent()

JSFUNC(getPersistent)(STR('variable')); COM(// get persistent variable)

i()

JSKEYWORD(const) JSVARIABLE(packageName) = JSFUNC(i)(STR('package-name')); COM(// require(CLOSEP's (importsCLOSEP the given nodejs package,) COM(// sets persistent var 'packageName' to it (auto-camelCasedCLOSEP,) COM(// and returns it) JSKEYWORD(const) JSVARIABLE(myPkg) = JSFUNC(i)(STR('myPkg'), STR('package-name')); COM(// same, but sets the name of the persistent var) JSFUNC(i)(STR('myPkg'), STR('./path/to/file.js')); COM(// works with js files too) JSFUNC(i)(STR('myPkg'), STR('../path/to/file.js')); JSFUNC(i)(STR('myPkg'), STR('/Users/Shared/tests/file.js'));

l()

JSFUNC(l)(STR('variable'), STR('new value')); COM(// set local variable) JSFUNC(l)(STR('variable')); COM(// get local variable)

log()

JSFUNC(log)(STR('text to log')); COM(// logs a piece of text to the report for this step)

p()

JSFUNC(p)(STR('variable'), STR('new value')); COM(// set persistent variable) JSFUNC(p)(STR('variable')); COM(// get persistent variable)

runInstance

JSVARIABLE(runInstance); COM(// represents the test runner "thread" that's) COM(// running this step and branch (see RunInstanceCLOSEP) JSVARIABLE(runInstance.runner); COM(// represents the test runner (see RunnerCLOSEP) JSVARIABLE(runInstance.tree); COM(// represents the whole tree (see TreeCLOSEP) JSVARIABLE(runInstance.currBranch); COM(// represents the current branch (see BranchCLOSEP) JSVARIABLE(runInstance.currStep); COM(// represents the current step (see StepCLOSEP) COM(// These objects can be used to dynamically view, create, and/or edit tests at runtime)

setGlobal()

JSFUNC(setGlobal)(STR('variable'), STR('new value')); COM(// set global variable)

setLocal()

JSFUNC(setLocal)(STR('variable'), STR('new value')); COM(// set local variable)

setPersistent()

JSFUNC(setPersistent)(STR('variable'), STR('new value')); COM(// set persistent variable)

setStepTimeout()

JSFUNC(setStepTimeout)(JSCONST(30)); COM(// sets step timeout to 30 secs for all steps in this branch after this one)

Sequential (..)

.. above a step block

Simple case

Nav to STR('/page') MOD(..) COM(// makes the whole step block run sequentially) Type STR('1111') into STR('textbox') Type STR('2222') into STR('textbox') Type STR('3333') into STR('textbox') Verify success COM(// produces 1 branch:) COM(// 1CLOSEP nav, type 1111, type 2222, type 3333, verify success)

Function calls as step block members

COM(// Note: Acts differently from function calls under a .. step.) COM(// If a function call has multiple branches, multiple branches will be generated:) FD(* Go to cart) COM(// two different ways of getting to the cart) Nav to STR('/cart') Click STR('cart icon') MOD(..) Go to cart Add peanuts to cart Verify peanuts added COM(// produces branches:) COM(// 1CLOSEP nav to /cart, add peanuts, verify peanuts) COM(// 2CLOSEP click cart icon, add peanuts, verify peanuts)

.. on a step

Simple case

Sequential test MOD(..) COM(// flatten branches at or below me into one sequential branch) One Two Three Four Five COM(// produces 1 branch:) COM(// 1CLOSEP sequential test, one, two, three, four, five)

With a step block

Nav to STR('/page') MOD(..) Type STR('1111') into STR('textbox') Type STR('2222') into STR('textbox') Type STR('3333') into STR('textbox') Verify success COM(// produces 1 branch:) COM(// 1CLOSEP nav, type 1111, verify success, type 2222, verify success, type 3333, verify success)

With a function call

FD(* Type in) Type STR('1111') into STR('textbox') Type STR('2222') into STR('textbox') Type STR('3333') into STR('textbox') Nav to STR('/page') MOD(..) Type in Verify success COM(// produces 1 branch:) COM(// 1CLOSEP nav, type 1111, verify success, type 2222, verify success, type 3333, verify success) Nav to STR('/page') Type in MOD(..) COM(// all we did was move the .. down one line) Verify success COM(// produces the same branch as before)

Inside a function declaration

FD(* Open cart) MOD(..) COM(// the 3 steps here execute sequentially) Nav to STR('/') Click STR('cart icon') Verify STR('cart') is visible Open cart COM(// it's sequential inside, but not sequential out here) Do stuff Do more stuff COM(// produces 1 branch:) COM(// 1CLOSEP open cart (nav to /, click cart icon, verify cartCLOSEP, do stuff, do more stuff)

Non-parallel (!, !!)

!

STR({username}) is STR('pete') MOD(!) COM(// no two branches going through this step may execute simultaneously) Nav to STR('/') COM(// useful for testing a stateful shared resource, like a test account) COM(// etc.) Nav to STR('/page1') COM(// etc.)

!!

STR({username}) is STR('bob') MOD(!!) COM(// no two branches going through this step may execute simultaneously,) Nav to STR('/') COM(// unless --test-server was set) COM(// etc.) COM(// useful for things like safaridriver, which can't run more) Nav to STR('/page1') COM(// than one instance locally, but can handle multiple instances) COM(// etc.) COM(// in a selenium grid)

Comments

Standard use

COM(// full-line comment) Step COM(// comment at the end of a step)

Comments ignore whole lines

Open Chrome COM(// this is still a valid step block) COM(// Open Firefox) COM(// if whole line starts with //, it's ignored as if it weren't there) Open Safari Do something COM(// produces branches:) COM(// 1CLOSEP open chrome, do something) COM(// 2CLOSEP open safari, do something)

Inside code blocks

Code block step { COM(// normal js comments occur here) COM(/* normal js comments occur here */) }

Groups and freq (#)

Groups

Open Chrome Open Firefox Navigate to STR('google.com') MOD(#google) MOD(#happy-path) Do a search Navigate to STR('pets.com') MOD(#pets) MOD(#happy-path) Buy cat food

Only run Google branch: smashtest --groups=google

Only run branches in happy path OR pets: smashtest --groups="happy-path,pets"

Only run branches in happy path AND pets: smashtest --groups="happy-path+pets"

Only run Firefox branches: smashtest --groups=firefox (built-in group)

Only run branches in happy path AND pets, or firefox AND pets: smashtest --groups="happy-path+pets,firefox+pets"

Frequency

Open Chrome Navigate to STR('google.com') Do something you want tested very often MOD(#high) COM(// good for quick smoke tests) Do something you want usually tested MOD(#med) COM(// your normal test suite) Do something you want usually tested COM(// #med is the default freq of a branch if #high/med/low is omitted) Do something you want tested once in a while MOD(#low) COM(// good for long-running, low-risk, edge-casey stuff) This branch will be med MOD(#med) MOD(#some-group) COM(// the later step controls the branch's freq) This branch will be low

Run low/med/high tests: smashtest --min-frequency=low

Run med/high tests: smashtest --min-frequency=med

Run high tests: smashtest --min-frequency=high

Branches are also run in order of frequency, from high to low (and are shuffled within their freq).

These are the reserved hashtags used by smashtest

MOD(#desktop), MOD(#mobile),

MOD(#chrome), MOD(#firefox), MOD(#safari), MOD(#ie), MOD(#edge),

MOD(#low), MOD(#med), MOD(#high)

Conditional tests

It's best to avoid conditional tests. They should only be used to implement rare exceptions.

If step

If A then B { JSKEYWORD(if)(JSVARIABLE(A)) { COM(// only does B if A is true) JSFUNC(B)(); } }

If browser is...

MOD(-) Test something Do not allow Safari { JSKEYWORD(if)(JSVARIABLE(browser.params.name) == STR('safari')) { COM(// pass and end the whole branch if the browser is safari, do nothing otherwise) JSVARIABLE(runInstance.currBranch).JSFUNC(markBranch)(STR('pass')); } } MOD(-) This step only runs if the browser isn't safari

If viewport is...

MOD(-) Test something If mobile { JSKEYWORD(if)(EC(!)JSVARIABLE(runInstance.currBranch.groups).JSFUNC(includes)(STR('mobile'))) { COM(// pass and end the whole branch if the viewport isn't mobile, do nothing otherwise) JSVARIABLE(runInstance.currBranch).JSFUNC(markBranch)(STR('pass')); } } MOD(-) This step only runs if the viewport is mobile

Skipping (-, -s, .s, $s)

Skip one step

-s (recommended)

One COM(// runs) Skipped step MOD(-s) COM(// doesn't run, marked skipped in report) Two COM(// runs)

-

One COM(// runs) Skipped step MOD(-) COM(// doesn't run, regular textual step in report) Two COM(// runs)

//

One COM(// runs) COM(// Skipped step) COM(// doesn't run, not outputted to report) Two COM(// will run, but needs to be dedented once)

Skip all branches passing through a step

$s

One COM(// doesn't run) Two COM(// doesn't run) Skipped step MOD($s) COM(// skips any branch that passes through this step, still expands function calls (error if declaration not foundCLOSEP) Three COM(// doesn't run) Four COM(// doesn't run)

Skip step and all steps below

.s

One COM(// runs) Two COM(// runs) Skipped step MOD(.s) COM(// doesn't run, still expands function calls (error if declaration not foundCLOSEP) Three COM(// doesn't run) Four COM(// doesn't run) COM(// Also skips entire duplicate branches caused by .s) COM(// Be careful, when .s is inside a function declaration it will skip steps after the function call as well)

//

One COM(// runs) Two COM(// runs) COM(// Skipped step) COM(// doesn't run) COM(//) COM(// Three) COM(// doesn't run) COM(// Four) COM(// doesn't run) COM(// Useful if you don't want function calls to expand,) COM(// but won't remind you in the console/report that skipped steps exist)

// on step block member

One COM(// runs) COM(// Two) COM(// doesn't run, and doesn't run any steps below) Three COM(// runs) Four COM(// runs, except for "Two")

Collapsing (+, +?)

Collapse (+)

FD(* Select best item from dropdown) MOD(+) COM(// function calls here will be collapsed by default in the report) Click STR('dropdown') Scroll to STR('best item') Click STR('best item') Some big operation with lots of steps MOD(+) COM(// this function call will be collapsed by default in the report)

If there's an error inside a MOD(+)'ed function, or if the function is currently running, it will be uncollapsed automatically.

It's recommended to mark precondition steps with MOD(+), since their details aren't central to the test.

Hidden (+?)

FD(* Init) MOD(+?) COM(// function calls here will be hidden in the report) Do some internal stuff COM(// this function call will be hidden in the report) Some internal thing somebody reading the report doesn't care about MOD(+?)

If there's an error inside a MOD(+?)'ed function, it will be visible in the report.

Only ($)

One $

Open Chrome Navigate to STR('google.com') Do a search DEBUG($ Navigate to 'pets.com') COM(// only runs branches that pass through this step) Buy cat food COM(// i.e., the branch ending here)

Multiple $'s at different indents

Open Chrome DEBUG($ Open Firefox) Open Safari Desktop DEBUG($ Mobile) Navigate to STR('google.com') Do a search COM(// only runs the Firefox mobile branch that ends here)

Multiple $'s with the same parent

Open Chrome DEBUG($ Open Firefox) DEBUG($ Open Safari) Desktop Mobile Navigate to STR('google.com') Do a search COM(// only runs the Firefox and Safari branches (4 of themCLOSEP)

On a function declaration

DEBUG($ * Do a search) COM(// only runs branches that call this function) COM(// etc.) Log in as STR('bob') Do a search Log in as STR('vishal') Do a search

With ~'s

Open Chrome DEBUG(Open Firefox $) Open Safari Desktop DEBUG(Mobile $) DEBUG(Navigate to 'google.com' ~) COM(// use to help isolate a branch for a debug)

Debug (~, ~~)

Debug modifier (~)

Navigate to STR('google.com') MOD(~ Click 'button') COM(// isolate this branch, run in REPL, and pause right before this step) Click STR('other thing')
Navigate to STR('google.com') MOD(Click 'button' ~) COM(// isolate this branch, run in REPL, and pause right after this step) Click STR('other thing')

A branch run in debug will also pause after a failing step.

If a MOD(~) goes through multiple branches, the first one is chosen.

No report is generated and the list of previously passed branches (for -s) is retained.

Express debug modifier (~~)

Navigate to STR('google.com') MOD(~~ Click 'button') COM(// only run this branch, but no pausing and don't run in REPL) Click STR('other thing')

Just like for MOD(~), no report is generated and the list of previously passed branches (for -s) is retained.

Debug flag

You can also run smashtest --debug=[hash] to debug the branch with the given hash.

Be careful. If you change any step, the hash of its branch will change. The recommended technique is to place a MOD($), MOD(~~), or MOD(~) on a line you change, then run that. You'll want to rerun all the branches that pass through that line anyway.

Hooks (***)

What's a hook?

A hook is a code block function that runs before or after a step, branch, or test suite.

They're not meant for testing or setup/teardown. They're for internal stuff, such as reporting, importing code, js function declarations, screenshots, and logging.

Only code blocks are allowed, and they cannot have children or modifiers. Hooks aren't listed in the report. If a hook fails, the step/branch it corresponds to will take the error and fail.

Types

Before Every Branch

Parent step A B FD(*** Before Every Branch) { COM(// this code runs before every branch that goes through the parent step) COM(// i.e.,) COM(// 1CLOSEP this code, parent step, A) COM(// 2CLOSEP this code, parent step, B) } FD(*** Before Every Branch) { COM(// this code runs before every branch in existence) }

After Every Branch

Parent step FD(*** After Every Branch) { COM(// this code runs after every branch that goes through the parent step (whether it passes or failsCLOSEP) } FD(*** After Every Branch) { COM(// this code runs after every branch in existence (whether it passes or failsCLOSEP) }

Before Every Step

Parent step FD(*** Before Every Step) { COM(// this code runs before every step of every branch that goes through the parent step) } FD(*** Before Every Step) { COM(// this code runs before every step of every branch in existence) }

After Every Step

Parent step FD(*** After Every Step) { COM(// this code runs after every step of every branch that goes through the parent step) } FD(*** After Every Step) { COM(// this code runs after every step of every branch in existence) }

Before Everything

FD(*** Before Everything) { COM(// this code runs before the whole test suite begins) COM(// only valid at 0 indents) }

After Everything

FD(*** After Everything) { COM(// this code runs after the whole test suite ends) COM(// only valid at 0 indents) }

Where should setup and teardown logic go?

Setup code should go in the actual test.

Teardown code should go in the same setup logic, such that the previous state is cleaned out prior to actual testing. It should go into a hook only if necessary.

MOD(-) My test Setup Testing FD(* Setup) Clean previous state Set things up for new test

UI Testing

Smashtest is a general platform for testing which comes with several built-in packages. One of these packages handles web UI testing with selenium webdriver.

This section discusses the steps and js functions that this package makes available.

Also, check out this UI test example.

Browsers and devices

Open a browser

All UI steps MUST come after an "Open [browser]" step
Open Chrome COM(// run exclusively with --groups=chrome) Open Firefox COM(// run exclusively with --groups=firefox) Open Safari COM(// run exclusively with --groups=safari) Open IE COM(// run exclusively with --groups=ie) Open Edge COM(// run exclusively with --groups=edge) Open browser STR('chrome') COM(// use a string recognized by webdriver as a browser name)

Set viewport size

COM(// Run these exclusively with --groups=desktop) Desktop COM(// sets browser to 1920 x 1080) Laptop COM(// sets browser to 1024 x 768) Laptop L COM(// sets browser to 1440 x 900) COM(// Run these exclusively with --groups=mobile) Mobile COM(// sets browser to 375 x 667) Mobile Portrait COM(// sets browser to 375 x 667) Mobile Landscape COM(// sets browser to 667 x 375) Mobile S COM(// sets browser to 320 x 480) Mobile S Portrait COM(// sets browser to 320 x 480) Mobile S Landscape COM(// sets browser to 480 x 320) Mobile M COM(// sets browser to 375 x 667) Mobile M Portrait COM(// sets browser to 375 x 667) Mobile M Landscape COM(// sets browser to 667 x 375) Mobile L COM(// sets browser to 425 x 667) Mobile L Portrait COM(// sets browser to 425 x 667) Mobile L Landscape COM(// sets browser to 667 x 425) Tablet COM(// sets browser to 768 x 1024) Tablet Portrait COM(// sets browser to 768 x 1024) Tablet Landscape COM(// sets browser to 1024 x 768)

Set device type

COM(// These steps set the viewport to the given device, and do device emulation in Chrome) BlackBerry Z30 COM(// sets browser to 360 x 640) Blackberry PlayBook COM(// sets browser to 600 x 1024) Galaxy Note 3 COM(// sets browser to 360 x 640) Galaxy Note II COM(// sets browser to 360 x 640) Galaxy S III COM(// sets browser to 360 x 640) Galaxy S5 COM(// sets browser to 360 x 640) Kindle Fire HDX COM(// sets browser to 800 x 1280) LG Optimus L70 COM(// sets browser to 384 x 640) Laptop with HiDPI screen COM(// sets browser to 1440 x 900) Laptop with MDPI screen COM(// sets browser to 1280 x 800) Laptop with touch COM(// sets browser to 1280 x 950) Microsoft Lumia 550 COM(// sets browser to 640 x 360) Microsoft Lumia 950 COM(// sets browser to 360 x 640) Nexus 4 COM(// sets browser to 384 x 640) Nexus 5 COM(// sets browser to 360 x 640) Nexus 5X COM(// sets browser to 412 x 732) Nexus 6 COM(// sets browser to 412 x 732) Nexus 6P COM(// sets browser to 412 x 732) Nexus 7 COM(// sets browser to 600 x 960) Nexus 10 COM(// sets browser to 800 x 1280) Nokia Lumia 520 COM(// sets browser to 320 x 533) Nokia N9 COM(// sets browser to 480 x 854) Pixel 2 COM(// sets browser to 411 x 731) Pixel 2 XL COM(// sets browser to 411 x 823) iPhone 4 COM(// sets browser to 320 x 480) iPhone 5/SE COM(// sets browser to 320 x 568) iPhone 6/7/8 COM(// sets browser to 375 x 667) iPhone 6/7/8 Plus COM(// sets browser to 414 x 736) iPhone X COM(// sets browser to 375 x 812) iPad COM(// sets browser to 768 x 1024) iPad Mini COM(// sets browser to 768 x 1024) iPad Pro COM(// sets browser to 1024 x 1366)

Usage examples

Open Chrome Open Firefox Open Safari Desktop MOD(-) Desktop test 1 COM(// etc.) MOD(-) Desktop test 2 COM(// etc.) Mobile MOD(-) Mobile test 1 COM(// etc.) MOD(-) Mobile test 2 COM(// etc.) iPhone X MOD(-) iPhone X test
COM(// Login function with different steps on desktop vs. mobile) COM(// Run with `smashtest` or `smashtest *.smash`) COM(// ----- main.smash -----) Open Chrome On Desktop On Mobile COM(// etc.) Login COM(// the login that's called depends on desktop vs. mobile) COM(// ----- viewports.smash -----) FD(* On Desktop) Desktop COM(// calls `Desktop` step exposed by `Open Chrome`) FD(* On Mobile) Mobile COM(// calls `Mobile` step exposed by `Open Chrome`) COM(// ----- desktop.smash -----) FD(* On Desktop) FD(* Login) MOD(-) Desktop implementation of Login COM(// ----- mobile.smash -----) FD(* On Mobile) FD(* Login) MOD(-) Mobile implementation of Login

Capabilities

Sometimes you need to set custom browser capabilities and options, such as when you need to provide a username and password for a cloud service (BrowserStack, SauceLabs, etc.)

Set custom capabilities

Open Chrome Set custom capabilities { JSFUNC(g)(STR('browser capabilities'), { STR('name'): STR('foobar') COM(// This is the Capabilities object. Capabilities go here.) COM(// See withCapabilities(CLOSEP) }); } COM(// Note: you can set this before or after the "Open [browser]" step)

Set custom options

Open Chrome Set custom options { JSKEYWORD(const) JSVARIABLE(chrome) = JSFUNC(i)(STR('selenium-webdriver/chrome')); JSKEYWORD(let) JSVARIABLE(opts) = JSKEYWORD(new) JSVARIABLE(chrome).JSFUNC(Options)(); COM(// Call functions on opts here) COM(// See set[browser]Options(CLOSEP functions) JSFUNC(g)(STR('browser options'), opts); } COM(// Note: you can set this before or after the "Open [browser]" step)

Browser steps

You must run an Open [browser] step before using any of the steps below.
{{element}} below can either be an ElementFinder string or an existing WebElement object.
All steps that involve mouse interaction (i.e., clicking) choose the first clickable element that matches {{element}}. If nothing found, they choose the first non-clickable element that matches {{element}}.

Interact

Click STR('elementfinder') Native click STR('elementfinder') COM(// same as click, but uses js click instead of webdriver click) COM(// try this when webdriver click doesn't work) Double click STR('elementfinder') Hover over STR('elementfinder') Scroll to STR('elementfinder') Check STR('elementfinder') COM(// clicks the element, if it's currently unchecked) Uncheck STR('elementfinder') COM(// clicks the element, if it's currently checked)

Set

Type STR('text') into STR('elementfinder') Type STR('text[enter]') into STR('elementfinder') COM(// see list of keys (case-insensitiveCLOSEP) Type STR('[none]') into STR('elementfinder') COM(// step does nothing) COM(// good for including inaction when testing different inputs) Clear STR('elementfinder') COM(// clear a textbox, etc.) Set STR('elementfinder') to STR('value') Set STR('elementfinder') to STR('[none]') COM(// step does nothing) COM(// good for including inaction when testing different inputs) Select STR('6') from STR('elementfinder') COM(// selects an <option> from a <select>) COM(// if an <option> with this exact value cannot be found,) COM(// searches for an <option> that contains the value,) COM(// trimmed and case-insensitive) Select STR('[none]') from STR('elementfinder') COM(// step does nothing) COM(// good for including inaction when testing different inputs) Select element STR('option elementfinder') from STR('dropdown elementfinder') COM(// selects an <option> from a <select>) STR({variable}) = Value of STR('elementfinder')

Navigate

Navigate to STR('https://site.com/page') Navigate to STR('http://site.com/page') Navigate to STR('site.com/page') COM(// defaults to http) Navigate to STR('/page') COM(// uses domain browser is currently on) Nav to STR('/page') COM(// shorthand for Navigate) Go Back Go Forward Refresh STR({current url}) = Current url COM(// returns current absolute url)

Window

Set dimensions to width=STR('1024') height=STR('768') Maximize window Open new tab COM(// opens new tab ("window"CLOSEP and switches to it) Switch to window whose title contains STR('hello') Switch to window whose url contains STR('/page') Switch to the STR('1st') window Switch to the STR('4th') window Switch to iframe STR('elementfinder') Switch to topmost iframe STR({window title}) = Window title COM(// returns current window title)

Alerts

Accept alert COM(// clicks ok in alert modal, error if no alert open) Dismiss alert COM(// clicks cancel in alert modal, error if no alert open)

Wait

Avoid this kind of wait. Instead, consider Verify or Wait Until.
Wait STR('2') secs COM(// sleeps this long) Wait STR('2') seconds Wait STR('1') sec Wait STR('1') second

Cookies and storage

STR({cookie}) = Get cookie STR('name') Verify cookie { JSVARIABLE(cookie); COM(// object containing cookie info) JSVARIABLE(cookie.value); COM(// value of cookie) } Set cookie STR('name') to STR('value') Set cookie STR('name') to STR('value'), expiring in STR('65') secs Delete cookie STR('name') Delete all cookies Clear local storage Clear cookies and local storage

Print and log

Log STR('text to log to this step in report') STR('elementfinder') COM(// outputs found elements to browser's console) STR("elementfinder") COM(// and number of found elements to regular console) STR([elementfinder]) COM(// useful when using REPL)

Verify steps

You must run an Open [browser] step before using any of the steps below.

Verify

'Verify' steps wait up to 2 secs for the verify to pass before failing.
Verify STR('elementfinder') is visible Verify STR('elementfinder') is not visible Verify at page STR('Page title') COM(// passes if current page title (case-insensitiveCLOSEP) Verify at page STR('site.com/page') COM(// or url contains this text) Verify at page STR('/page') Verify at page STR('Page .*') COM(// passes if current page title) Verify at page STR('(.*CLOSEPpage') COM(// or url matches this regex) Verify at page STR('^http(.*CLOSEPpage.+$') Verify cookie STR('name') contains STR('value') Verify alert contains STR('hello') COM(// passes if alert is open and contains this text) COM(// Note: this step doesn't wait up to 2 secs - it's immediate)

Wait until

'Wait until' steps wait up to 15 secs, or the specified amount, for the verify to pass before failing.
Wait until STR('elementfinder') is visible Wait until STR('elementfinder') is visible (up to STR('30') secs) Wait until STR('elementfinder') is not visible Wait until STR('elementfinder') is not visible (up to STR('30') secs) Wait until at page STR('Page title') COM(// passes if current page title (case-insensitiveCLOSEP) Wait until at page STR('site.com/page') COM(// or url contains this text) Wait until at page STR('/page') Wait until at page STR('Page .*') COM(// passes if current page title) Wait until at page STR('(.*CLOSEPpage') COM(// or url matches this regex) Wait until at page STR('^http(.*CLOSEPpage.+$') Wait until at page STR('Page title') (up to STR('30') secs) Wait until at page STR('site.com/page') (up to STR('30') secs) Wait until at page STR('/page') (up to STR('30') secs) Wait until at page STR('Page .*') (up to STR('30') secs) Wait until at page STR('(.*CLOSEPpage') (up to STR('30') secs) Wait until at page STR('^http(.*CLOSEPpage.+$') (up to STR('30') secs) Wait until cookie STR('name') contains STR('value') Wait until cookie STR('name') contains STR('value') (up to STR('30') secs)

Assert

Verify STR({variable}) equals STR('value') Verify STR({variable}) is STR('value') Verify STR({variable}) == STR('value') Verify STR({variable}) is greater than STR('value') Verify STR({variable}) > STR('value') Verify STR({variable}) is greater than or equal to STR('value') Verify STR({variable}) >= STR('value') Verify STR({variable}) is less than STR('value') Verify STR({variable}) < STR('value') Verify STR({variable}) is less than or equal to STR('value') Verify STR({variable}) <= STR('value')

Network conditions and throttling

Chrome only

Since Chrome is the only browser that supports emulating network conditions, this step only works in that browser. In all other browsers, this step quietly does nothing.

See conditional tests for ways of excluding other browsers, if it's cleaner that way.

You must run an Open [browser] step before using the step below.
COM(// Makes the browser emulate the given network conditions) COM(// latency is additional latency in ms) COM(// max download and upload speeds are in bytes/sec) Set network conditions to offline=STR('true') latency=STR('200') max-download-speed=STR('300000') max-upload-speed=STR('400000')

Mocking time and geolocation

You must run an Open [browser] step before using any of the steps below.

Time

COM(// Makes the browser think the date and time is the one that's given (hijacks js DateCLOSEP) COM(// Takes any string the js Date object can interpret) Mock time to STR('4/1/2003') Mock time to STR('2011 Aug 19 4:45 pm') Mock time to STR('2020-09-02 19:19:45') COM(// Where {date} contains a js Date object) Mock time to STR({date})

Geolocation

COM(// Makes the browser think the user's current location is the one that's given) Mock location to latitude=STR('28.538336') longitude=STR('-81.379234') COM(// that's Orlando, FL, USA) COM(// Current pre-defined locations (case-insensitiveCLOSEP) Mock location to STR('Berlin') Mock location to STR('London') Mock location to STR('Moscow') Mock location to STR('New York') Mock location to STR('Mumbai') Mock location to STR('San Francisco') Mock location to STR('Seattle') Mock location to STR('Shanghai') Mock location to STR('São Paulo') Mock location to STR('Sao Paulo') Mock location to STR('Tokyo')

Stop

COM(// Stops all time, geolocation, and http mocks and restores originals) Stop all mocks

Mocking APIs

You must run an Open [browser] step before using any of the js functions listed below.
The following js functions work with both GET and POST

String response

Mock an endpoint { JSKEYWORD(await) JSFUNC(mockHttp)(STR('GET'), STR('/endpoint'), STR('canned response')); } COM(// An XHR GET from the browser to /endpoint will always get a) COM(// 200 with 'canned response' body)

JSON response

Mock an endpoint with a JSON response { JSKEYWORD(await) JSFUNC(mockHttp)(STR('GET'), STR('/endpoint'), {key: STR('val')}); } COM(// An XHR GET from the browser to /endpoint will always get a) COM(// 200 with json body '{"key":"val"}')

Detailed response

Mock an endpoint and specify the response status code, http headers, and body { JSKEYWORD(await) JSFUNC(mockHttp)(STR('GET'), STR('/endpoint'), [JSCONST(201), {STR('Content-Type'): STR('text/plain')}, STR('canned response')] ); } COM(// An XHR GET from the browser to /endpoint will always get a) COM(// 201 with the given http headers and 'canned response' body)

Function response

Mock an endpoint with a function { JSKEYWORD(await) JSFUNC(mockHttp)(STR('GET'), STR('/endpoint'), JSKEYWORD(function)(xhr) { JSVARIABLE(xhr).JSFUNC(respond)(JSCONST(201), {STR('Content-Type'): STR('text/plain')}, STR('canned response')); }); } COM(// An XHR GET from the browser to /endpoint will always get a) COM(// 201 with the given http headers and 'canned response' body)

Regex endpoint

Mock every endpoint that matches a regex { JSKEYWORD(await) JSFUNC(mockHttp)(STR('GET'), STR(/)EC(\/end.*)STR(/), STR('canned response')); } COM(// An XHR GET from the browser to any matching endpoint will always get a) COM(// 200 with 'canned response' body)

Stop mocks

Stop all http mocks and restore original endpoints { JSKEYWORD(await) JSFUNC(mockHttpStop)(); }

Configure

Configure the mock server { JSKEYWORD(await) JSFUNC(mockHttpConfigure)({autoRespond: JSCONST(false)}); COM(// See "Fake server options" at sinon's page for a list of configuration options) }

Check out sinon's fake xhr and server (the underlying library) for more details on what you can do.

ElementFinders

What's an ElementFinder?

An ElementFinder (or EF) is a string that matches elements on a page.

They contain one or more lines, where each line is a comma-separated list of props. A prop (or property) describes an element, or the state of an element.

Click STR('login button') COM(// login button is an EF consisting of one prop, login button) Click STR(['follow', next to 'bob']) COM(// 'follow', next to 'bob' is an EF) COM(// 'follow' and next to 'bob' are both props) COM(// remember that [brackets] delimit strings, like "quotes" or 'quotes')

Alternatively, a list of props may be separated by spaces, but only if each item is either 'text', an ord (see below), or a prop that's already been defined (multiple words are ok, but no 'inputs').

Click STR(['Login' button]) COM(// 'Login' button is an EF, 'Login' and button are both props) Click STR([4th 'Login' button]) COM(// 4th (the ord propCLOSEP, must come first in space-separated EFs) Click STR('red button') COM(// if red button is not already defined, it will try to) COM(// interpret this as a list of two props, red and button)
Use space-separated inside steps

You should use space-separated EFs in steps (such as Click) because they sound natural and are easier to read. For example, Click STR([red 'login' button]) sounds better than Click STR([button, red, 'login']).

The JSFUNC($)() function is used to find an element given an EF. Since it throws an error if nothing is found in time, JSFUNC($)() can also be used to verify the existence (and visibility) of an element.

Verify login box { COM(// Verifies the existence of at least one visible element) COM(// that matches the EF with prop login box) JSKEYWORD(await) JSFUNC($)(STR(`login box`)); } Get login box { COM(// Sets elem to the first visible WebElement) COM(// that matches the EF with prop login box) JSKEYWORD(let) JSVARIABLE(elem) = JSKEYWORD(await) JSFUNC($)(STR(`login box`)); } Verify focused login box { COM(// Verifies the existence of at least one visible element) COM(// that matches the EF with these 4 props) JSKEYWORD(await) JSFUNC($)(STR(`login box, focused, 'text inside', .selector`)); COM(// Similar example using props separated by either spaces or commas) COM(// css selectors and props with input have to be separated with commas) JSKEYWORD(await) JSFUNC($)(STR(`focused 'text inside' login box, .selector`)); } COM(// See code reference for details on $(CLOSEP)

Likewise, JSFUNC($$)() finds multiple matching elements, given an EF.

To play around with EFs, run smashtest -r, open a browser, nav to a page, and type in various quoted STR('EFs') (or, you can run a test in debug). The browser console (DevTools) will contain logs for every EF match.

Props

Use defined props whenever possible

Although selectors can be valid props on their own, avoid steps like Click STR('#some-elem'). Instead, define a prop and use it in the step, e.g., Click STR('login box'). Your tests will be easier to read and refactorable.

Setting

Props are defined with the JSFUNC(props)() function. More on that on the code reference page.

On homepage { COM(// Define props for all the elements on the homepage) COM(// Format: 'prop name': `EF` or function) JSFUNC(props)({ STR('login box'): STR(`.msgbox, enabled, 2nd`), STR('about link'): STR(`selector "a[name='about']"`) }); }

Matching rules

Starting with all elements in the DOM, as props are applied (left to right), each one narrows down the list of elements. The elements that remain at the end are the ones that match the EF.

Whenever a prop is encountered, it is interpreted according to the first rule it matches in the following list:

  1. 'text'
    • Matches elements where the given text is contained in its innerText, value, placeholder, associated label's innerText, or currently selected <option>'s innerText (for selects).
    • Case insensitive, leading and trailing whitespace is ignored, and all whitespace is treated as a single space (both the 'text' and text in the DOM).

  2. 1st, 2nd, 3rd, etc. (ord)
    • E.g., 4th = take the elements currently matched and choose only the 4th one.
    • In a comma-separated prop list, you will usually list an ord last, since it narrows you down to just one element.
    • In a space-separated prop list, the ord must come first (it's more natural-sounding that way), but will be applied last.

  3. defined property
    • These are the definitions set by JSFUNC(props)()
    • They may take an input string, such as STR(propname 'input string')
      • You must use commas to separate these from other props
    • There are several props that come pre-defined

  4. css selector
    • If nothing else matches, a prop is interpreted to be a css selector
    • You must use commas to separate these from other props

If an EF fails to match an element, it will throw an error containing the full EF, prettified, with a --> explanation next to the lines that didn't match.

Mind your selectors!

If you use a tagname selector and it actually matches the name of a defined prop, the defined prop will be used. To be safe, use STR(selector 'tagname') as your prop.

The same is true for selectors containing commas (ORs). Always use STR(selector '.selector1, .selector2') since commas separate props by default.

Implicit visible prop

By default, all EFs get an implicit STR(visible) prop (meaning, only match visible elements). The exception is when you explicitly use a STR(visible), STR(not visible), or STR(any visibility) prop. More on those here.

Not

Start a prop with "not" to find elements that don't match that prop. E.g., STR(not fuzzy) or STR(not .selector),

Counters

Match multiple elements by preceding a line with a counter.

JSKEYWORD(await) JSFUNC($$)(STR(`login box`)); COM(// match 1 or more login boxes) JSKEYWORD(await) JSFUNC($$)(STR(`1 x login box`)); COM(// match exactly 1 login box) JSKEYWORD(await) JSFUNC($$)(STR(`3 x login box`)); COM(// match exactly 3 login boxes) JSKEYWORD(await) JSFUNC($$)(STR(`0+ x login box`)); COM(// match 0 or more login boxes) JSKEYWORD(await) JSFUNC($$)(STR(`1+ x login box`)); COM(// match 1 or more login boxes) JSKEYWORD(await) JSFUNC($$)(STR(`2+ x login box`)); COM(// match 2 or more login boxes) JSKEYWORD(await) JSFUNC($$)(STR(`2- x login box`)); COM(// match 2 or more login boxes) JSKEYWORD(await) JSFUNC($$)(STR(`2-5 x login box`)); COM(// match between 2 and 5 login boxes, inclusive)

Child elements

Simple example

COM(// Matches one or more .list elements that contain these 3 children, in that order:) JSKEYWORD(await) JSFUNC($$)(STR(`) STR( .list) STR( .item1) STR( .item2) STR( .item3) STR(`)); COM(// This is a subset matching, meaning that other elements) COM(// can exist in and around the 3 children inside .list) COM(// The top parent (.listCLOSEP can start at any indentation that's a multiple of 4)

Multi-level with counters

JSKEYWORD(await) JSFUNC($$)(STR(`) STR( 1+ x .list) COM(// matches 1 or more .list elements that contain these children:) STR( 4 x .item) COM(// 4 .item's) STR( button[name=q]) COM(// 1 button with attribute name set to "q") COM(// blank lines are ok) STR( .section) COM(// 1 .section that contains these children:) STR( #textbox) COM(// 1 #textbox) STR( enabled login box) COM(// 1 login box that's enabled) STR( button, contains 'click me') COM(// 1 button that contains 'click me') STR(`)); COM(// Note that // comments are allowed inside EFs)

Any order

JSKEYWORD(await) JSFUNC($$)(STR(`) STR( #list) STR( any order) COM(// the 3 children can be in any order) STR( .item, "NYC") STR( .item, "Tokyo") STR( .item, "Paris") STR(`));

Matching children

COM(// A [] around a line will match and return those elements,) COM(// as opposed to the top parent) COM(// This EF will match 4 elements:) COM(// a 'Tokyo' item and the 3 items that follow) JSKEYWORD(await) JSFUNC($$)(STR(`) STR( #list) STR( .item, 'NYC') STR( [.item, 'Tokyo']) STR( [3 x .item]) STR(`));
Mind your []'s!

To match the selector STR([attr="something"]), use the prop STR(selector '[attr="something"]'). Otherwise, the []'s will be interpreted as a matching operator.

Implicit body

JSKEYWORD(await) JSFUNC($$)(STR(`) STR( #one) COM(// multiple lines at the top indent) STR( #two) STR(`)); COM(// implicitly equates to) JSKEYWORD(await) JSFUNC($$)(STR(`) STR( body) STR( #one) STR( #two) STR(`));

Element array

Do stricter validations with element arrays:

COM(// An element array is a line that starts with a *) COM(// It will match as many elements as it can, and verify that) COM(// all children listed underneath map 1-to-1 with each element matched.) COM(// The ordering and number must be the same. If not, an error occurs.) JSKEYWORD(await) JSFUNC($$)(STR(`) STR( #list) STR( * .item) COM(// this is an element array) STR( .item, 'NYC') STR( .item, 'Tokyo') STR( 2 x .item) STR(`)); COM(// There must be exactly 4 items inside of #list:) COM(// a 'NYC', a 'Tokyo', and 2 more items of any kind) COM(// in that exact order. Otherwise you get an error.) COM(// Note: You can include the `any order` keyword, as shown before,) COM(// to allow any ordering of the children)

Default ElementFinder props

Smashtest defines these props by default. Remember, you can put STR(not) in front of any of them.

  • STR(visible) = matches if an element is visible to the user (width and height > 0, no hidden styles, opacity > 0, opacity of all ancestors > 0)
    • This prop is implicitly applied to all EFs, unless 'any visibility', 'visible', or 'not visible' are already in that EF
  • STR(any visibility) = matches elements regardless of visibility, disables implicit 'visible' prop
    • If you define a new prop as STR('new prop'): STR(`#some-invisible-elem, any visibility`), you must use that new prop as Click STR('new prop, any visibility')
    • Likewise, if you define another prop as STR('even newer prop'): STR(`new prop, any visibility`), you must use that prop as Click STR('even newer prop, any visibility'), applying 'any visibility' all the way up the chain
  • STR(enabled) = matches if 'disabled' attribute isn't present
  • STR(disabled) = matches if 'disabled' attribute is present
  • STR(checked) = matches if element.checked is true
  • STR(unchecked) = matches if element.checked is false
  • STR(selected) = matches if element.selected is true (used for selects)
  • STR(focused) = matches if element currently has focus
  • STR(element) = matches any element
  • STR(clickable) = matches if element is of a clickable type (a, button, label, input, textarea, select, option) or has cursor:pointer style
  • STR(page title 'title') = causes error if the page title isn't equal to the given string
  • STR(page title contains 'title') = causes error if the page title doesn't contain the given string (case-insensitive)
  • STR(page url 'url') = causes error if the page url isn't equal to the given string (relative or absolute)
  • STR(page url contains 'url') = causes error if the page url doesn't contain the given string
  • STR(next to 'text') = matches the one element that's closest in the DOM to the given text
    • Takes each elem and expands the container around it to its parent, parent's parent etc. until a container containing the text is found - matches that one element associated with that container (matches multiple elems if there's a tie)
  • STR(value 'text') = matches if element.value is equal to the given text
  • STR(contains 'text') = matches if the given text is contained in an element's innerText, value, placeholder, associated label's innerText, or currently selected <option>'s innerText (for selects).
    • Case insensitive, leading and trailing whitespace is ignored, and all other whitespace is treated as a single space (in both the 'text' and the DOM).
  • STR('text') = same as STR(contains 'text')
  • STR(contains exact 'text') = matches if an element's innerText, value, placeholder, associated label's innerText, or currently selected <option>'s innerText (for selects) equals the given text exactly
  • STR(innertext 'text') = matches if an element's innerText contains the given text
  • STR(selector 'selector') = matches if an element matches the given css selector
  • STR(xpath 'xpath') = matches if an element matches the given xpath
  • STR(style 'name:value') = matches if an element has the style with the given name set to the given value
  • STR(position 'N') = matches the one element from the pool of currently matched elements that's in position N (where the first index is 1)
    • Same as ords (1st, 2nd, etc.)
  • STR(textbox) = matches if an element is a textbox or textarea

Screenshots

When enabled, screenshots are taken before and after every step in branches that include an Open [browser] step. They're available in the report by clicking a step.

While good for debugging failures, screenshots can take up a lot of disk space and slow down test execution. Limit screenshots with these flags:

  • --screenshots=[true/false]
    • set to true to enable screenshots

  • --max-screenshots=[N]
    • doesn't take any more screenshots after N are taken
    • screenshots that are taken and later deleted due to --step-data don't count against N

  • --step-data=[all/fail/none]
    • set to 'fail' to only retain screenshots (and other step data) for failed branches
    • set to 'none' to not retain any screenshots (or other step data)

Code reference

These js functions and objects are accessible within any code block after an Open [browser] step.
Don't forget await!

Async js function calls should always be preceded with await. If errors are thrown but aren't captured inside a step, or if things just seem wonky, it's probably because you forgot an await.

Finding elements

$()

COM(// Finds the first matching WebElement) JSKEYWORD(let) JSVARIABLE(elem) = JSKEYWORD(await) JSFUNC($)(STR('elementfinder')); COM(// waits up to 2 secs, otherwise throws error) COM(// Can be used to verify existence of an ElementFinder (by throwing error if not foundCLOSEP) JSKEYWORD(await) JSFUNC($)(STR(`) STR(#list) STR(.item) STR(.item) STR(`));
COM(/**) COM( * Finds the first matching element. Waits up to timeout ms.) COM( * Scrolls to the matching element, if found.) COM( * @param {String or WebElement} element - An EF representing the EF to use. If set to a WebElement, returns that WebElement.) COM( * @param {Boolean} [tryClickable] - If true, first try searching among clickable elements only. If no elements are found, searches among non-clickable elements.) COM( * @param {WebElement} [parentElem] - If set, only searches at or within this parent element) COM( * @param {Number} [timeout] - How many ms to wait before giving up (2000 ms if omittedCLOSEP) COM( * @param {Boolean} [isContinue] - If true, and if an error is thrown, that error's continue will be set to true) COM( * @return {Promise} Promise that resolves to first WebDriver WebElement that was found) COM( * @throws {Error} If a matching element wasn't found in time, or if an element array wasn't properly matched in time) COM( */) JSKEYWORD(await) JSFUNC($)(element, tryClickable, parentElem, timeout, isContinue);

$$()

COM(// Finds all the matching WebElements) JSKEYWORD(let) JSVARIABLE(elems) = JSKEYWORD(await) JSFUNC($$)(STR('elementfinder')); COM(// waits up to 2 secs, otherwise throws error)
COM(/**) COM( * Finds the matching elements. Waits up to timeout ms.) COM( * See $(CLOSEP for param details) COM( * If element is an EF and a counter isn't set on the top element, sets it to 1+) COM( * @return {Promise} Promise that resolves to Array of WebDriver WebElements that were found) COM( * @throws {Error} If matching elements weren't found in time, or if an element array wasn't properly matched in time) COM( */) JSKEYWORD(await) JSFUNC($$)(element, parentElem, timeout, isContinue); COM(// Note: If you want this function to return an empty array and) COM(// not throw an error, pass in an EF with counter `0+ x element`)

not$()

COM(// Ensures no matching elements exist) JSKEYWORD(await) JSFUNC(not$)(STR('elementfinder')); COM(// waits up to 2 secs, otherwise throws error)
COM(/**) COM( * Throws an error if the given element(sCLOSEP don't disappear before the timeout) COM( * See $(CLOSEP for param details) COM( * @return {Promise} Promise that resolves if the given element(sCLOSEP disappear before the timeout) COM( * @throws {Error} If matching elements still found after timeout) COM( */) JSKEYWORD(await) JSFUNC(not$)(element, parentElem, timeout, isContinue);

str()

COM(// str(CLOSEP escapes strings for use in ElementFinders:) JSKEYWORD(let) JSVARIABLE(s) = STR("string)EC(\'\n)STR("); JSKEYWORD(await) JSFUNC($)(STR(`) STR(#list) STR(.item) STR(.item, contains ')MOD(${)JSFUNC(str)(JSVARIABLE(s))MOD(})STR(') STR(`));

ElementFinder props

props()

COM(// Define props with ElementFinders or functions) COM(// If a prop listed already exists, it is overridden) COM(// Prop definitions exist for all future steps in the branch, until overridden) JSFUNC(props)({ COM(// ElementFinder definitions:) STR('message box'): STR(`.msgbox, enabled`), STR('search results'): STR(`) STR(#list) STR(.result, 'one') STR(.result, 'two') STR(`), STR('groovy'): STR(`'contains this groovy text'`), STR('retro button'): STR(`selector '.button', groovy`), COM(// Function definitions:) STR('fuzzy'): JSKEYWORD(function)(elems, input) { COM(// This function will be injected into the browser) COM(// elems is an array of DOM Elements) COM(// input is the input string (e.g., fuzzy 'input here'CLOSEP, undefined if not set) COM(// return an array of Elements from elem that match this prop) JSKEYWORD(return) JSVARIABLE(elems); } });

propsAdd()

COM(// Same as props(CLOSEP, but adds to a definition if it already exists) JSFUNC(propsAdd)({ STR('message box'): STR(`.msgbox`) });
Define a message box { JSFUNC(props)({ STR('message box'): STR(`.msgbox`) }); } Add to the definition of a message box { JSFUNC(propsAdd)({ STR('message box'): STR(`#msgbox`) }); } COM(// Now, an element will be a 'message box' if it matches .msgbox OR #msgbox)

propsClear()

COM(// Clears the definitions of the props listed) JSFUNC(propsClear)([STR('message box'), STR('groovy')]);

Executing JS in browser

The js executed inside the browser only has access to the args that were passed in and the vars/functions accessible inside the browser. All vars from the code block or test must be passed in as args.

executeScript()

JSKEYWORD(await) JSFUNC(executeScript)(JSKEYWORD(function)(CLOSEP { COM(// executes js in the browser) COM(// see webdriverjs's executeScript(CLOSEP) });
JSKEYWORD(let) JSVARIABLE(arg1) = STR('one'); JSKEYWORD(let) JSVARIABLE(arg2) = JSCONST(2); JSKEYWORD(let) JSVARIABLE(v) = JSKEYWORD(await) JSFUNC(executeScript)(JSKEYWORD(function)(arg1, arg2CLOSEP { COM(// arg1 and arg2 are accessible here) JSKEYWORD(return) JSVARIABLE(arg1) + JSVARIABLE(arg2); }, JSVARIABLE(arg1), JSVARIABLE(arg2)); COM(// v is "one2")

executeAsyncScript()

JSKEYWORD(await) JSFUNC(executeAsyncScript)(JSKEYWORD(function)(doneCLOSEP { COM(// executes js in the browser) COM(// must call done(CLOSEP callback at the end) COM(// see webdriverjs's executeAsyncScript(CLOSEP) JSFUNC(done)(); });
JSKEYWORD(let) JSVARIABLE(arg1) = STR('one'); JSKEYWORD(let) JSVARIABLE(arg2) = JSCONST(2); JSKEYWORD(let) JSVARIABLE(v) = JSKEYWORD(await) JSFUNC(executeAsyncScript)(JSKEYWORD(function)(arg1, arg2, doneCLOSEP { COM(// arg1 and arg2 are accessible here) JSFUNC(done)(JSVARIABLE(arg1) + JSVARIABLE(arg2)CLOSEP; }, JSVARIABLE(arg1), JSVARIABLE(arg2)); COM(// v is "one2")

browser

browser

JSVARIABLE(browser); COM(// BrowserInstance object that represents the open browser) COM(// Browser details) JSVARIABLE(browser.params.name); JSVARIABLE(browser.params.version); JSVARIABLE(browser.params.platform); JSVARIABLE(browser.params.width); JSVARIABLE(browser.params.height); JSVARIABLE(browser.params.deviceEmulation); JSVARIABLE(browser.params.isHeadless); JSVARIABLE(browser.params.testServer);

browser.driver

JSVARIABLE(browser.driver); COM(// webdriverjs WebDriver object that represents the open browser's driver)

Mocking

mockTime()

Try to use the Mock time to {{date}} step instead.
JSKEYWORD(let) JSVARIABLE(date) = JSKEYWORD(new) JSCLASS(Date)(); JSKEYWORD(await) JSFUNC(mockTime)(JSVARIABLE(date)); COM(// set the browser's date to this one)
COM(/**) COM( * Mock's the current page's Date object to simulate the given time. Time will run forward normally.) COM( * See sinon's fake timers for more details) COM( * @param {Date} time - The time to set the browser to) COM( */) JSKEYWORD(await) JSFUNC(mockTime)(time);

mockLocation()

Try to use the Mock location steps instead.
JSKEYWORD(await) JSFUNC(mockLocation)(JSCONST(28.538336), JSCONST(-81.379234)); COM(// sets the browser's location to the given latitude and longitude)

mockHttp()

COM(// Responds with 200 'canned response' when an XHR GET in the browser tries to hit /endpoint on the current domain) COM(// See mocking APIs for more details) JSKEYWORD(await) JSFUNC(mockHttp)(STR('GET'), STR('/endpoint'), STR('canned response'));
COM(/**) COM( * Mocks the current page's XHR. Sends back the given response for any http requests to the given method/url from the current page.) COM( * You can use multiple calls to this function to set up multiple routes. If a request doesn't match a route, it will get a 404 response.) COM( * See sinon's fake xhr and server for more details) COM( * @param {String} method - The HTTP method ('GET', 'POST', etc.CLOSEP) COM( * @param {String or RegExp} url - A url or a regex that matches urls) COM( * @param response - A String representing the response body, or) COM( * An Object representing the response body (it will be converted to JSONCLOSEP, or) COM( * an array in the form [ [status code], { header1: "value1", etc. }, [response body string or object] ], or) COM( * a function) COM( * See server.respondWith(CLOSEP from sinon's documentation) COM( */) JSKEYWORD(await) JSFUNC(mockHttp)(method, url, response);

mockHttpConfigure()

JSKEYWORD(await) JSFUNC(mockHttpConfigure)({autoRespond: JSCONST(false)});
COM(/**) COM( * Sets configs on the currently mocked XHR) COM( * @param {Object} config - The options to set (key value pairsCLOSEP) COM( * See fake server options for details on what config options are available) COM( * Fails silently if no mock is currently active) COM( */) JSKEYWORD(await) JSFUNC(mockHttpConfigure)(config);

mockTimeStop()

JSKEYWORD(await) JSFUNC(mockTimeStop)(); COM(// stops time mocks and restores original time)

mockLocationStop()

JSKEYWORD(await) JSFUNC(mockLocationStop)(); COM(// stops geolocation mocks and restores original geolocation)

mockHttpStop()

JSKEYWORD(await) JSFUNC(mockHttpStop)(); COM(// stops http mocks and restores original endpoints)

mockStop()

Try to use the Stop all mocks step instead.
JSKEYWORD(await) JSFUNC(mockStop)(); COM(// stops all time, geolocation, and http mocks, restores originals)

injectSinon()

JSKEYWORD(await) JSFUNC(injectSinon)(); COM(// injects sinon js library into browser) COM(// sinon becomes accessible in browser via global var 'sinon') COM(// automatically called when one of the mocking functions above invoked) COM(// does nothing if sinon is already available inside browser)

API Testing

One of Smashtest's built-in packages handles HTTP API testing. This section discusses the js functions that this package makes available.

Here are two examples of API testing:

Book room for STR('0') nights Book room for STR('1') nights Book room for STR('2') nights Verify success Book room for STR('100') nights Verify error FD(* Book room for )STR({{n}})FD( nights) { JSKEYWORD(await) JSVARIABLE(api).JSFUNC(get)(STR(`https://api.com/book/)JSVARIABLE(${n})STR(`)); } FD(* Verify success) { JSVARIABLE(response).JSFUNC(verify)({ statusCode: JSCONST(200) }); } FD(* Verify error) { JSVARIABLE(response).JSFUNC(verify)({ statusCode: { $min: JSCONST(400), $max: JSCONST(499) } }); }
Make a request { JSKEYWORD(await) JSVARIABLE(api).JSFUNC(post)({ url: STR(`https://)JSVARIABLE(${host})STR(/endpoint`), headers: { STR('content-type'): STR('application/json') }, body: { id: JSCONST(123), user: STR('jerry') }, timeout: JSCONST(1500) }); } Verify response { JSVARIABLE(response).JSFUNC(verify)({ statusCode: JSCONST(200), headers: { STR('content-type'): STR('text/html; charset=utf-8') }, error: JSCONST(null), body: { id: JSCONST(123), user: STR('jerry'), results: [ STR('$anyOrder'), COM(// loose-matching of a JSON response body) { cost: { $min: JSCONST(10) }, items: JSCONST(6), name: { $contains: STR('apples') } }, { cost: { $min: JSCONST(15) }, items: JSCONST(7), name: { $typeof: STR('string'), $contains: STR('berries') } } ] } }); }

Also, check out this API test example.

Request

See request's documentation for details on the functions below

request()

JSKEYWORD(await) JSFUNC(request)(STR('https://api.com/endpoint')); COM(// GET by default)
JSKEYWORD(await) JSFUNC(request)({ method: STR('GET'), COM(// GET, POST, PUT, PATCH, DELETE, HEAD, or OPTIONS) url: STR('https://api.com/endpoint'), timeout: JSCONST(1500) COM(// setting timeout (in msCLOSEP is recommended) COM(// otherwise the default OS TCP timeout applies,) COM(// which may be longer than the 60 sec step timeout) });

get()

JSKEYWORD(await) JSFUNC(get)(STR('https://api.com/endpoint'));
JSKEYWORD(await) JSFUNC(get)({ url: STR('https://api.com/endpoint'), headers: { STR('content-type'): STR('text/plain') }, timeout: JSCONST(1500) });

post()

JSKEYWORD(await) JSFUNC(post)({ url: STR('https://api.com/endpoint'), headers: { STR('content-type'): STR('text/plain') }, body: STR(`body goes here`), timeout: JSCONST(1500) });
COM(// JSON body) JSKEYWORD(await) JSFUNC(post)({ url: STR('https://api.com/endpoint'), body: { something: JSCONST(true) }, json: JSCONST(true), COM(// converts body to json and sets content-type header) timeout: JSCONST(1500) });

put()

JSKEYWORD(await) JSFUNC(put)({ url: STR('https://api.com/endpoint'), headers: { STR('content-type'): STR('text/plain') }, body: STR(`body goes here`), timeout: JSCONST(1500) });

patch()

JSKEYWORD(await) JSFUNC(patch)({ url: STR('https://api.com/endpoint'), headers: { STR('content-type'): STR('text/plain') }, body: STR(`body goes here`), timeout: JSCONST(1500) });

del()

JSKEYWORD(await) JSFUNC(del)({ url: STR('https://api.com/endpoint'), headers: { STR('content-type'): STR('text/plain') }, body: STR(`body goes here`), timeout: JSCONST(1500) });

head()

JSKEYWORD(await) JSFUNC(head)(STR('https://api.com/endpoint'));

options()

JSKEYWORD(await) JSFUNC(options)(STR('https://api.com/endpoint'));

api.defaults()

JSVARIABLE(api).JSFUNC(defaults)({proxy: STR('http://localproxy.com')}); COM(// see request's documentation for details)

Cookies

COM(// create cookies) JSKEYWORD(let) JSVARIABLE(jar) = JSVARIABLE(api).JSFUNC(jar)(); JSKEYWORD(let) JSVARIABLE(cookie1) = JSVARIABLE(api).JSFUNC(cookie)(STR('key1=value1')); JSKEYWORD(let) JSVARIABLE(cookie2) = JSVARIABLE(api).JSFUNC(cookie)(STR('key2=value2')); JSKEYWORD(let) JSVARIABLE(url) = STR('http://site.com'); JSVARIABLE(jar).JSFUNC(setCookie)(JSVARIABLE(cookie1), JSVARIABLE(url)); JSVARIABLE(jar).JSFUNC(setCookie)(JSVARIABLE(cookie2), JSVARIABLE(url)); COM(// make a request that includes the cookies) JSKEYWORD(await) JSFUNC(get)({url: STR('http://site.com/endpoint'), jar: JSVARIABLE(jar)});

Response and verify

Simple example

Make a request { JSKEYWORD(await) JSFUNC(get)(STR('https://site.com/endpoint')); } Verify the response { COM(// The global variable 'response' is automatically filled with the last response) COM(// response.verify(CLOSEP checks that the actual response object matches) COM(// the expected response object that's passed in) JSVARIABLE(response).JSFUNC(verify)({ COM(// you can list zero or more of these expected keys:) statusCode: JSCONST(200), COM(// expected status code) headers: { COM(// expected headers) STR('content-type'): STR('application/json') }, error: JSCONST(null), COM(// expected error object from request library) body: { COM(// expected body (js obj if body is json, string otherwiseCLOSEP) one: STR('two') }, rawBody: STR('{"one":"two"}'), COM(// expected raw response body) response: {} COM(// expected response object from request library) }); }
Test development pattern

A good pattern for developing API tests is to make a request step followed by a response step that just does JSFUNC(c)(JSVARIABLE(response)); (similar to console.log(response))

Then, manually verify the response in the console and copy it into a JSVARIABLE(response).JSFUNC(verify)(); in the response step.

Matching in response.verify()

Basic rules

  • A value in the expected object must == the corresponding value in the actual object, or an error will occur.

  • Expected arrays are exact match by default.
    • E.g., expected [ A, B, C ] means actual must be [ A, B, C ] exactly.

  • Expected objects are subset matching by default.
    • E.g., expected { one: 1 } will match { one: 1, two: 2 } but not { one: 3 } or { two: 2 }.

  • If matching fails, an error is thrown containing the entire actual object, prettified, with --> explanation next to each line that didn't match.

Special matching

Replace values in the expected object with { $key: value } for special loose matching, as described below:

  • { $typeof: "type" }
    • Makes sure the corresponding value in the actual object is of this type (uses js's typeof).
    • You can use "array" to match arrays.

  • { $regex: /regex/ } or { $regex: "regex" }
    • Makes sure the corresponding value in the actual object is a string that matches this regex.

  • { $contains: "string" }
    • Makes sure the corresponding value in the actual object is a string that contains this string.

  • { $max: N }
    • Makes sure the corresponding value in the actual object is a number that isn't greater than N.

  • { $min: N }
    • Makes sure the corresponding value in the actual object is a number that isn't less than N.

  • { $code: (actual) => { return actual == 'something'; } }
    { $code: "return actual == 'something'"}
    { $code: "actual == 'something'"}
    • Makes sure the corresponding value in the actual object causes this code to evaluate to true or return true.

  • { $length: N }
    • Makes sure the corresponding value in the actual object is an array, string, or object (with a length property set) whose length is N.

  • { $maxLength: N }
    • Makes sure the corresponding value in the actual object is an array, string, or object (with a length property set) whose length isn't greater than N.

  • { $minLength: N }
    • Makes sure the corresponding value in the actual object is an array, string, or object (with a length property set) whose length isn't less than N.

  • { $exact: true, a: A, b: B }
    • Makes sure the corresponding value in the actual object is an object that matches exactly (every expected key exists and has the expected value, and no other keys exist).
    • E.g., { $exact: true, one: 1 } will match { one: 1 } but not { one: 1, two: 2 }, { one: 3 }, or { two: 2 }.

  • { $every: A }
    • Makes sure the corresponding value in the actual object is an array where every item matches A.
    • E.g., { $every: 'Q' } would match [ 'Q', 'Q', 'Q' ].
    • E.g., { $every, { $contains: "foo" } } would match [ "foobar", "barfoo", "foo" ].

  • [ "$subset", A, B, C ]
    • Makes sure the corresponding value in the actual object is an array that contains A, B, and C in any order (and it could have more items).

  • [ "$anyOrder", A, B, C ]
    • Makes sure the corresponding value in the actual object is an array that contains A, B, and C (and nothing else) in any order.
    • You can use $subset and $anyOrder together such as [ "$subset", "$anyOrder", A, B, C ] to match an array containing A, B, and C (and potentially more items) in any order.

  • You can have multiple criteria in the same object.
    • E.g., { $typeof: "string", $length: 10, $regex: /[A-Z]+/ }
    • E.g., [ "$subset", "$anyOrder", A, B, C ]

  • When the expected value is an object, you can include both regular keys and $-keys in the { expected object }.

  • Undefineds
    • { one: undefined, two: 2 } will match { one: undefined, two: 2 } or { two: 2 }
    • To validate that one: undefined is actually there, use one: { $typeof: 'undefined' }
    • To validate that one: undefined is actually not there, use { $exact: true, two: 2 } with no "one" key

Comparer

Verify JS objects in general by using the built-in JSCLASS(Comparer) object. For example:

COM(// Same functionality as response.verify(expectedObjCLOSEP;) JSCLASS(Comparer).JSFUNC(expect)(JSVARIABLE(actualObj)).JSVARIABLE(to).JSFUNC(match)(JSVARIABLE(expectedObj));
COM(/**) COM( * Compares the actual object against the expected object) COM( * @param {Object} actualObj - The object to check. Must not have circular references or multiple references to the same object inside. Could be an array.) COM( * @param {Object} expectedObj - The object specifying criteria for actualObj to match) COM( * @param {String} [errorStart] - String to mark the start of an error, '-->' with ANSI color codes if omitted) COM( * @param {String} [errorEnd] - String to mark the end of an error, '' with ANSI color codes if omitted) COM( * @param {String} [errorHeader] - String to put at the top of the entire error message, '' if omitted) COM( * @param {Boolean} [jsonClone] - If true, compares using the rough clone method, aka JSON.stringify + JSON.parse (which handles multiple references to the same object inside actualObj, but also removes functions and undefineds, and converts them to null in arraysCLOSEP) COM( * @throws {Error} If actualObj doesn't match expectedObj) COM( */) JSCLASS(Comparer).JSFUNC(expect)(JSVARIABLE(actualObj), JSVARIABLE(errorStart), JSVARIABLE(errorEnd), JSVARIABLE(errorHeader), JSVARIABLE(jsonClone)).JSVARIABLE(to).JSFUNC(match)(JSVARIABLE(expectedObj));

REPL

What's a REPL?

REPL stands for "read–eval–print loop". It's a way of driving Smashtest from the command-line by entering text commands.

$ smashtest -r ──────────────────────────────────────────────────────────────────────────────────────────── Smashtest 1.6.0 enter step to run it, enter or x = exit > open chrome Start: open chrome End: open chrome passed (1.085 s) Start: Use browser browser.smash:22 End: Use browser passed (0 s) enter step to run it, enter or x = exit > nav to 'site.com'

Try it yourself. Run smashtest -r or smashtest --repl, or debug a file with the MOD(~) modifier.

Command list

  • Entering a step to run it
    • One-line steps ok
    • Multiple lines ok if step has code block (first line is Step name { and last line is })
    • Just type in { to start an anonymous step with a code block
    • No function declarations, hooks, or step blocks allowed
    • When browser is open, a convenient step is STR([<ElementFinder here>]). It will print the elements found to the browser's console and the number of elements found to the console you're typing into.
  • Ctrl + C = exit
  • Enter key = run next step or exit (if no more steps ahead)
  • s = skip next step
  • p = repeat previous step
  • r = resume running
  • x = exit
  • .break = sometimes you get stuck, this gets you out
  • .clear = break, and also clear the local context
  • .editor = enter editor mode
  • .exit = exit
  • .help = print a help message
  • .load = load js from a file into this REPL session
  • .save = save all evaluated commands in this REPL session to a file

Packages

Make a package

Share your steps and functions by distributing a package:

  1. Implement functionality inside a js file.
  2. Distribute that file as an npm package via the npm registry.
  3. Distribute simple function declarations that users can download directly from you or copy into their .smash files:
    FD(* My awesome function) { JSFUNC(i)(STR('my_awesome_npm_package')).JSFUNC(myAwesomeFunction)(); COM(// i(CLOSEP is similar to require(CLOSEP) }

Use a package

To install a package:

  1. npm install the package globally (via npm -g install), in the same directory as the .smash files that will use it, or in any directory above that.
  2. Copy the above function declaration into one of your files, or download a .smash file with that declaration already inside it. Make sure it runs with the other .smash files when you run your tests.
  3. Now you can use the step anywhere in your tests:
    My awesome function

Promote

We'll help promote your package! Simply contact us.

Contact Us

Say hello! We're very friendly :)

Not Found