npm i -g smashtest

Multiple browsers and devices

UI and API

Run tests in parallel

Human-readable steps

Awesome live reports

Run locally or in CI/CD

Screenshots

Live report

Preview in report

Debug mode

Error reporting

Console

Syntax highlighting

Sample test

Getting Started

Start by installing Smashtest, then writing your first .smash file

Setup

1. Install NodeJS

Make sure you have NodeJS installed. We recommend the LTS version. Use node -v to check.

2. Install Selenium Webdriver (if you're doing web UI testing)

1. Make sure all of the browsers you want to automate are installed
2. Make sure you have Java installed. Use java -showversion to check.
3. Choose one of the following options:

3. Install Smashtest

From a console, run npm install -g smashtest

(if that gives you permissions errors, put sudo before npm)

4. Install a grammar (highly recommended)

A grammar will highlight your code, making your .smash files pretty and much easier to read. They are currently available on the Atom and VSCode editors.

Writing your first test

Here's what to do

1. Create a new text file called "helloworld.smash"
2. Put the following into the file:
   Open Chrome Navigate to STR('google.com') Type STR('hello world[enter]') into STR('textbox')
3. Open a console and cd to that file's directory
4. Run smashtest
5. Watch test run
6. Open live report while test is running (see console output for path)

So what did I just do?

Smashtest comes with a lot of built-in steps, a few of which you just used.

Think of steps as being arranged like comments on a website. Just like comments are indented to the comment they're replying to, steps are indented to the step they come after. In this example, there's only one line of steps, hence there's only one test. We call tests branches.

So let's add a branch!

Notice line 7. It's at the same indent level as line 5. That means both lines will follow line 3.

Therefore, this file will have two branches:

1. Open Chrome, Navigate to 'google.com', Type 'hello world[enter]' into 'textbox'
2. Open Chrome, Navigate to 'google.com', Type 'hello universe[enter]' into 'textbox'

Try running it yourself. Then, give yourself a pat on the back!

Debug and learn

Try this debugging technique...

Put a MOD(~) in front of line 5.

That single branch will be isolated, the browser will run non-headless (you can see it), and the test will pause before that line. You can now use the console to try out steps, move to the next step, repeat a previous step, etc.

This mode is called the REPL. It's great for learning and debugging.

Development technique

Using MOD(~) is actually a recommended test development technique:

1. Write a step
2. Put a MOD(~) at the end of it
3. Run smashtest, which runs the browser to that line
4. Come up with all the permutations that can branch off from that point (using the browser as a guide)
5. List them as steps, indented to the step with the MOD(~)
6. Repeat

Examples

Feel free to play with this UI test example and this API test example.

Basic language syntax

Branches

Step blocks

Sequential

Textual steps

Code blocks

Functions

Variables

ElementFinders

Examples

Web UI

TodoMVC tests  •  what's being tested

REST API

OnWater API tests  •  what's being tested

How to read and run

main.smash is the main entry point for tests in both examples.

To run an example, clone the project, then run smashtest inside the example's directory.

Lessons

1 - What is Smashtest?

2 - Writing your first test

3 - Language for testing

4 - Web UI testing - Steps and ElementFinders

Running Tests

Command line

Run smashtest [files] [options]

[files] can be a filename or a glob (e.g., tests/*.smash)
[options] are explained here.

All files are concatenated into one big piece of text, and everything at indent 0 (e.g., function declarations) is accessible to all the other files.

After branches are compiled, they are all run.

smashtest will run all .smash files in the current directory (but not sub-directories, unless --recursive is set).

Ordering of branches

Branches are run in order of frequency, high to low.

Otherwise, branches are shuffled randomly. Here's why:

1. You can stop the run at any time and get a good sampling of different functionality
2. Diversifies the browsers and devices used at any given time, so as to maximize the capabilities of a grid (such as selenium grid)

Disable with the --random=false flag.

Headless

For UI testing, browsers will run headless by default, except if you're debugging (in which case they will run visibly).

Safari, IE, and Edge don't support headless, so they always run visibly.

Override with the --headless=[true/false] flag.

Command-line options

Command-line flags vs. config file

Pass in options via command-line flags (e.g., smashtest test.smash --headless=false --max-parallel=7) or by setting them in a config file.

The config file must be in the same directory where smashtest is called from, must be valid json, and must be called smashtest.json:

Command-line flags will override config file options when they conflict.

List of options

--debug=[hash]

Only run the branch with the given hash, in debug mode. Ignore $'s, ~'s, groups, and frequency.

--groups=[value] or --groups="group1,group2+group3"

Only run branches that are part of one of the groups covered by the expression.

Some group names are set automatically. For example, you can do --groups=chrome, --groups=firefox, --groups=safari, --groups=ie, --groups=edge to only run branches that use those browsers.

+ = AND, , = OR, where AND takes precendence. In other words, --groups="one,two+three,four" means run branches part of group one, or both two and three, or four.

--g:[name]=[value]

Sets a global variable before every branch.

--headless=[true/false]

Whether to run browsers as headless. Overrides default headless behavior (which is headless unless debugging).

--help or -?

Show a help prompt

--max-parallel=[N]

Do not run more than N branches simultaneously. 5 by default.

If you're hitting a selenium grid (i.e., with --test-server), set this high enough so as to max out the available slots (capabilities). Tests will block on Open [browser] if a capability isn't available yet.

--max-screenshots=[N]

Do not take more than N screenshots. No limit by default.

Screenshots taken but deleted after a branch passed do not count against N.

--min-frequency=[high/med/low]

Only run branches at or above this frequency. Set to "med" if omitted.

--no-debug

Fail if there are any $'s or ~'s. Useful to prevent debugging in CI.

--output-errors=[true/false]

Whether to output all errors to console. True by default.

Good for when you're developing tests and expect a few to fail (goes well with -s). Also good for unit tests that run quick and you don't want to switch to the report each time.

--p:[name]=[value]

Sets a persistent variable.

--random=[true/false]

Whether to randomize the order of branches. True by default.

--recursive

Scan the current directory and subdirectories for .smash files (when no filenames are passed in). Without this flag, only the current directory is scanned.

--repl or -r

Opens the REPL (drive Smashtest from command line)

--report-domain=[domain or domain:port]

Domain where the report server should run. This is where the report page gets its live updates.

Port indicates what port smashtest should run on. If omitted, an open port will be chosen, starting with port 9000.

Domain indicates the domain of the machine you're running smashtest on. When this option is omitted, localhost is chosen by default. Choose a domain other than localhost for when you're running tests in CI and want people to hit the report externally.

--report-history=[true/false]

Whether to keep a history of all reports by date/time. If true, will output each report and passed-data file to smashtest/reports/[datetime]/. Otherwise, will output each report and passed-data to smashtest/, possibly overriding the ones from the previous run.

--report-path="[absolute path]"

Sets a custom absolute path for the report and passed-data. Normally these files are outputted to smashtest/ (from the current directory), but if this flag is set, they will be outputted to [absolute path]/smashtest/.

--report-server=[true/false]

Whether to run a server during run for live report updates. Default is true.

--screenshots=[true/false]

Whether to take screenshots before and after each step. Default is false.

--skip-passed=[true/false/filename], -s, -a

Doesn't run branches that passed last time. Carries them and their passed state over into the new report.

Useful when you're fixing a breaking test and don't want to rerun the whole suite again, or when you only want to run new or updated tests.

• --skip-passed=true
  don't run branches that passed last time (carry them over), and look to ./smashtest/passed-data for the data
  
  
• --skip-passed=false
  run all branches that are supposed to run
  
  
• --skip-passed=[filename]
  don't run branches that passed last time, and use the given file as the record
  
  
• -s
  same as --skip-passed=true
  
  
• -a
  same as --skip-passed=false

Consider copying smashtest/passed-data to a different filename from time-to-time in order to save it. Then restore it by copying that file back in. This is because every new run will overwrite whatever's in smashtest/passed-data.

--step-data=[all/fail/none]

Keep step data for all steps, only failed steps, or no steps. Set to 'all' by default. Includes screenshots. Helps reduce size of reports.

--test-server=[url]

Location of the test server (e.g., http://localhost:4444/wd/hub for selenium server).

--version or -v

Outputs the version of Smashtest.

More memory

If you need more memory, set the NODE_OPTIONS nodejs variable.

For example, in MacOS you'd enter export NODE_OPTIONS="--max_old_space_size=[max MB to use]" && smashtest [files]

Selective test running

-s

You can skip running branches that passed in the previous run by running smashtest -s or by using the --skip-passed=true flag.

Useful when you're fixing a breaking test and don't want to rerun the whole suite again, or when you only want to run new or updated tests.

Only modifier ($)

Groups

Only run Google branch: smashtest --groups=google

Only run branches in happy path OR pets: smashtest --groups="happy-path,pets"

Only run branches in happy path AND pets: smashtest --groups="happy-path+pets"

Only run Firefox branches: smashtest --groups=firefox (built-in group)

Only run branches in happy path AND pets, or firefox AND pets: smashtest --groups="happy-path+pets,firefox+pets"

Frequency

Run low/med/high tests: smashtest --min-frequency=low

Run med/high tests: smashtest --min-frequency=med

Run high tests: smashtest --min-frequency=high

Branches are also run in order of frequency, from high to low (and are shuffled within their freq).

These are the reserved hashtags used by smashtest

MOD(#desktop), MOD(#mobile),

MOD(#chrome), MOD(#firefox), MOD(#safari), MOD(#ie), MOD(#edge),

MOD(#low), MOD(#med), MOD(#high)

CI/CD integration

Test server

If you're running your tests from a test server, such as selenium grid, include smashtest --test-server=[url] with the test server's url. A selenium server, for example, runs on http://localhost:4444/wd/hub by default.

If you're using a cloud service, be sure to set your capabilities in your tests as well.

Report server

Reports are outputted to ./smashtest/report.html from the directory smashtest is run from. Make sure users who access the report have access to the contents of the ./smashtest directory.

To ensure the report can receive live updates during a run, set smashtest --report-domain=[domain:port] where the domain is of the box running smashtest, and port is the port where you want report apis to be accessible.

Flakiness

After running smashtest, run smashtest -s one or more times to rerun failed tests (mitigating environmental or selenium flakiness).

Or, consider running smashtest -s --screenshots=true, so that failed tests always get rerun with screenshots on every step.

Exit codes

The smashtest process exits with exit code 1 if at least one branch failed, 0 otherwise.

No debug

Run smashtest --no-debug to fail run if a MOD(~), MOD(~~), MOD($), exists anywhere in your tests. This way, you're certain you're running the full suite (and nobody accidentally committed their local debug modifiers).

Reports

Location

The live report is outputted to smashtest/report.html from the directory where smashtest was run.

If --report-history=true is set, the report will be at smashtest/reports/smashtest-[timestamp]/report.html.

Failed branches

Similar failed branches are grouped together. This not only shortens the report, it also helps you debug. For example, if upon expanding a group you see that the same test fails regardless of browser, you know the browser isn't the culprit.

Colors

In the report, steps that execute code are colored as passed, failed, running, skipped, or not run yet, while non-executable textual steps and functions are colored in plain gray. Branches themselves have similar colors.

What's in a name?

We believe that tests shouldn't be named.

Tests have a tendency to look very similar to other tests, and it's actually quite common to have branches that only differ by one step. Names tend to just be a crude summary of one or two steps (e.g., "logged-in cart test 016"). Having to think of a name only slows you down.

Smashtest identifies branches by a unique hash, and just shows a list of steps in the report, collapsed by similarity to keep things clean.

If you really want to name a test, use textual steps.

Performance constraints

Due to performance constraints, reports are limited to 500 branches of each type (passed, failed, etc.), and currently running is limited to 20.

Language

Indents

Smash, the language that's run by smashtest, uses exactly 4 spaces for each indent (no tabs).

Each step is indented to the step it follows (like comments on a website).

Blank lines

Blank lines are generally ignored. Use them for stylistic organization, or to group similar steps.

The only time they matter is in a step block. A step block cannot have blank lines between members, and must have one blank line below (if it has children). You can also use blank lines to prevent a step block from forming.

Scope

All files passed to smashtest will be concatenated into one long piece of text at runtime. Everything at indent 0 (e.g., function declarations) will be accessible in all other files.

For example:

Modifiers

Modifiers are symbols that come before or after the step:

Each modifier must be surrounded by spaces.

The only modifier which acts differently based on if it comes before vs. after the step is MOD(~)

Step blocks

Simple step blocks

Vertical list of 2 or more steps at the same indent, no blank lines in between. Must end in an empty line if it has children.

Multi-level step blocks

Anon

Named

Think of square brackets as a "big parenthesis" around multiple steps.

A named step block will look like a function call in the report. They keep things neat.

Always put modifiers before the [.

With code blocks

Sequential

More on the sequential modifier.

Textual steps (-)

Textual steps serve to mark and organize. They don't execute anything.

They're gray in the report (to differentiate them from executable steps).

Uses

• Can be a heading: grouping and describing the steps that come after and/or the whole branch
  • Alternatively, use functions or named step blocks to group multiple steps into one named task. They will appear as collapsible sections in the report.


• Can be a comment you want to appear in the report
  • For example, to describe the state of the app or test at that moment in time (or a state that's beginning or ending)
  • A requirement, description, id, or name
  • Describe a manual step


• Can be used to "comment out" an executable step (though it's recommended to use MOD(-s) or COM(//) in this case)

Recommended test structure

Textual steps are great for dividing tests based on category. In this example, the app is divided and further sub-divided into functionalities, ending in function calls that are implemented in another file. Notice how the "When" steps correspond to the category, while the "Givens" and "Thens" vary from test to test.

Functions (*, **)

Function calls

Function declarations

Patterns

Rules for matching calls to declarations

Simple case

Open Chrome COM(// 4CLOSEP looks among children of this step, finds line 14) Navigate to STR('/page1') COM(// 3CLOSEP looks among children of this step, finds nothing) Login COM(// 2CLOSEP looks among children of this step, finds nothing) My function COM(// 1CLOSEP function call *** START HERE ***) Click STR('something') Click STR('something else') Click STR('something else') Navigate to STR('/page2') Login FD(* My function) COM(// the one that's matched ("overrides" line 17CLOSEP) COM(// etc.) FD(* My function) COM(// etc.)

For every function call, searches for a matching function declaration among the children of that call's parent. If nothing is found, searches among the grandparent's children, then the great-grandparent's children, etc. until it searches among all declarations at indent 0 before erroring out.

Function calls and declarations are case insensitive, leading and trailing whitespace is ignored, and whitespace in the middle is always treated as a single space.

Don't forget to escape chars!

Since 'strings' and {vars} designate inputs in a function call, always use \', \", \[, \], \{, \} when using those actual characters inside a function call's text, to prevent an input from forming:

Clear userEC(\')s credentials MOD(-) Textual step's text COM(// not necessary for textual steps)

Matching multiple declarations

MOD(-) Matching multiple function declarations under the same parent FD(* F) A FD(* F) B F COM(// matches both line 2 and line 5) COM(// produces branches:) COM(// 1CLOSEP F, A) COM(// 2CLOSEP F, B)

Making vars available below

FD(* F) STR({name}) = STR('bob') COM(// this variable will be accessible after a function call to F) F Type STR({name}) into STR('textbox') COM(// will type 'bob')

Making funcs available below

FD(* F) FD(* A) B FD(* G) FD(* A) C FD(* A) D F COM(// makes public function A at line 2 available below) A COM(// B is run here) F COM(// makes public function A at line 2 available below) G COM(// makes public function A at line 6 available below) A COM(// C is run here)

Equivalents

COM(// file1.smash) COM(// -----------) FD(* A) FD(* B) MOD(-) 1 FD(* B) MOD(-) 2

COM(// file2.smash) COM(// -----------) FD(* A) FD(* B) MOD(-) 3

is equivalent to

FD(* A) FD(* B) MOD(-) 1 MOD(-) 2 MOD(-) 3

Calls to itself

FD(* Nav to homepage) Nav to STR('/') MOD(-) Some test Open Chrome Nav to homepage COM(// calls line 8) FD(* Nav to homepage) COM(// this "intercepts" navs to homepage to do security stuff) Do security checks Nav to homepage COM(// ignores line 8 because recursion not allowed, so calls line 1)

Variables

Using

Setting

{var} = 'str'

STR({variable}) = STR('string') COM(// everything in Smashtest is a string) STR({variable}) = STR("string") STR({variable}) = STR([string]) STR({variable}) is STR('string') COM(// same as =) STR({variable})=STR('11'), STR({variable})=STR('22'), STR({variable})=STR('33') STR({variable})=STR('{host}/path/name') STR({variable})=STR('{var2}') COM(// cloned a var) STR({ Variable Name With Caps and Whitespace }) = STR('string')

{var} = Func with code block

STR({variable}) = Get hello COM(// variable is set to "hello world!") FD(* Get hello) { JSKEYWORD(return) STR("hello ") + STR("world!"); COM(// any kind of js value will work (not just stringCLOSEP) }

STR({variable}) = Get goodbye { COM(// variable is set to "goodbye!") JSKEYWORD(return) STR("goodbye!"); }

{var} = Func with branches

FD(* A bad username) STR({x}) = STR('00') COM(// you can use any variable name, not just {x}) STR({x}) = STR('baduser') STR({x}) = STR('[none]') STR({x}) = STR('') STR({x}) = STR(' ') STR({username}) = A bad username Type STR({username}) into STR({textbox}) COM(// produces branches:) COM(// 1CLOSEP type '00') COM(// 2CLOSEP type 'baduser') COM(// 3CLOSEP type '[none]') COM(// 4CLOSEP type '') COM(// 5CLOSEP type ' ')

Variable names are case sensitive (unlike step names), leading and trailing whitespace is ignored, and whitespace in the middle is always treated as a single space.

They're case sensitive because they're converted to js vars in code blocks (and js is case-sensitive).

Escape sequences

You're not allowed to have a \ in a variable name, and you're not allowed to have a \0, \x, \u, or \c inside a 'string literal'. To get around this, use:

STR({variable}) = special char string { JSKEYWORD(return) STR("\u2665 \cJ"); }

Types

Inside a code block

Lookahead (:)

STR({var)LC(:)STR(}) will get the value of the variable when it's set later in the branch. This allows you to refactor common steps higher up into the tree.

Setting a lookahead var's value

You cannot set a lookahead var's value inside a code block, only in a STR({var})=STR('something') or a STR({var})=Function call (but the function call has to be sync - i.e., no awaits and an immediate return).

Code blocks

Types

Modifiers

Await

COM(// Code blocks are async, meaning you can use the 'await' keyword) Verify success { JSKEYWORD(await) JSFUNC($)(STR('#success')); }

Don't forget await!

Async js function calls should always be preceded with await. If errors are thrown but aren't captured inside a step, or if things just seem wonky, it's probably because you forgot an await.

Prev

Variables

Settings vars { JSFUNC(l)(STR('var1'), STR('new value')); COM(// local variable (any value will work, not just stringsCLOSEP) JSFUNC(g)(STR('var2'), STR('new value')); COM(// global variable) JSFUNC(p)(STR('var3'), STR('new value')); COM(// persistent variable) } Getting vars { JSKEYWORD(let) JSVARIABLE(v) = JSVARIABLE(var1); COM(// works if variable name is a single word with no special chars) JSKEYWORD(let) JSVARIABLE(v) = JSFUNC(l)(STR('var1')); COM(// local variable) JSKEYWORD(let) JSVARIABLE(v) = JSFUNC(g)(STR('var2')); COM(// global variable) JSKEYWORD(let) JSVARIABLE(v) = JSFUNC(p)(STR('var3')); COM(// persistent variable) }

Use l()/g()/p() to set variables

Don't set a {variable} with JSVARIABLE(variable) = STR('value'); as this will not persist past the end of the code block.

Be mindful when using 'let'

Don't use 'let' to declare a new variable with the same name as an existing {variable}, or you'll get a runtime error (double-declaration). You can use 'var' to get around this.

Timeouts

The default timeout for a step is 60 secs. If it hasn't completed by then, it will fail with a timeout error.

Steps generally have their own, more stringent timeouts as well. These are usually implemented by an (async) function called within the step that has its own timeout. We made the default timeout really high since we wanted to leave the more "realistic" timeout to each step's individual discretion.

The default timeout can be changed via JSFUNC(setStepTimeout)(secsCLOSEP for all steps after the current one in the current branch.

Errors

Code reference

These js functions and objects are accessible within any code block.

c()

dir()

g()

getGlobal()

getLocal()

getPersistent()

i()

l()

log()

p()

runInstance

setGlobal()

setLocal()

setPersistent()

setStepTimeout()

Sequential (..)

.. above a step block

.. on a step

Non-parallel (!, !!)

Comments

Standard use

Comments ignore whole lines

Inside code blocks

Groups and freq (#)

Groups

Only run Google branch: smashtest --groups=google

Only run branches in happy path OR pets: smashtest --groups="happy-path,pets"

Only run branches in happy path AND pets: smashtest --groups="happy-path+pets"

Only run Firefox branches: smashtest --groups=firefox (built-in group)

Only run branches in happy path AND pets, or firefox AND pets: smashtest --groups="happy-path+pets,firefox+pets"

Frequency

Run low/med/high tests: smashtest --min-frequency=low

Run med/high tests: smashtest --min-frequency=med

Run high tests: smashtest --min-frequency=high

Branches are also run in order of frequency, from high to low (and are shuffled within their freq).

These are the reserved hashtags used by smashtest

MOD(#desktop), MOD(#mobile),

MOD(#chrome), MOD(#firefox), MOD(#safari), MOD(#ie), MOD(#edge),

MOD(#low), MOD(#med), MOD(#high)

Conditional tests

It's best to avoid conditional tests. They should only be used to implement rare exceptions.

If step

If browser is...

If viewport is...

Skipping (-, -s, .s, $s)

Skip one step

Skip all branches passing through a step

Skip step and all steps below

Collapsing (+, +?)

Collapse (+)

If there's an error inside a MOD(+)'ed function, or if the function is currently running, it will be uncollapsed automatically.

It's recommended to mark precondition steps with MOD(+), since their details aren't central to the test.

Hidden (+?)

If there's an error inside a MOD(+?)'ed function, it will be visible in the report.

Only ($)

One $

Multiple $'s at different indents

Multiple $'s with the same parent

On a function declaration

With ~'s

Debug (~, ~~)

Debug modifier (~)

A branch run in debug will also pause after a failing step.

If a MOD(~) goes through multiple branches, the first one is chosen.

No report is generated and the list of previously passed branches (for -s) is retained.

Express debug modifier (~~)

Just like for MOD(~), no report is generated and the list of previously passed branches (for -s) is retained.

Debug flag

You can also run smashtest --debug=[hash] to debug the branch with the given hash.

Be careful. If you change any step, the hash of its branch will change. The recommended technique is to place a MOD($), MOD(~~), or MOD(~) on a line you change, then run that. You'll want to rerun all the branches that pass through that line anyway.

Hooks (***)

What's a hook?

A hook is a code block function that runs before or after a step, branch, or test suite.

They're not meant for testing or setup/teardown. They're for internal stuff, such as reporting, importing code, js function declarations, screenshots, and logging.

Only code blocks are allowed, and they cannot have children or modifiers. Hooks aren't listed in the report. If a hook fails, the step/branch it corresponds to will take the error and fail.

Types

Where should setup and teardown logic go?

Setup code should go in the actual test.

Teardown code should go in the same setup logic, such that the previous state is cleaned out prior to actual testing. It should go into a hook only if necessary.

UI Testing

Smashtest is a general platform for testing which comes with several built-in packages. One of these packages handles web UI testing with selenium webdriver.

This section discusses the steps and js functions that this package makes available.

Also, check out this UI test example.

Browsers and devices

Open a browser

All UI steps MUST come after an "Open [browser]" step

Open Chrome COM(// run exclusively with --groups=chrome) Open Firefox COM(// run exclusively with --groups=firefox) Open Safari COM(// run exclusively with --groups=safari) Open IE COM(// run exclusively with --groups=ie) Open Edge COM(// run exclusively with --groups=edge) Open browser STR('chrome') COM(// use a string recognized by webdriver as a browser name)

Set viewport size

Set device type

Usage examples

Capabilities

Sometimes you need to set custom browser capabilities and options, such as when you need to provide a username and password for a cloud service (BrowserStack, Sauce Labs, etc.)

Set custom capabilities

Set custom options

Browser steps

You must run an Open [browser] step before using any of the steps below.

{{element}} below can either be an ElementFinder string or an existing WebElement object.

All steps that involve mouse interaction (i.e., clicking) choose the first clickable element that matches {{element}}. If nothing found, they choose the first non-clickable element that matches {{element}}.

Interact

Set

Navigate

Window

Alerts

Wait

Avoid this kind of wait. Instead, consider Verify or Wait Until.

Wait STR('2') secs COM(// sleeps this long) Wait STR('2') seconds Wait STR('1') sec Wait STR('1') second

Cookies and storage

Print and log

Verify steps

You must run an Open [browser] step before using any of the steps below.

Verify

'Verify' steps wait up to 2 secs for the verify to pass before failing.

Verify STR('elementfinder') is visible Verify STR('elementfinder') is not visible Verify STR('elementfinder') is STR('state-elementfinder') Verify every STR('elementfinder') is STR('state-elementfinder') Verify at page STR('Page title') COM(// passes if current page title (case-insensitiveCLOSEP) Verify at page STR('site.com/page') COM(// or url contains this text) Verify at page STR('/page') Verify at page STR('Page .*') COM(// passes if current page title) Verify at page STR('(.*CLOSEPpage') COM(// or url matches this regex) Verify at page STR('^http(.*CLOSEPpage.+$') Verify cookie STR('name') contains STR('value') Verify alert contains STR('hello') COM(// passes if alert is open and contains this text) COM(// Note: this step doesn't wait up to 2 secs - it's immediate)

Wait until

'Wait until' steps wait up to 15 secs, or the specified amount, for the verify to pass before failing.

Wait until STR('elementfinder') is visible Wait until STR('elementfinder') is visible (up to STR('30') secs) Wait until STR('elementfinder') is not visible Wait until STR('elementfinder') is not visible (up to STR('30') secs) Wait until STR('elementfinder') is STR('state-elementfinder') Wait until STR('elementfinder') is STR('state-elementfinder') (up to STR('30') secs) Wait until every STR('elementfinder') is STR('state-elementfinder') Wait until every STR('elementfinder') is STR('state-elementfinder') (up to STR('30') secs) Wait until at page STR('Page title') COM(// passes if current page title (case-insensitiveCLOSEP) Wait until at page STR('site.com/page') COM(// or url contains this text) Wait until at page STR('/page') Wait until at page STR('Page .*') COM(// passes if current page title) Wait until at page STR('(.*CLOSEPpage') COM(// or url matches this regex) Wait until at page STR('^http(.*CLOSEPpage.+$') Wait until at page STR('Page title') (up to STR('30') secs) Wait until at page STR('site.com/page') (up to STR('30') secs) Wait until at page STR('/page') (up to STR('30') secs) Wait until at page STR('Page .*') (up to STR('30') secs) Wait until at page STR('(.*CLOSEPpage') (up to STR('30') secs) Wait until at page STR('^http(.*CLOSEPpage.+$') (up to STR('30') secs) Wait until cookie STR('name') contains STR('value') Wait until cookie STR('name') contains STR('value') (up to STR('30') secs)

Assert

Network conditions and throttling

Chrome only

Since Chrome is the only browser that supports emulating network conditions, this step only works in that browser. In all other browsers, this step quietly does nothing.

See conditional tests for ways of excluding other browsers, if it's cleaner that way.

You must run an Open [browser] step before using the step below.

Mocking time and geolocation

You must run an Open [browser] step before using any of the steps below.

Time

Geolocation

Stop

Mocking APIs

You must run an Open [browser] step before using any of the js functions listed below.

The following js functions work with both GET and POST

String response

JSON response

Detailed response

Function response

Regex endpoint

Stop mocks

Configure

Check out sinon's fake xhr and server (the underlying library) for more details on what you can do.

ElementFinders

What's an ElementFinder?

An ElementFinder (or EF) is a string that matches elements on a page.

They contain one or more lines, where each line is a comma-separated list of props. A prop (or property) describes an element, or the state of an element.

Alternatively, a list of props may be separated by spaces, but only if each item is either 'text', an ord (see below), or a prop that's already been defined (multiple words are ok, but no 'inputs').

Click STR(['Login' button]) COM(// 'Login' button is an EF, 'Login' and button are both props) Click STR([4th 'Login' button]) COM(// 4th (the ord propCLOSEP, must come first in space-separated EFs) Click STR('red button') COM(// if red button is not already defined, it will try to) COM(// interpret this as a list of two props, red and button)

Use space-separated inside steps

You should use space-separated EFs in steps (such as Click) because they sound natural and are easier to read. For example, Click STR([red 'login' button]) sounds better than Click STR([button, red, 'login']).

The JSFUNC($)() function is used to find an element given an EF. Since it throws an error if nothing is found in time, JSFUNC($)() can also be used to verify the existence (and visibility) of an element.

Likewise, JSFUNC($$)() finds multiple matching elements, given an EF.

To play around with EFs, run smashtest -r, open a browser, nav to a page, and type in various quoted STR('EFs') (or, you can run a test in debug). The browser console (DevTools) will contain logs for every EF match.

Props

Use defined props whenever possible

Although selectors can be valid props on their own, avoid steps like Click STR('#some-elem'). Instead, define a prop and use it in the step, e.g., Click STR('login box'). Your tests will be easier to read and refactorable.

Setting

Props are defined with the JSFUNC(props)() function. More on that on the code reference page.

On homepage { COM(// Define props for all the elements on the homepage) COM(// Format: 'prop name': `EF` or function) JSFUNC(props)({ STR('login box'): STR(`.msgbox, enabled, 2nd`), STR('about link'): STR(`selector "a[name='about']"`) }); }

Matching rules

Starting with all elements in the DOM, as props are applied (left to right), each one narrows down the list of elements. The elements that remain at the end are the ones that match the EF.

Whenever a prop is encountered, it is interpreted according to the first rule it matches in the following list:

1. 'text'
   • Matches elements where the given text is contained in its innerText, value, placeholder, associated label's innerText, or currently selected <option>'s innerText (for selects).
   • Case insensitive, leading and trailing whitespace is ignored, and all whitespace is treated as a single space (both the 'text' and text in the DOM).


2. 1st, 2nd, 3rd, etc. (ord)
   • E.g., 4th = take the elements currently matched and choose only the 4th one.
   • In a comma-separated prop list, you will usually list an ord last, since it narrows you down to just one element.
   • In a space-separated prop list, the ord must come first (it's more natural-sounding that way), but will be applied last.


3. defined prop
   • These are the definitions set by JSFUNC(props)()
   • They may take an input string, such as STR(propname 'input string')
     • You must use commas to separate these from other props
   • There are several props that come pre-defined


4. css selector
   • If nothing else matches, a prop is interpreted to be a css selector
   • You must use commas to separate these from other props

If an EF fails to match an element, it will throw an error containing the full EF, prettified, with a --> explanation next to the lines that didn't match.

Mind your selectors!

If you use a tagname selector and it actually matches the name of a defined prop, the defined prop will be used. To be safe, use STR(selector 'tagname') as your prop.

The same is true for selectors containing commas (ORs). Always use STR(selector '.selector1, .selector2') since commas separate props by default.

Implicit visible prop

By default, all EFs get an implicit STR(visible) prop (meaning, only match visible elements). The exception is when you explicitly use a STR(visible), STR(not visible), or STR(any visibility) prop. More on those here.

Not

Start a prop with "not" to find elements that don't match that prop. E.g., STR(not fuzzy) or STR(not .selector),

Counters

Match multiple elements by preceding a line with a counter.

Child elements

Simple example

COM(// Matches one or more .list elements that contain these 3 children, in that order:) JSKEYWORD(await) JSFUNC($$)(STR(`) STR( .list) STR( .item1) STR( .item2) STR( .item3) STR(`)); COM(// This is a subset matching, meaning that other elements) COM(// can exist in and around the 3 children inside .list) COM(// The top parent (.listCLOSEP can start at any indentation that's a multiple of 4)

Multi-level with counters

JSKEYWORD(await) JSFUNC($$)(STR(`) STR( 1+ x .list) COM(// matches 1 or more .list elements that contain these children:) STR( 4 x .item) COM(// 4 .item's) STR( button[name=q]) COM(// 1 button with attribute name set to "q") COM(// blank lines are ok) STR( .section) COM(// 1 .section that contains these children:) STR( #textbox) COM(// 1 #textbox) STR( enabled login box) COM(// 1 login box that's enabled) STR( button, contains 'click me') COM(// 1 button that contains 'click me') STR(`)); COM(// Note that // comments are allowed inside EFs)

Any order

JSKEYWORD(await) JSFUNC($$)(STR(`) STR( #list) STR( any order) COM(// the 3 children can be in any order) STR( .item, "NYC") STR( .item, "Tokyo") STR( .item, "Paris") STR(`));

Matching children

COM(// A [] around a line will match and return those elements,) COM(// as opposed to the top parent) COM(// This EF will match 4 elements:) COM(// a 'Tokyo' item and the 3 items that follow) JSKEYWORD(await) JSFUNC($$)(STR(`) STR( #list) STR( .item, 'NYC') STR( [.item, 'Tokyo']) STR( [3 x .item]) STR(`));

Mind your []'s!

To match the selector STR([attr="something"]), use the prop STR(selector '[attr="something"]'). Otherwise, the []'s will be interpreted as a matching operator.

Implicit body

JSKEYWORD(await) JSFUNC($$)(STR(`) STR( #one) COM(// multiple lines at the top indent) STR( #two) STR(`)); COM(// implicitly equates to) JSKEYWORD(await) JSFUNC($$)(STR(`) STR( body) STR( #one) STR( #two) STR(`));

Element array

Do stricter validations with element arrays:

Default ElementFinder props

Smashtest defines these props by default. Remember, you can put STR(not) in front of any of them.

• STR(visible) = matches if an element is visible to the user (width and height > 0, no hidden styles, opacity > 0, opacity of all ancestors > 0)
  • This prop is implicitly applied to all EFs, unless 'any visibility', 'visible', or 'not visible' are already in that EF
• STR(any visibility) = matches elements regardless of visibility, disables implicit 'visible' prop
  • If you define a new prop as STR('new prop'): STR(`#some-invisible-elem, any visibility`), you must use that new prop as Click STR('new prop, any visibility')
  • Likewise, if you define another prop as STR('even newer prop'): STR(`new prop, any visibility`), you must use that prop as Click STR('even newer prop, any visibility'), applying 'any visibility' all the way up the chain
• STR(enabled) = matches if 'disabled' attribute isn't present
• STR(disabled) = matches if 'disabled' attribute is present
• STR(checked) = matches if element.checked is true
• STR(unchecked) = matches if element.checked is false
• STR(selected) = matches if element.selected is true (used for selects)
• STR(focused) = matches if element currently has focus
• STR(element) = matches any element
• STR(clickable) = matches if element is of a clickable type (a, button, label, input, textarea, select, option) or has cursor:pointer style
• STR(page title 'title') = causes error if the page title isn't equal to the given string
• STR(page title contains 'title') = causes error if the page title doesn't contain the given string (case-insensitive)
• STR(page url 'url') = causes error if the page url isn't equal to the given string (relative or absolute)
• STR(page url contains 'url') = causes error if the page url doesn't contain the given string
• STR(next to 'text') = matches the one element that's closest in the DOM to the given text
  • Takes each elem and expands the container around it to its parent, parent's parent etc. until a container containing the text is found - matches that one element associated with that container (matches multiple elems if there's a tie)
• STR(value 'text') = matches if element.value is equal to the given text
• STR(contains 'text') = matches if the given text is contained in an element's innerText, value, placeholder, associated label's innerText, or currently selected <option>'s innerText (for selects).
  • Case insensitive, leading and trailing whitespace is ignored, and all other whitespace is treated as a single space (in both the 'text' and the DOM).
• STR('text') = same as STR(contains 'text')
• STR(contains exact 'text') = matches if an element's innerText, value, placeholder, associated label's innerText, or currently selected <option>'s innerText (for selects) equals the given text exactly
• STR(innertext 'text') = matches if an element's innerText contains the given text
• STR(selector 'selector') = matches if an element matches the given css selector
• STR(xpath 'xpath') = matches if an element matches the given xpath
• STR(style 'name:value') = matches if an element has the style with the given name set to the given value
• STR(position 'N') = matches the one element from the pool of currently matched elements that's in position N (where the first index is 1)
  • Same as ords (1st, 2nd, etc.)
• STR(textbox) = matches if an element is a textbox or textarea

Screenshots

When enabled, screenshots are taken before and after every step in branches that include an Open [browser] step. They're available in the report by clicking a step.

While good for debugging failures, screenshots can take up a lot of disk space and slow down test execution. Limit screenshots with these flags:

• --screenshots=[true/false]
  • set to true to enable screenshots


• --max-screenshots=[N]
  • doesn't take any more screenshots after N are taken
  • screenshots that are taken and later deleted due to --step-data don't count against N


• --step-data=[all/fail/none]
  • set to 'fail' to only retain screenshots (and other step data) for failed branches
  • set to 'none' to not retain any screenshots (or other step data)

Code reference

These js functions and objects are accessible within any code block after an Open [browser] step.

Don't forget await!

Async js function calls should always be preceded with await. If errors are thrown but aren't captured inside a step, or if things just seem wonky, it's probably because you forgot an await.

Finding elements

ElementFinder props

Executing JS in browser

The js executed inside the browser only has access to the args that were passed in and the vars/functions accessible inside the browser. All vars from the code block or test must be passed in as args.

executeScript()

JSKEYWORD(await) JSFUNC(executeScript)(JSKEYWORD(function)(CLOSEP { COM(// executes js in the browser) COM(// see webdriverjs's executeScript(CLOSEP) });

JSKEYWORD(let) JSVARIABLE(arg1) = STR('one'); JSKEYWORD(let) JSVARIABLE(arg2) = JSCONST(2); JSKEYWORD(let) JSVARIABLE(v) = JSKEYWORD(await) JSFUNC(executeScript)(JSKEYWORD(function)(arg1, arg2CLOSEP { COM(// arg1 and arg2 are accessible here) JSKEYWORD(return) JSVARIABLE(arg1) + JSVARIABLE(arg2); }, JSVARIABLE(arg1), JSVARIABLE(arg2)); COM(// v is "one2")

executeAsyncScript()

JSKEYWORD(await) JSFUNC(executeAsyncScript)(JSKEYWORD(function)(doneCLOSEP { COM(// executes js in the browser) COM(// must call done(CLOSEP callback at the end) COM(// see webdriverjs's executeAsyncScript(CLOSEP) JSFUNC(done)(); });

JSKEYWORD(let) JSVARIABLE(arg1) = STR('one'); JSKEYWORD(let) JSVARIABLE(arg2) = JSCONST(2); JSKEYWORD(let) JSVARIABLE(v) = JSKEYWORD(await) JSFUNC(executeAsyncScript)(JSKEYWORD(function)(arg1, arg2, doneCLOSEP { COM(// arg1 and arg2 are accessible here) JSFUNC(done)(JSVARIABLE(arg1) + JSVARIABLE(arg2)CLOSEP; }, JSVARIABLE(arg1), JSVARIABLE(arg2)); COM(// v is "one2")

browser

Mocking

mockTime()

Try to use the Mock time to {{date}} step instead.

JSKEYWORD(let) JSVARIABLE(date) = JSKEYWORD(new) JSCLASS(Date)(); JSKEYWORD(await) JSFUNC(mockTime)(JSVARIABLE(date)); COM(// set the browser's date to this one)

COM(/**) COM( * Mock's the current page's Date object to simulate the given time. Time will run forward normally.) COM( * See sinon's fake timers for more details) COM( * @param {Date} time - The time to set the browser to) COM( */) JSKEYWORD(await) JSFUNC(mockTime)(time);

mockLocation()

Try to use the Mock location steps instead.

JSKEYWORD(await) JSFUNC(mockLocation)(JSCONST(28.538336), JSCONST(-81.379234)); COM(// sets the browser's location to the given latitude and longitude)

mockHttp()

COM(// Responds with 200 'canned response' when an XHR GET in the browser tries to hit /endpoint on the current domain) COM(// See mocking APIs for more details) JSKEYWORD(await) JSFUNC(mockHttp)(STR('GET'), STR('/endpoint'), STR('canned response'));

COM(/**) COM( * Mocks the current page's XHR. Sends back the given response for any http requests to the given method/url from the current page.) COM( * You can use multiple calls to this function to set up multiple routes. If a request doesn't match a route, it will get a 404 response.) COM( * See sinon's fake xhr and server for more details) COM( * @param {String} method - The HTTP method ('GET', 'POST', etc.CLOSEP) COM( * @param {String or RegExp} url - A url or a regex that matches urls) COM( * @param response - A String representing the response body, or) COM( * An Object representing the response body (it will be converted to JSONCLOSEP, or) COM( * an array in the form [ [status code], { header1: "value1", etc. }, [response body string or object] ], or) COM( * a function) COM( * See server.respondWith(CLOSEP from sinon's documentation) COM( */) JSKEYWORD(await) JSFUNC(mockHttp)(method, url, response);

mockHttpConfigure()

JSKEYWORD(await) JSFUNC(mockHttpConfigure)({autoRespond: JSCONST(false)});

COM(/**) COM( * Sets configs on the currently mocked XHR) COM( * @param {Object} config - The options to set (key value pairsCLOSEP) COM( * See fake server options for details on what config options are available) COM( * Fails silently if no mock is currently active) COM( */) JSKEYWORD(await) JSFUNC(mockHttpConfigure)(config);

mockTimeStop()

JSKEYWORD(await) JSFUNC(mockTimeStop)(); COM(// stops time mocks and restores original time)

mockLocationStop()

JSKEYWORD(await) JSFUNC(mockLocationStop)(); COM(// stops geolocation mocks and restores original geolocation)

mockHttpStop()

JSKEYWORD(await) JSFUNC(mockHttpStop)(); COM(// stops http mocks and restores original endpoints)

mockStop()

Try to use the Stop all mocks step instead.

JSKEYWORD(await) JSFUNC(mockStop)(); COM(// stops all time, geolocation, and http mocks, restores originals)

injectSinon()

JSKEYWORD(await) JSFUNC(injectSinon)(); COM(// injects sinon js library into browser) COM(// sinon becomes accessible in browser via global var 'sinon') COM(// automatically called when one of the mocking functions above invoked) COM(// does nothing if sinon is already available inside browser)

API Testing

One of Smashtest's built-in packages handles HTTP API testing. This section discusses the js functions that this package makes available.

Here are two examples of API testing:

Request

See request's documentation for details on the functions below

request()

get()

post()

put()

patch()

del()

head()

options()

api.defaults()

Cookies

Response and verify

Simple example

Make a request { JSKEYWORD(await) JSFUNC(get)(STR('https://site.com/endpoint')); } Verify the response { COM(// The global variable 'response' is automatically filled with the last response) COM(// response.verify(CLOSEP checks that the actual response object matches) COM(// the expected response object that's passed in) JSVARIABLE(response).JSFUNC(verify)({ COM(// you can list zero or more of these expected keys:) statusCode: JSCONST(200), COM(// expected status code) headers: { COM(// expected headers) STR('content-type'): STR('application/json') }, error: JSCONST(null), COM(// expected error object from request library) body: { COM(// expected body (js obj if body is json, string otherwiseCLOSEP) one: STR('two') }, rawBody: STR('{"one":"two"}'), COM(// expected raw response body) responseObj: {} COM(// expected response object from request library) }); COM(// To access the response details:) JSVARIABLE(response).JSVARIABLE(statusCode); COM(// status code) JSVARIABLE(response).JSVARIABLE(headers); COM(// js obj where keys are header names) JSVARIABLE(response).JSVARIABLE(error); COM(// error obj from request library) JSVARIABLE(response).JSVARIABLE(body); COM(// js obj if body is json, string otherwise) JSVARIABLE(response).JSVARIABLE(rawBody); COM(// string containing unparsed body data) JSVARIABLE(response).JSVARIABLE(responseObj); COM(// response obj from request library) }

Test development pattern

A good pattern for developing API tests is to make a request step followed by a response step that just does JSFUNC(c)(JSVARIABLE(response)); (similar to console.log(response))

Then, manually verify the response in the console and copy it into a JSVARIABLE(response).JSFUNC(verify)(); in the response step.

Matching in response.verify()

Comparer

REPL

What's a REPL?

REPL stands for "read–eval–print loop". It's a way of driving Smashtest from the command-line by entering text commands.

Try it yourself. Run smashtest -r or smashtest --repl, or debug a file with the MOD(~) modifier.

Command list

• Entering a step to run it
  • One-line steps ok
  • Multiple lines ok if step has code block (first line is Step name { and last line is })
  • Just type in { to start an anonymous step with a code block
  • No function declarations, hooks, or step blocks allowed
  • When browser is open, a convenient step is STR([<ElementFinder here>]). It will print the elements found to the browser's console and the number of elements found to the console you're typing into.
• Ctrl + C = exit
• Enter key = run next step or exit (if no more steps ahead)
• s = skip next step
• p = repeat previous step
• r = resume running
• x = exit
• .break = sometimes you get stuck, this gets you out
• .clear = break, and also clear the local context
• .editor = enter editor mode
• .exit = exit
• .help = print a help message
• .load = load js from a file into this REPL session
• .save = save all evaluated commands in this REPL session to a file

Packages

Make a package

Share your steps and functions by distributing a package:

1. Implement functionality inside a js file.
2. Distribute that file as an npm package via the npm registry.
3. Distribute simple function declarations that users can download directly from you or copy into their .smash files:
   FD(* My awesome function) { JSFUNC(i)(STR('my_awesome_npm_package')).JSFUNC(myAwesomeFunction)(); COM(// i(CLOSEP is similar to require(CLOSEP) }

Use a package

To install a package:

1. npm install the package globally (via npm -g install), in the same directory as the .smash files that will use it, or in any directory above that.
2. Copy the above function declaration into one of your files, or download a .smash file with that declaration already inside it. Make sure it runs with the other .smash files when you run your tests.
3. Now you can use the step anywhere in your tests:
   My awesome function

Promote

We'll help promote your package! Simply contact us.

Contact Us

Say hello! We're very friendly :)

• hello@smashtest.io
• Gitter
• GitHub
• NPM
• LinkedIn
• Twitter
• Facebook

Not Found