Smashtest • Generate tests fast
Smashtest is an open-source tool and language for rapidly generating tests.
Greatly speed up your web automated testing by writing tests in a tree-like format.
Trees represent how we think when we're testing.
They allow us to list all the permutations that branch off from any given point.
Screenshots
Sample test
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | Open Chrome Open Firefox Open Safari Navigate to 'site.com' Click 'Sign In' Type {username:} into 'username box' {username} is 'joe' {username} is 'bob' {username} is 'mary' Verify success {username} is 'baduser' Verify error |
Getting Started
Start by installing Smashtest, then writing your first .smash file
Setup
1. Install NodeJS
Make sure you have NodeJS installed. We recommend the LTS version. Use node -v to check.
2. Install Selenium Webdriver (if you're doing web UI testing)
- Make sure all of the browsers you want to automate are installed
- Make sure you have Java installed. Use java -showversion to check.
- Choose one of the following options:
Option 1: Webdriver manager
Webdriver managers take care of the installation process for you, but usually require a second console to be open during test runs, require an additional command-line flag to be passed in, and might not have support for less-popular browsers, such as Safari. You have to run a single update command when a browser has a major release.
-
Mac users, consider using webdriver-manager, which supports Chrome and Firefox:
- In a new console, run npm install -g webdriver-manager (if that gives you permissions errors, put sudo before npm).
- Run webdriver-manager update to download the latest versions of everything.
- Run webdriver-manager start. This must always be running when executing tests.
-
Windows users, consider using selenium-standalone, which supports Chrome, Firefox, IE, and Edge:
- In a new console, run npm install -g selenium-standalone (if that gives you permissions errors, put sudo before npm).
- Run selenium-standalone install to download the latest versions of everything.
- Run selenium-standalone start. This must always be running when executing tests.
Whenever running Smashtest, always run it as smashtest --test-server=http://localhost:4444/wd/hub (or whatever the port is), or include that flag in the config file.
If you upgrade your browser, especially Chrome, make sure you run the update/install command (step 2) to make sure your drivers are in sync with the browser versions you have installed.
Option 2: Manual install
This option allows you to run everything from just one console, handles any browser, and doesn't require you to pass in additional command-line flags, but the install takes longer, and you have to do a manual update when a browser has a major release.
-
You'll need to download individual executables (called "drivers") for each browser you want to automate:
Note: Safari 10+ on MacOS comes pre-installed with SafariDriver
-
Each driver executable should be placed in your system's PATH
- On MacOS that's usually /usr/local/bin (run echo $PATH to find out)
- On Windows that's usually C:\Program Files (run echo %PATH% to find out)
-
Download the latest version of selenium standalone. For example, click the 3.9 folder, then download selenium-server-standalone-3.9.1.jar.
-
Set the SELENIUM_SERVER_JAR environment variable to the absolute path of the jar file you just downloaded.
- On MacOS add the line export SELENIUM_SERVER_JAR=[path here] to the file ~/.bash_profile, then restart your console
- On Windows run setx -m SELENIUM_SERVER_JAR "[path here]" (you may have to start a second command prompt as administrator), then restart your original command prompt
If you upgrade your browser, especially Chrome, make sure you re-download and install the correct driver versions.
Option 3: Cloud service or Grid
Alternatively, you can run Smashtest with a Selenium Grid. This is an advanced, distributed configuration.
-
To use a grid from a cloud service (Sauce Labs, BrowserStack, etc.):
- Set up the capabilities in your test. See your cloud service documentation for details.
- Set --test-server to your cloud service's url, or include that flag in the config file.
-
To install a grid locally:
- Follow steps 1-4 under Option 2 (but do not set the SELENIUM_SERVER_JAR variable)
- In a console, run java -jar selenium-server-standalone-[version].jar -role hub and keep it running
- In another console, run java -jar selenium-server-standalone-[version].jar -role node -hub http://localhost:4444/grid/register and keep it running
- Whenever running Smashtest, always run it as smashtest --test-server=http://localhost:4444/wd/hub (or whatever the port is), or include that flag in the config file
See the grid page for more information.
3. Install Smashtest
From a console, run npm install -g smashtest
(if that gives you permissions errors, put sudo before npm)
4. Install a grammar (highly recommended)
A grammar will highlight your code, making your .smash files pretty and much easier to read. They are currently available on the Atom and VSCode editors.
Atom
- Download Atom
- Install the Smashtest package
- Set configuration for smash files
- Ctrl/Cmd + Shift + P
- Type "open config" and hit enter (this will open your config.cson file)
-
Add the following to the bottom of the file:
12345".smash.source":editor:autoIndentOnPaste: falsecommentStart: "//"tabLength: 4
VSCode
- Download VSCode
- Install the Smashtest extension
Writing your first test
Here's what to do
- Create a new text file called "helloworld.smash"
-
Put the following into the file:
12345Open ChromeNavigate to 'google.com'Type 'hello world[enter]' into 'textbox'
- Open a console and cd to that file's directory
- Run smashtest
- Watch test run
- Open live report while test is running (see console output for path)
So what did I just do?
Smashtest comes with a lot of built-in steps, a few of which you just used.
Think of steps as being arranged like comments on a website. Just like comments are indented to the comment they're replying to, steps are indented to the step they come after. In this example, there's only one line of steps, hence there's only one test. We call tests branches.
So let's add a branch!
1 2 3 4 5 6 7 | Open Chrome Navigate to 'google.com' Type 'hello world[enter]' into 'textbox' Type 'hello universe[enter]' into 'textbox' |
Notice line 7. It's at the same indent level as line 5. That means both lines will follow line 3.
Therefore, this file will have two branches:
- Open Chrome, Navigate to 'google.com', Type 'hello world[enter]' into 'textbox'
- Open Chrome, Navigate to 'google.com', Type 'hello universe[enter]' into 'textbox'
Try running it yourself. Then, give yourself a pat on the back!
Debug and learn
Try this debugging technique...
1 2 3 4 5 6 7 | Open Chrome Navigate to 'google.com' ~ Type 'hello world[enter]' into 'textbox' Type 'hello universe[enter]' into 'textbox' |
Put a ~ in front of line 5.
That single branch will be isolated, the browser will run non-headless (you can see it), and the test will pause before that line. You can now use the console to try out steps, move to the next step, repeat a previous step, etc.
This mode is called the REPL. It's great for learning and debugging.
Development technique
Using ~ is actually a recommended test development technique:
- Write a step
- Put a ~ at the end of it
- Run smashtest, which runs the browser to that line
- Come up with all the permutations that can branch off from that point (using the browser as a guide)
- List them as steps, indented to the step with the ~
- Repeat
Examples
Feel free to play with this UI test example and this API test example.
Basic language syntax
Branches
1 2 3 4 5 6 7 8 9 10 | Open Chrome // executed in both branches Navigate to 'site.com' // executed in both branches Click 'one' // branch 1 ends here Click 'two' // branch 2 ends here // produces branches: // 1) open, nav, click 'one' // 2) open, nav, click 'two' |
1 2 3 4 5 6 7 8 9 10 11 | Open Chrome Navigate to 'google.com' Do a search // this step ends branch 1 Navigate to 'pets.com' Buy cat food // this step ends branch 2 Buy dog food // this step ends branch 3 |
Step blocks
1 2 3 4 5 6 7 | Open Chrome // this group of 5 steps is known as a "step block" (same indent, no blank lines in between) Open Firefox Open Safari Open Edge Open IE // one empty line under a step block is mandatory Navigate to 'site.com' // 5 separate branches end in this step |
Sequential
1 2 3 4 5 6 7 8 9 10 | .. Open Chrome Nav to 'site.com' Click 'button' // is the same as Open Chrome Nav to 'site.com' Click 'button' |
Textual steps
1 2 3 4 5 6 7 8 9 10 11 12 | This step is a function call // it executes an action - This step is a textual step // it's just a piece of text to organize your tests Look, I can put the "-" modifier at the end too! - Navigate to 'site.com' - Logged-in tests // etc. - Logged-out tests // etc. |
Code blocks
1 2 3 4 5 6 7 8 9 10 11 12 13 | Open Chrome Navigate to 'site.com' Click the logo { // this is a code block // you can do anything js or nodejs supports (await $('#logo')).click(); } // must end at the same indent level as the starting line |
Functions
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | Open Chrome Navigate to 'site.com' Click 'element' // all 3 steps are function calls to built-in ("packaged") functions * Log In // this is a function declaration Click 'log in box' Type 'username' into 'username box' Click 'ok' Open Chrome Navigate to 'site.com' Log In // this is a function call (it will execute the 3 login steps) Log Out // another function call log out // steps are case insensitive * Log Out { // this is a code block (await $('.logout-button')).click(); } |
Variables
1 2 3 4 5 6 | {username} = 'superman' // this sets the global variable 'superman' {username} is 'superman' // same as above {username} is "superman" // same as above {username} is [superman] // same as above Type '{username} is a handsome guy' into 'textbox' |
ElementFinders
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | Open Chrome Navigate to 'https://www.site.com' On homepage { // describe props, which are things on the page and/or the state of those things props({ // a special syntax for finding elements 'login button': `#login`, 'search results': ` #list .result, 'one' .result, 'two' `, 'groovy': `'contains this groovy text'`, 'retro button': `selector '.button', groovy` }); } Type 'hello world' into 'message box' // using a prop makes it easier to read and refactor |
Examples
Web UI
TodoMVC tests • what's being tested
REST API
OnWater API tests • what's being tested
How to read and run
main.smash is the main entry point for tests in both examples.
To run an example, clone the project, then run smashtest inside the example's directory.
Lessons
Running Tests
Command line
Run smashtest [files] [options]
[files] can be a filename or a glob
(e.g., tests/*.smash)
[options] are explained here.
All files are concatenated into one big piece of text, and everything at indent 0 (e.g., function declarations) is accessible to all the other files.
After branches are compiled, they are all run.
smashtest will run all .smash files in the current directory (but not sub-directories, unless --recursive is set).
Ordering of branches
Branches are run in order of frequency, high to low.
Otherwise, branches are shuffled randomly. Here's why:
- You can stop the run at any time and get a good sampling of different functionality
- Diversifies the browsers and devices used at any given time, so as to maximize the capabilities of a grid (such as selenium grid)
Disable with the --random=false flag.
Headless
For UI testing, browsers will run headless by default, except if you're debugging (in which case they will run visibly).
Safari, IE, and Edge don't support headless, so they always run visibly.
Override with the --headless=[true/false] flag.
Command-line options
Command-line flags vs. config file
Pass in options via command-line flags (e.g., smashtest test.smash --headless=false --max-parallel=7) or by setting them in a config file.
The config file must be in the same directory where smashtest is called from, must be valid json, and must be called smashtest.json:
1 2 3 4 | { "headless": false, "max-parallel": 7 } |
Command-line flags will override config file options when they conflict.
List of options
--debug=[hash]
Only run the branch with the given hash, in debug mode. Ignore $'s, ~'s, groups, and frequency.
--groups=[value] or --groups="group1,group2+group3"
Only run branches that are part of one of the groups covered by the expression.
Some group names are set automatically. For example, you can do --groups=chrome, --groups=firefox, --groups=safari, --groups=ie, --groups=edge to only run branches that use those browsers.
+ = AND, , = OR, where AND takes precendence. In other words, --groups="one,two+three,four" means run branches part of group one, or both two and three, or four.
--g:[name]=[value]
Sets a global variable before every branch.
--headless=[true/false]
Whether to run browsers as headless. Overrides default headless behavior (which is headless unless debugging).
--help or -?
Show a help prompt
--max-parallel=[N]
Do not run more than N branches simultaneously. 5 by default.
If you're hitting a selenium grid (i.e., with --test-server), set this high enough so as to max out the available slots (capabilities). Tests will block on Open [browser] if a capability isn't available yet.
--max-screenshots=[N]
Do not take more than N screenshots. No limit by default.
Screenshots taken but deleted after a branch passed do not count against N.
--min-frequency=[high/med/low]
Only run branches at or above this frequency. Set to "med" if omitted.
--no-debug
Fail if there are any $'s or ~'s. Useful to prevent debugging in CI.
--output-errors=[true/false]
Whether to output all errors to console. True by default.
Good for when you're developing tests and expect a few to fail (goes well with -s). Also good for unit tests that run quick and you don't want to switch to the report each time.
--p:[name]=[value]
Sets a persistent variable.
--random=[true/false]
Whether to randomize the order of branches. True by default.
--recursive
Scan the current directory and subdirectories for .smash files (when no filenames are passed in). Without this flag, only the current directory is scanned.
--repl or -r
Opens the REPL (drive Smashtest from command line)
--report-domain=[domain or domain:port]
Domain where the report server should run. This is where the report page gets its live updates.
Port indicates what port smashtest should run on. If omitted, an open port will be chosen, starting with port 9000.
Domain indicates the domain of the machine you're running smashtest on. When this option is omitted, localhost is chosen by default. Choose a domain other than localhost for when you're running tests in CI and want people to hit the report externally.
--report-history=[true/false]
Whether to keep a history of all reports by date/time. If true, will output each report and passed-data file to smashtest/reports/[datetime]/. Otherwise, will output each report and passed-data to smashtest/, possibly overriding the ones from the previous run.
--report-path="[absolute path]"
Sets a custom absolute path for the report and passed-data. Normally these files are outputted to smashtest/ (from the current directory), but if this flag is set, they will be outputted to [absolute path]/smashtest/.
--report-server=[true/false]
Whether to run a server during run for live report updates. Default is true.
--screenshots=[true/false]
Whether to take screenshots before and after each step. Default is false.
--skip-passed=[true/false/filename], -s, -a
Doesn't run branches that passed last time. Carries them and their passed state over into the new report.
Useful when you're fixing a breaking test and don't want to rerun the whole suite again, or when you only want to run new or updated tests.
-
--skip-passed=true
don't run branches that passed last time (carry them over), and look to ./smashtest/passed-data for the data
-
--skip-passed=false
run all branches that are supposed to run
-
--skip-passed=[filename]
don't run branches that passed last time, and use the given file as the record
- -s
same as --skip-passed=true
- -a
same as --skip-passed=false
Consider copying smashtest/passed-data to a different filename from time-to-time in order to save it. Then restore it by copying that file back in. This is because every new run will overwrite whatever's in smashtest/passed-data.
--step-data=[all/fail/none]
Keep step data for all steps, only failed steps, or no steps. Set to 'all' by default. Includes screenshots. Helps reduce size of reports.
--test-server=[url]
Location of the test server (e.g., http://localhost:4444/wd/hub for selenium server).
--version or -v
Outputs the version of Smashtest.
More memory
If you need more memory, set the NODE_OPTIONS nodejs variable.
For example, in MacOS you'd enter export NODE_OPTIONS="--max_old_space_size=[max MB to use]" && smashtest [files]
Selective test running
-s
You can skip running branches that passed in the previous run by running smashtest -s or by using the --skip-passed=true flag.
Useful when you're fixing a breaking test and don't want to rerun the whole suite again, or when you only want to run new or updated tests.
Only modifier ($)
1 2 3 4 5 6 7 8 9 | Open Chrome Navigate to 'google.com' Do a search $ Navigate to 'pets.com' // only runs branches that pass through this step Buy cat food // i.e., the branch ending here |
1 2 3 4 5 6 7 8 9 10 | Open Chrome $ Open Firefox Open Safari Desktop $ Mobile Navigate to 'google.com' Do a search // only runs the Firefox mobile branch that ends here |
1 2 3 4 5 6 7 8 | Open Chrome Open Firefox $ Open Safari Desktop Mobile $ Navigate to 'google.com' ~ // use to help isolate a branch for a debug |
Groups
1 2 3 4 5 6 7 8 | Open Chrome Open Firefox Navigate to 'google.com' #google #happy-path Do a search Navigate to 'pets.com' #pets #happy-path Buy cat food |
Only run Google branch: smashtest --groups=google
Only run branches in happy path OR pets: smashtest --groups="happy-path,pets"
Only run branches in happy path AND pets: smashtest --groups="happy-path+pets"
Only run Firefox branches: smashtest --groups=firefox (built-in group)
Only run branches in happy path AND pets, or firefox AND pets: smashtest --groups="happy-path+pets,firefox+pets"
Frequency
1 2 3 4 5 6 7 8 9 10 11 12 | Open Chrome Navigate to 'google.com' Do something you want tested very often #high // good for quick smoke tests Do something you want usually tested #med // your normal test suite Do something you want usually tested // #med is the default freq of a branch if #high/med/low is omitted Do something you want tested once in a while #low // good for long-running, low-risk, edge-casey stuff This branch will be med #med #some-group // the later step controls the branch's freq This branch will be low |
Run low/med/high tests: smashtest --min-frequency=low
Run med/high tests: smashtest --min-frequency=med
Run high tests: smashtest --min-frequency=high
Branches are also run in order of frequency, from high to low (and are shuffled within their freq).
CI/CD integration
Test server
If you're running your tests from a test server, such as selenium grid, include smashtest --test-server=[url] with the test server's url. A selenium server, for example, runs on http://localhost:4444/wd/hub by default.
If you're using a cloud service, be sure to set your capabilities in your tests as well.
Report server
Reports are outputted to ./smashtest/report.html from the directory smashtest is run from. Make sure users who access the report have access to the contents of the ./smashtest directory.
To ensure the report can receive live updates during a run, set smashtest --report-domain=[domain:port] where the domain is of the box running smashtest, and port is the port where you want report apis to be accessible.
Flakiness
After running smashtest, run smashtest -s one or more times to rerun failed tests (mitigating environmental or selenium flakiness).
Or, consider running smashtest -s --screenshots=true, so that failed tests always get rerun with screenshots on every step.
Exit codes
The smashtest process exits with exit code 1 if at least one branch failed, 0 otherwise.
No debug
Run smashtest --no-debug to fail run if a ~, ~~, $, exists anywhere in your tests. This way, you're certain you're running the full suite (and nobody accidentally committed their local debug modifiers).
Reports
Location
The live report is outputted to smashtest/report.html from the directory where smashtest was run.
If --report-history=true is set, the report will be at smashtest/reports/smashtest-[timestamp]/report.html.
Failed branches
Similar failed branches are grouped together. This not only shortens the report, it also helps you debug. For example, if upon expanding a group you see that the same test fails regardless of browser, you know the browser isn't the culprit.
Colors
In the report, steps that execute code are colored as passed, failed, running, skipped, or not run yet, while non-executable textual steps and functions are colored in plain gray. Branches themselves have similar colors.
What's in a name?
We believe that tests shouldn't be named.
Tests have a tendency to look very similar to other tests, and it's actually quite common to have branches that only differ by one step. Names tend to just be a crude summary of one or two steps (e.g., "logged-in cart test 016"). Having to think of a name only slows you down.
Smashtest identifies branches by a unique hash, and just shows a list of steps in the report, collapsed by similarity to keep things clean.
If you really want to name a test, use textual steps.
Performance constraints
Due to performance constraints, reports are limited to 500 branches of each type (passed, failed, etc.), and currently running is limited to 20.
Language
Indents
Smash, the language that's run by smashtest, uses exactly 4 spaces for each indent (no tabs).
Each step is indented to the step it follows (like comments on a website).
Blank lines
Blank lines are generally ignored. Use them for stylistic organization, or to group similar steps.
The only time they matter is in a step block. A step block cannot have blank lines between members, and must have one blank line below (if it has children). You can also use blank lines to prevent a step block from forming.
1 2 3 4 5 | Log in as 'joe' // this is a simple step block Log in as 'bob' Log in as 'mary' Do test stuff |
Scope
All files passed to smashtest will be concatenated into one long piece of text at runtime. Everything at indent 0 (e.g., function declarations) will be accessible in all other files.
For example:
1 2 3 4 | // file1.smash // ----------- Open Chrome Do test stuff // declared in file2.smash |
1 2 3 4 5 | // file2.smash // ----------- * Do test stuff Navigate to 'site.com' Click 'button' |
Modifiers
Modifiers are symbols that come before or after the step:
1 | ! - + This step is surrounded by modifiers ~ $ #med -s |
Each modifier must be surrounded by spaces.
The only modifier which acts differently based on if it comes before vs. after the step is ~
Step blocks
Simple step blocks
1 2 3 4 5 6 7 8 9 10 | Log in as 'joe' // this group of 3 steps is a step block Log in as 'bob' Log in as 'mary' // one empty line under a step block is mandatory Do test stuff // produces branches: // 1) log in as joe, do test stuff // 2) log in as bob, do test stuff // 3) log in as mary, do test stuff |
Vertical list of 2 or more steps at the same indent, no blank lines in between. Must end in an empty line if it has children.
Multi-level step blocks
Anon
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | open chrome nav to 'searchengine.com' [ type 'hello world[enter]' into 'search box' type 'hello world' into 'search box' click 'search' ] verify search results // called after each leaf in the bracketed branches // produces branches: // 1) open, nav, type w/ enter, verify // 2) open, nav, type, click, verify |
Named
1 2 3 4 5 6 7 8 9 10 11 | open chrome nav to 'searchengine.com' enter search terms [ // same as the first example, but names the step block type 'hello world[enter]' into 'search box' type 'hello world' into 'search box' click 'search' ] verify search results |
Think of square brackets as a "big parenthesis" around multiple steps.
A named step block will look like a function call in the report. They keep things neat.
Always put modifiers before the [.
With code blocks
1 2 3 4 5 6 7 8 9 10 11 12 | // these 3 steps form a step block, even though they have code blocks Click one { (await $('#one')).click(); } Click two { (await $('#two')).click(); } Click three { (await $('#three')).click(); } Verify action was completed |
Sequential
1 2 3 4 5 6 7 8 9 10 | .. Open Chrome Nav to 'site.com' Click 'button' // is the same as Open Chrome Nav to 'site.com' Click 'button' |
More on the sequential modifier.
Textual steps (-)
1 2 3 4 5 6 7 8 9 | - Tests with a logged-in user // a textual step Log in as 'bob' Log in as 'mary' // etc. - Tests with invalid users // a textual step Log in as '' Log in as 'baduser' // etc. |
Textual steps serve to mark and organize. They don't execute anything.
They're gray in the report (to differentiate them from executable steps).
Uses
-
Can be a heading: grouping and describing the steps that come after and/or the whole branch
- Alternatively, use functions or named step blocks to group multiple steps into one named task. They will appear as collapsible sections in the report.
-
Can be a comment you want to appear in the report
- For example, to describe the state of the app or test at that moment in time (or a state that's beginning or ending)
- A requirement, description, id, or name
- Describe a manual step
- Can be used to "comment out" an executable step (though it's recommended to use -s or // in this case)
Recommended test structure
Textual steps are great for dividing tests based on category. In this example, the app is divided and further sub-divided into functionalities, ending in function calls that are implemented in another file. Notice how the "When" steps correspond to the category, while the "Givens" and "Thens" vary from test to test.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | Given I am at my note-taking app - Notes - Creating - with a normal string Given no notes exist Given notes exist When a note is created Then it is properly displayed - with a whitespace-only string When a note is created with an empty string When a note is created with whitespace only Then no note is created // ... - Updating // ... - Deleting // ... - Login // ... - Register // ... - Search // ... |
Functions (*, **)
Function calls
1 2 3 4 5 6 7 | Function call here Function call with 'strings' and "strings" and [strings] as inputs Function call with {variables} and {{variables}} as inputs Function call with '{variables} inside strings' as inputs |
Function declarations
Public
1 2 3 4 5 | * Function declaration here Navigate to 'site.com' Check 'checkbox' Function declaration here // function call, runs the nav and check steps above |
A function declaration is accessible to all steps under its parent (or all steps, if declared at indent 0).
With inputs
1 2 3 4 5 | * Function declaration that {{takes}} in {{inputs}} Navigate to 'site.com/{{takes}}/{{inputs}}' Check 'checkbox' Function declaration that 'string input' in {var input} // function call |
With brackets
1 2 3 4 5 | * Log In [ // optional brackets Click 'login button' Type 'username' into 'username box' Click 'sign in button' ] |
With code block
1 2 3 4 5 | * Log In { (await $('.login-button')).click(); (await $('.username')).sendKeys('username'); (await $('.signin-button')).click(); } |
1 2 3 4 5 6 7 8 9 | * Log In { (await $('.login-button')).click(); } Type 'username' into 'username box' Click sign-in { (await $('.signin-button')).click(); } |
Multiple branches
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | * Nav to the cart page // has 2 branches, which represent 2 ways of doing this thing Navigate to '/' Click 'cart button' Navigate to '/cart' Nav to the cart page Click 'checkout' // produces branches: // 1) nav to /, click cart, click checkout // 2) nav to /cart, click checkout |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | * Choose a browser Open Chrome Open Firefox * Choose a viewport Desktop Mobile Choose a browser Choose a viewport Do a test // produces branches: // 1) open chrome, desktop, do a test // 2) open firefox, desktop, do a test // 3) open chrome, mobile, do a test // 4) open firefox, mobile, do a test |
Declarations inside declarations (calling in context)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | * On Desktop // when this is called, all child function declarations are made accessible to future steps * On Homepage * Logout // logout actions for desktop homepage * On Cart page * Logout // logout actions for desktop cart page * On Mobile * On Homepage * Logout // logout actions for mobile homepage * On Cart page * Logout // logout actions for mobile cart page * On Desktop Desktop // built-in step for desktop viewport * On Mobile Mobile // built-in step for mobile viewport .. Open Chrome On Desktop Navigate to '/cart' On Cart page Logout // executes the logout for the desktop cart page |
Gherkin
1 2 3 4 5 6 7 | * I log in // matches all 4 function calls below // etc. Given I log in // if an exact match cannot be found, gherkin (given/when/then/and) is stripped When I log in Then I log in And I log in |
{var} = F
1 2 3 4 5 6 7 8 9 10 11 12 | * Choose a username {x} = 'bob' {x} = 'joe' {x} = 'mary' {username} = Choose a username Type {username} into 'username box' // produces branches: // 1) type 'bob' // 2) type 'joe' // 3) type 'mary' |
Private
1 2 3 4 5 6 7 8 9 10 11 | * On cart page ** Private function // not made accessible after a call to "On cart page" // etc. Private function // call is ok Something Private function // call is ok On cart page Private function // compile-time error |
Hooks
1 2 3 | *** Before Every Branch { // stuff to do before every branch begins } |
Patterns
Encapsulating and refactoring
1 2 3 4 5 6 7 8 | * Order dinner // multiple branches for different ways of accomplishing the same thing - Variant 1 Add beans to meal Add rice to meal - Variant 2 Add rice to meal Add beans to meal |
Organizing
1 2 3 4 5 6 7 8 9 10 11 12 13 | // main-tests.smash - bird's eye view of all tests helps ensure we have full coverage // ---------------- - Test app - Homepage tests Display homepage test - Cart tests Empty cart test Full cart test - Search tests Empty search test Base case search test |
1 2 3 4 5 6 7 | // cart-tests.smash // ---------------- * Empty cart test // etc. * Full cart test // etc. |
Dividing a single declaration into multiple files
1 2 3 4 5 6 7 8 9 10 11 | // logout.smash // ------------ * On Desktop // this func declaration split into multiple files (to keep logout together) * On Homepage * Logout // etc. * On Mobile * On Homepage * Logout // etc. |
1 2 3 4 5 6 7 8 9 10 11 | // search.smash // ------------ * On Desktop // this func declaration split into multiple files (to keep search together) * On Homepage * Do a search // etc. * On Mobile * On Homepage * Do a search // etc. |
"On" pattern
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | // Call a function starting with "On" to indicate that this is the current state // (and not an action to be performed, such as navigating there) // e.g., "On [page]" can do verifications and set up props/functions for that page * On cart page { // call when on the cart page // set up ElementFinder props for everything on this page props({ 'list of items': ` #list .item .item .item `, 'checkout button': `#checkout` }); } // do some initial page verifications Verify at page '/cart' Verify 'list of items' is visible // expose cart-related functions * Add item to cart // etc. Open Chrome Nav to '/cart' On cart page Add item to cart |
Enforce permutations
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | // this pattern ensures that all permutations of these function calls are implemented below On Desktop On Mobile Logged In Logged Out Verify something // compile-time error if we're missing a permutation here: * On Desktop * Logged In * Verify something // etc. * Logged Out * Verify something // etc. * On Mobile * Logged In * Verify something // etc. * Logged Out * Verify something // etc. |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | * Try every string permutation // calling this function ensures that all 3 permutations String is empty // are implemented directly below, so you don't forget String is whitespace String is normal // in another file... Type {string:} into 'textbox' Try every string permutation // error if any of the 3 are missing * String is empty {string} = '' // ... * String is whitespace {string} = ' ' // ... * String is normal {string} = 'normal' // ... |
Rules for matching calls to declarations
Simple case
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | Open Chrome // 4) looks among children of this step, finds line 14 Navigate to '/page1' // 3) looks among children of this step, finds nothing Login // 2) looks among children of this step, finds nothing My function // 1) function call *** START HERE *** Click 'something' Click 'something else' Click 'something else' Navigate to '/page2' Login * My function // the one that's matched ("overrides" line 17) // etc. * My function // etc. |
For every function call, searches for a matching function declaration among the children of that call's parent. If nothing is found, searches among the grandparent's children, then the great-grandparent's children, etc. until it searches among all declarations at indent 0 before erroring out.
Function calls and declarations are case insensitive, leading and trailing whitespace is ignored, and whitespace in the middle is always treated as a single space.
Since 'strings' and {vars} designate inputs in a function call, always use \', \", \[, \], \{, \} when using those actual characters inside a function call's text, to prevent an input from forming:
1 2 | Clear user\'s credentials - Textual step's text // not necessary for textual steps |
Matching multiple declarations
1 2 3 4 5 6 7 8 9 10 11 12 | - Matching multiple function declarations under the same parent * F A * F B F // matches both line 2 and line 5 // produces branches: // 1) F, A // 2) F, B |
Making vars available below
1 2 3 4 5 | * F {name} = 'bob' // this variable will be accessible after a function call to F F Type {name} into 'textbox' // will type 'bob' |
Making funcs available below
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | * F * A B * G * A C * A D F // makes public function A at line 2 available below A // B is run here F // makes public function A at line 2 available below G // makes public function A at line 6 available below A // C is run here |
Equivalents
1 2 3 4 5 6 7 | // file1.smash // ----------- * A * B - 1 * B - 2 |
1 2 3 4 5 | // file2.smash // ----------- * A * B - 3 |
is equivalent to
1 2 3 4 5 | * A * B - 1 - 2 - 3 |
Calls to itself
1 2 3 4 5 6 7 8 9 10 | * Nav to homepage Nav to '/' - Some test Open Chrome Nav to homepage // calls line 8 * Nav to homepage // this "intercepts" navs to homepage to do security stuff Do security checks Nav to homepage // ignores line 8 because recursion not allowed, so calls line 1 |
Variables
Using
1 2 3 4 5 | // In a function call Buy tickets for {num adults} adults and {{num children}} children // In a string literal Nav to '{host}/path/name' // {host} is replaced with its value |
Setting
{var} = 'str'
1 2 3 4 5 6 7 8 9 10 11 | {variable} = 'string' // everything in Smashtest is a string {variable} = "string" {variable} = [string] {variable} is 'string' // same as = {variable}='11', {variable}='22', {variable}='33' {variable}='{host}/path/name' {variable}='{var2}' // cloned a var { Variable Name With Caps and Whitespace } = 'string' |
{var} = Func with code block
1 2 3 4 5 | {variable} = Get hello // variable is set to "hello world!" * Get hello { return "hello " + "world!"; // any kind of js value will work (not just string) } |
1 2 3 | {variable} = Get goodbye { // variable is set to "goodbye!" return "goodbye!"; } |
{var} = Func with branches
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | * A bad username {x} = '00' // you can use any variable name, not just {x} {x} = 'baduser' {x} = '[none]' {x} = '' {x} = ' ' {username} = A bad username Type {username} into {textbox} // produces branches: // 1) type '00' // 2) type 'baduser' // 3) type '[none]' // 4) type '' // 5) type ' ' |
Variable names are case sensitive (unlike step names), leading and trailing whitespace is ignored, and whitespace in the middle is always treated as a single space.
They're case sensitive because they're converted to js vars in code blocks (and js is case-sensitive).
You're not allowed to have a \ in a variable name, and you're not allowed to have a \0, \x, \u, or \c inside a 'string literal'. To get around this, use:
1 2 3 | {variable} = special char string { return "\u2665 \cJ"; } |
Types
{Global}
Global variables are accessible to all steps for the rest of the branch (inside and outside function calls).
1 2 3 4 5 | {variable} = 'string' // global variable F * F Type {variable} into 'textbox' // accessible here |
1 2 3 4 5 | F Type {variable} into 'textbox' // accessible here * F {variable} = 'string' |
{{Local}}
Local variables are only accessible inside the current function call.
1 2 3 4 5 6 | {{variable}} = 'string' // local variable F Type {{variable}} into 'textbox' // accessible here * F Type {{variable}} into 'textbox' // NOT accessible here |
1 2 3 4 5 6 7 | F Type {{variable}} into 'textbox' // NOT accessible here * F {{variable}} = 'string' Type {{variable}} into 'textbox' // accessible here // goes out of scope here |
Persistent
Persistent variables exist for the lifetime of the whole suite run. They can only be get and set inside code blocks. They are usually used for internal stuff, like storing functions and libraries.
Inside a code block
Getting
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | {variable} = 'something' Get a variable { let v = variable; // just use it as a js variable // for this to work, variable name must be a single word, // no chars other than A-Z, a-z, 0-9, - _ . // and not have the same name as a js keyword // when different types of vars have the same name, // local takes precedence over global, which takes precedence over persistent } Get a local variable { let v = l('variable name'); } Get a global variable { let v = g('variable name'); } Get a persistent variable { let v = p('variable name'); } |
Setting
1 2 3 4 5 6 7 8 9 10 11 | Set a local variable { l('variable name', 'new value'); // any kind of js value will work (not just string) } Set a global variable { g('variable name', 'new value'); } Set a persistent variable { p('variable name', 'new value'); } |
Lookahead (:)
{var:} will get the value of the variable when it's set later in the branch. This allows you to refactor common steps higher up into the tree.
1 2 3 4 5 6 7 8 9 10 11 12 13 | Type {username:} into 'login box' // ignores current value of {username}, looks to the first // {username}='str' line further in the branch {username} = 'bob' {username} = 'mary' {username} = 'vishal' Verify success {username} = 'baduser' {username} = '[none]' {username} = '' Verify error |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | Choose {adults:} and {children:} from reservations panel {adults}='[none]' {adults}='0' {children}='[none]' {children}='0' {children}='1' {children}='8' Verify error {adults}='1' {adults}='8' {children}='[none]' {children}='0' {children}='1' {children}='8' Verify success |
You cannot set a lookahead var's value inside a code block, only in a {var}='something' or a {var}=Function call (but the function call has to be sync - i.e., no awaits and an immediate return).
Code blocks
Types
1 2 3 4 5 6 7 | * Function name { // this is a code block // you can do anything js or nodejs supports // only lines between the "{" and "}" lines are part of the code block } // must end at the same indent level as the starting line |
1 2 3 4 | Step name { // this step is implemented in here // kind of like a one-time function call } |
Modifiers
1 2 3 4 5 6 | // Modifiers can come before the name, or after the name but before the "{" .. + * Function name $ ! { } .. + Step name $ ! { } |
Await
1 2 3 4 | // Code blocks are async, meaning you can use the 'await' keyword Verify success { await $('#success'); } |
Async js function calls should always be preceded with await. If errors are thrown but aren't captured inside a step, or if things just seem wonky, it's probably because you forgot an await.
Prev
1 2 3 4 5 6 7 | // Pass a value from one code block to the next via 'return' and 'prev' Step one { return "hello"; } Step two { } |
1 2 3 4 5 6 7 8 9 | Step one { return "hello"; } Some intermediate step Step two { expect(prev).to.equal("hello"); } |
Variables
1 2 3 4 5 6 7 8 9 10 11 12 | Settings vars { l('var1', 'new value'); // local variable (any value will work, not just strings) g('var2', 'new value'); // global variable p('var3', 'new value'); // persistent variable } Getting vars { let v = var1; // works if variable name is a single word with no special chars let v = l('var1'); // local variable let v = g('var2'); // global variable let v = p('var3'); // persistent variable } |
Don't set a {variable} with variable = 'value'; as this will not persist past the end of the code block.
Don't use 'let' to declare a new variable with the same name as an existing {variable}, or you'll get a runtime error (double-declaration). You can use 'var' to get around this.
Timeouts
The default timeout for a step is 60 secs. If it hasn't completed by then, it will fail with a timeout error.
Steps generally have their own, more stringent timeouts as well. These are usually implemented by an (async) function called within the step that has its own timeout. We made the default timeout really high since we wanted to leave the more "realistic" timeout to each step's individual discretion.
The default timeout can be changed via setStepTimeout(secs) for all steps after the current one in the current branch.
Errors
Normal error
1 2 3 | Failing step { throw new Error("oops"); // this step and branch will fail and end immediately } |
Error.continue
1 2 3 4 5 | Failing step { let e = new Error("verify failed, but let's try the next verify anyway"); e.continue = true; // this error will fail the step and branch, throw e; // but the branch will continue running } |
Error.fatal
1 2 3 4 5 | Failing step { let e = new Error("something really bad"); e.fatal = true; // this error will end the whole test suite execution immediately throw e; } |
Stack traces
A stack trace will contain something like this, where [LINE NUMBER] represents the line in the .smash file where the error occurred:
at CodeBlock_for_[NAME OF CODE BLOCK FUNCTION] (eval at ...), <anonymous>:[LINE NUMBER]:[COL NUMBER]
Implement complex functions in their own js files to generate more traditional stack traces:
1 2 3 4 5 6 | // test.smash // ---------- Step { const yf = i('yf', './yourfile.js'); yf.something(); } |
1 2 3 4 5 6 | // yourfile.js // ---------- function something() { // etc. } module.exports.something = something; |
Code reference
c()
1 2 | c('print to console'); // same as console.log(), but prints it out more neatly // (and clear of the progress bar) |
dir()
1 | dir(); // returns the absolute directory of the file where this step is |
g()
1 2 | g('variable', 'new value'); // set global variable g('variable'); // get global variable |
getGlobal()
1 | getGlobal('variable'); // get global variable |
getLocal()
1 | getLocal('variable'); // get local variable |
getPersistent()
1 | getPersistent('variable'); // get persistent variable |
i()
1 2 3 4 5 6 7 8 9 | const packageName = i('package-name'); // require()'s (imports) the given nodejs package, // sets persistent var 'packageName' to it (auto-camelCased), // and returns it const myPkg = i('myPkg', 'package-name'); // same, but sets the name of the persistent var i('myPkg', './path/to/file.js'); // works with js files too i('myPkg', '../path/to/file.js'); i('myPkg', '/Users/Shared/tests/file.js'); |
l()
1 2 | l('variable', 'new value'); // set local variable l('variable'); // get local variable |
log()
1 | log('text to log'); // logs a piece of text to the report for this step |
p()
1 2 | p('variable', 'new value'); // set persistent variable p('variable'); // get persistent variable |
runInstance
1 2 3 4 5 6 7 8 9 10 11 12 | runInstance; // represents the test runner "thread" that's // running this step and branch (see RunInstance) runInstance.runner; // represents the test runner (see Runner) runInstance.tree; // represents the whole tree (see Tree) runInstance.currBranch; // represents the current branch (see Branch) runInstance.currStep; // represents the current step (see Step) // These objects can be used to dynamically view, create, and/or edit tests at runtime |
setGlobal()
1 | setGlobal('variable', 'new value'); // set global variable |
setLocal()
1 | setLocal('variable', 'new value'); // set local variable |
setPersistent()
1 | setPersistent('variable', 'new value'); // set persistent variable |
setStepTimeout()
1 | setStepTimeout(30); // sets step timeout to 30 secs for all steps in this branch after this one |
Sequential (..)
.. above a step block
Simple case
1 2 3 4 5 6 7 8 9 10 11 | Nav to '/page' .. // makes the whole step block run sequentially Type '1111' into 'textbox' Type '2222' into 'textbox' Type '3333' into 'textbox' Verify success // produces 1 branch: // 1) nav, type 1111, type 2222, type 3333, verify success |
Function calls as step block members
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | // Note: Acts differently from function calls under a .. step. // If a function call has multiple branches, multiple branches will be generated: * Go to cart // two different ways of getting to the cart Nav to '/cart' Click 'cart icon' .. Go to cart Add peanuts to cart Verify peanuts added // produces branches: // 1) nav to /cart, add peanuts, verify peanuts // 2) click cart icon, add peanuts, verify peanuts |
.. on a step
Simple case
1 2 3 4 5 6 7 8 9 | Sequential test .. // flatten branches at or below me into one sequential branch One Two Three Four Five // produces 1 branch: // 1) sequential test, one, two, three, four, five |
With a step block
1 2 3 4 5 6 7 8 9 10 | Nav to '/page' .. Type '1111' into 'textbox' Type '2222' into 'textbox' Type '3333' into 'textbox' Verify success // produces 1 branch: // 1) nav, type 1111, verify success, type 2222, verify success, type 3333, verify success |
With a function call
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | * Type in Type '1111' into 'textbox' Type '2222' into 'textbox' Type '3333' into 'textbox' Nav to '/page' .. Type in Verify success // produces 1 branch: // 1) nav, type 1111, verify success, type 2222, verify success, type 3333, verify success Nav to '/page' Type in .. // all we did was move the .. down one line Verify success // produces the same branch as before |
Inside a function declaration
1 2 3 4 5 6 7 8 9 10 11 | * Open cart .. // the 3 steps here execute sequentially Nav to '/' Click 'cart icon' Verify 'cart' is visible Open cart // it's sequential inside, but not sequential out here Do stuff Do more stuff // produces 1 branch: // 1) open cart (nav to /, click cart icon, verify cart), do stuff, do more stuff |
Non-parallel (!, !!)
!
1 2 3 4 5 | {username} is 'pete' ! // no two branches going through this step may execute simultaneously Nav to '/' // useful for testing a stateful shared resource, like a test account // etc. Nav to '/page1' // etc. |
!!
1 2 3 4 5 | {username} is 'bob' !! // no two branches going through this step may execute simultaneously, Nav to '/' // unless --test-server was set // etc. // useful for things like safaridriver, which can't run more Nav to '/page1' // than one instance locally, but can handle multiple instances // etc. // in a selenium grid |
Comments
Standard use
1 2 3 | // full-line comment Step // comment at the end of a step |
Comments ignore whole lines
1 2 3 4 5 6 7 8 9 | Open Chrome // this is still a valid step block // Open Firefox // if whole line starts with //, it's ignored as if it weren't there Open Safari Do something // produces branches: // 1) open chrome, do something // 2) open safari, do something |
Inside code blocks
1 2 3 4 | Code block step { // normal js comments occur here /* normal js comments occur here */ } |
Groups and freq (#)
Groups
1 2 3 4 5 6 7 8 | Open Chrome Open Firefox Navigate to 'google.com' #google #happy-path Do a search Navigate to 'pets.com' #pets #happy-path Buy cat food |
Only run Google branch: smashtest --groups=google
Only run branches in happy path OR pets: smashtest --groups="happy-path,pets"
Only run branches in happy path AND pets: smashtest --groups="happy-path+pets"
Only run Firefox branches: smashtest --groups=firefox (built-in group)
Only run branches in happy path AND pets, or firefox AND pets: smashtest --groups="happy-path+pets,firefox+pets"
Frequency
1 2 3 4 5 6 7 8 9 10 11 12 | Open Chrome Navigate to 'google.com' Do something you want tested very often #high // good for quick smoke tests Do something you want usually tested #med // your normal test suite Do something you want usually tested // #med is the default freq of a branch if #high/med/low is omitted Do something you want tested once in a while #low // good for long-running, low-risk, edge-casey stuff This branch will be med #med #some-group // the later step controls the branch's freq This branch will be low |
Run low/med/high tests: smashtest --min-frequency=low
Run med/high tests: smashtest --min-frequency=med
Run high tests: smashtest --min-frequency=high
Branches are also run in order of frequency, from high to low (and are shuffled within their freq).
Conditional tests
If step
1 2 3 4 5 | If A then B { if(A) { // only does B if A is true B(); } } |
If browser is...
1 2 3 4 5 6 7 8 9 10 | - Test something Do not allow Safari { if(browser.params.name == 'safari') { // pass and end the whole branch if the browser is safari, do nothing otherwise runInstance.currBranch.markBranch('pass'); } } - This step only runs if the browser isn't safari |
If viewport is...
1 2 3 4 5 6 7 8 9 10 | - Test something If mobile { if(!runInstance.currBranch.groups.includes('mobile')) { // pass and end the whole branch if the viewport isn't mobile, do nothing otherwise runInstance.currBranch.markBranch('pass'); } } - This step only runs if the viewport is mobile |
Skipping (-, -s, .s, $s)
Skip one step
-s (recommended)
1 2 3 | One // runs Skipped step -s // doesn't run, marked skipped in report Two // runs |
-
1 2 3 | One // runs Skipped step - // doesn't run, regular textual step in report Two // runs |
//
1 2 3 | One // runs // Skipped step // doesn't run, not outputted to report Two // will run, but needs to be dedented once |
Skip all branches passing through a step
$s
1 2 3 4 5 6 7 | One // doesn't run Two // doesn't run Skipped step $s // skips any branch that passes through this step, still expands function calls (error if declaration not found) Three // doesn't run Four // doesn't run |
Skip step and all steps below
.s
1 2 3 4 5 6 7 8 9 10 | One // runs Two // runs Skipped step .s // doesn't run, still expands function calls (error if declaration not found) Three // doesn't run Four // doesn't run // Also skips entire duplicate branches caused by .s // Be careful, when .s is inside a function declaration it will skip steps after the function call as well |
//
1 2 3 4 5 6 7 8 9 10 | One // runs Two // runs // Skipped step // doesn't run // // Three // doesn't run // Four // doesn't run // Useful if you don't want function calls to expand, // but won't remind you in the console/report that skipped steps exist |
// on step block member
1 2 3 4 5 | One // runs // Two // doesn't run, and doesn't run any steps below Three // runs Four // runs, except for "Two" |
Collapsing (+, +?)
Collapse (+)
1 2 3 4 5 6 | * Select best item from dropdown + // function calls here will be collapsed by default in the report Click 'dropdown' Scroll to 'best item' Click 'best item' Some big operation with lots of steps + // this function call will be collapsed by default in the report |
If there's an error inside a +'ed function, or if the function is currently running, it will be uncollapsed automatically.
It's recommended to mark precondition steps with +, since their details aren't central to the test.
Hidden (+?)
1 2 3 4 5 | * Init +? // function calls here will be hidden in the report Do some internal stuff // this function call will be hidden in the report Some internal thing somebody reading the report doesn't care about +? |
If there's an error inside a +?'ed function, it will be visible in the report.
Only ($)
One $
1 2 3 4 5 6 7 8 9 | Open Chrome Navigate to 'google.com' Do a search $ Navigate to 'pets.com' // only runs branches that pass through this step Buy cat food // i.e., the branch ending here |
Multiple $'s at different indents
1 2 3 4 5 6 7 8 9 10 | Open Chrome $ Open Firefox Open Safari Desktop $ Mobile Navigate to 'google.com' Do a search // only runs the Firefox mobile branch that ends here |
Multiple $'s with the same parent
1 2 3 4 5 6 7 8 9 10 | Open Chrome $ Open Firefox $ Open Safari Desktop Mobile Navigate to 'google.com' Do a search // only runs the Firefox and Safari branches (4 of them) |
On a function declaration
1 2 3 4 5 6 7 8 | $ * Do a search // only runs branches that call this function // etc. Log in as 'bob' Do a search Log in as 'vishal' Do a search |
With ~'s
1 2 3 4 5 6 7 8 | Open Chrome Open Firefox $ Open Safari Desktop Mobile $ Navigate to 'google.com' ~ // use to help isolate a branch for a debug |
Debug (~, ~~)
Debug modifier (~)
1 2 3 | Navigate to 'google.com' ~ Click 'button' // isolate this branch, run in REPL, and pause right before this step Click 'other thing' |
1 2 3 | Navigate to 'google.com' Click 'button' ~ // isolate this branch, run in REPL, and pause right after this step Click 'other thing' |
A branch run in debug will also pause after a failing step.
If a ~ goes through multiple branches, the first one is chosen.
No report is generated and the list of previously passed branches (for -s) is retained.
Express debug modifier (~~)
1 2 3 | Navigate to 'google.com' ~~ Click 'button' // only run this branch, but no pausing and don't run in REPL Click 'other thing' |
Just like for ~, no report is generated and the list of previously passed branches (for -s) is retained.
Debug flag
You can also run smashtest --debug=[hash] to debug the branch with the given hash.
Be careful. If you change any step, the hash of its branch will change. The recommended technique is to place a $, ~~, or ~ on a line you change, then run that. You'll want to rerun all the branches that pass through that line anyway.
Hooks (***)
What's a hook?
A hook is a code block function that runs before or after a step, branch, or test suite.
They're not meant for testing or setup/teardown. They're for internal stuff, such as reporting, importing code, js function declarations, screenshots, and logging.
Only code blocks are allowed, and they cannot have children or modifiers. Hooks aren't listed in the report. If a hook fails, the step/branch it corresponds to will take the error and fail.
Types
Before Every Branch
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | Parent step A B *** Before Every Branch { // this code runs before every branch that goes through the parent step // i.e., // 1) this code, parent step, A // 2) this code, parent step, B } *** Before Every Branch { // this code runs before every branch in existence } |
After Every Branch
1 2 3 4 5 6 7 8 9 | Parent step *** After Every Branch { // this code runs after every branch that goes through the parent step (whether it passes or fails) } *** After Every Branch { // this code runs after every branch in existence (whether it passes or fails) } |
Before Every Step
1 2 3 4 5 6 7 8 9 | Parent step *** Before Every Step { // this code runs before every step of every branch that goes through the parent step } *** Before Every Step { // this code runs before every step of every branch in existence } |
After Every Step
1 2 3 4 5 6 7 8 9 | Parent step *** After Every Step { // this code runs after every step of every branch that goes through the parent step } *** After Every Step { // this code runs after every step of every branch in existence } |
Before Everything
1 2 3 4 | *** Before Everything { // this code runs before the whole test suite begins // only valid at 0 indents } |
After Everything
1 2 3 4 | *** After Everything { // this code runs after the whole test suite ends // only valid at 0 indents } |
Where should setup and teardown logic go?
Setup code should go in the actual test.
Teardown code should go in the same setup logic, such that the previous state is cleaned out prior to actual testing. It should go into a hook only if necessary.
1 2 3 4 5 6 7 | - My test Setup Testing * Setup Clean previous state Set things up for new test |
UI Testing
Smashtest is a general platform for testing which comes with several built-in packages. One of these packages handles web UI testing with selenium webdriver.
This section discusses the steps and js functions that this package makes available.
Also, check out this UI test example.
Browsers and devices
Open a browser
1 2 3 4 5 6 7 | Open Chrome // run exclusively with --groups=chrome Open Firefox // run exclusively with --groups=firefox Open Safari // run exclusively with --groups=safari Open IE // run exclusively with --groups=ie Open Edge // run exclusively with --groups=edge Open browser 'chrome' // use a string recognized by webdriver as a browser name |
Set viewport size
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | // Run these exclusively with --groups=desktop Desktop // sets browser to 1920 x 1080 Laptop // sets browser to 1024 x 768 Laptop L // sets browser to 1440 x 900 // Run these exclusively with --groups=mobile Mobile // sets browser to 375 x 667 Mobile Portrait // sets browser to 375 x 667 Mobile Landscape // sets browser to 667 x 375 Mobile S // sets browser to 320 x 480 Mobile S Portrait // sets browser to 320 x 480 Mobile S Landscape // sets browser to 480 x 320 Mobile M // sets browser to 375 x 667 Mobile M Portrait // sets browser to 375 x 667 Mobile M Landscape // sets browser to 667 x 375 Mobile L // sets browser to 425 x 667 Mobile L Portrait // sets browser to 425 x 667 Mobile L Landscape // sets browser to 667 x 425 Tablet // sets browser to 768 x 1024 Tablet Portrait // sets browser to 768 x 1024 Tablet Landscape // sets browser to 1024 x 768 |
Set device type
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | // These steps set the viewport to the given device, and do device emulation in Chrome BlackBerry Z30 // sets browser to 360 x 640 Blackberry PlayBook // sets browser to 600 x 1024 Galaxy Note 3 // sets browser to 360 x 640 Galaxy Note II // sets browser to 360 x 640 Galaxy S III // sets browser to 360 x 640 Galaxy S5 // sets browser to 360 x 640 Kindle Fire HDX // sets browser to 800 x 1280 LG Optimus L70 // sets browser to 384 x 640 Laptop with HiDPI screen // sets browser to 1440 x 900 Laptop with MDPI screen // sets browser to 1280 x 800 Laptop with touch // sets browser to 1280 x 950 Microsoft Lumia 550 // sets browser to 640 x 360 Microsoft Lumia 950 // sets browser to 360 x 640 Nexus 4 // sets browser to 384 x 640 Nexus 5 // sets browser to 360 x 640 Nexus 5X // sets browser to 412 x 732 Nexus 6 // sets browser to 412 x 732 Nexus 6P // sets browser to 412 x 732 Nexus 7 // sets browser to 600 x 960 Nexus 10 // sets browser to 800 x 1280 Nokia Lumia 520 // sets browser to 320 x 533 Nokia N9 // sets browser to 480 x 854 Pixel 2 // sets browser to 411 x 731 Pixel 2 XL // sets browser to 411 x 823 iPhone 4 // sets browser to 320 x 480 iPhone 5/SE // sets browser to 320 x 568 iPhone 6/7/8 // sets browser to 375 x 667 iPhone 6/7/8 Plus // sets browser to 414 x 736 iPhone X // sets browser to 375 x 812 iPad // sets browser to 768 x 1024 iPad Mini // sets browser to 768 x 1024 iPad Pro // sets browser to 1024 x 1366 |
Usage examples
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | Open Chrome Open Firefox Open Safari Desktop - Desktop test 1 // etc. - Desktop test 2 // etc. Mobile - Mobile test 1 // etc. - Mobile test 2 // etc. iPhone X - iPhone X test |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | // Login function with different steps on desktop vs. mobile // Run with `smashtest` or `smashtest *.smash` // ----- main.smash ----- Open Chrome On Desktop On Mobile // etc. Login // the login that's called depends on desktop vs. mobile // ----- viewports.smash ----- * On Desktop Desktop // calls `Desktop` step exposed by `Open Chrome` * On Mobile Mobile // calls `Mobile` step exposed by `Open Chrome` // ----- desktop.smash ----- * On Desktop * Login - Desktop implementation of Login // ----- mobile.smash ----- * On Mobile * Login - Mobile implementation of Login |
Capabilities
Sometimes you need to set custom browser capabilities and options, such as when you need to provide a username and password for a cloud service (BrowserStack, Sauce Labs, etc.)
Set custom capabilities
1 2 3 4 5 6 7 8 9 10 11 | Open Chrome Set custom capabilities { g('browser capabilities', { 'name': 'foobar' // This is the Capabilities object. Capabilities go here. // See withCapabilities() }); } // Note: you can set this before or after the "Open [browser]" step |
Set custom options
1 2 3 4 5 6 7 8 9 10 11 12 13 | Open Chrome Set custom options { const chrome = i('selenium-webdriver/chrome'); let opts = new chrome.Options(); // Call functions on opts here g('browser options', opts); } // Note: you can set this before or after the "Open [browser]" step |
Browser steps
Interact
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | Click 'elementfinder' Native click 'elementfinder' // same as click, but uses js click instead of webdriver click // try this when webdriver click doesn't work Double click 'elementfinder' Hover over 'elementfinder' Scroll to 'elementfinder' Check 'elementfinder' // clicks the element, if it's currently unchecked Uncheck 'elementfinder' // clicks the element, if it's currently checked |
Set
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | Type 'text' into 'elementfinder' Type '[none]' into 'elementfinder' // step does nothing // good for including inaction when testing different inputs Clear 'elementfinder' // clear a textbox, etc. Set 'elementfinder' to 'value' Set 'elementfinder' to '[none]' // step does nothing // good for including inaction when testing different inputs Select '6' from 'elementfinder' // selects an <option> from a <select> // if an <option> with this exact value cannot be found, // searches for an <option> that contains the value, // trimmed and case-insensitive Select '[none]' from 'elementfinder' // step does nothing // good for including inaction when testing different inputs Select element 'option elementfinder' from 'dropdown elementfinder' // selects an <option> from a <select> {variable} = Value of 'elementfinder' |
Navigate
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | Navigate to 'https://site.com/page' Navigate to 'http://site.com/page' Navigate to 'site.com/page' // defaults to http Navigate to '/page' // uses domain browser is currently on Nav to '/page' // shorthand for Navigate Go Back Go Forward Refresh {current url} = Current url // returns current absolute url |
Window
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | Set dimensions to width='1024' height='768' Maximize window Open new tab // opens new tab ("window") and switches to it Switch to window whose title contains 'hello' Switch to window whose url contains '/page' Switch to the '1st' window Switch to the '4th' window Switch to iframe 'elementfinder' Switch to topmost iframe {window title} = Window title // returns current window title |
Alerts
1 2 3 | Accept alert // clicks ok in alert modal, error if no alert open Dismiss alert // clicks cancel in alert modal, error if no alert open |
Wait
1 2 3 4 | Wait '2' secs // sleeps this long Wait '2' seconds Wait '1' sec Wait '1' second |
Cookies and storage
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | {cookie} = Get cookie 'name' Verify cookie { cookie; // object containing cookie info cookie.value; // value of cookie } Set cookie 'name' to 'value' Set cookie 'name' to 'value', expiring in '65' secs Delete cookie 'name' Delete all cookies Clear local storage Clear cookies and local storage |
Print and log
1 2 3 4 5 | Log 'text to log to this step in report' 'elementfinder' // outputs found elements to browser's console "elementfinder" // and number of found elements to regular console [elementfinder] // useful when using REPL |
Verify steps
Verify
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | Verify 'elementfinder' is visible Verify 'elementfinder' is not visible Verify 'elementfinder' is 'state-elementfinder' Verify every 'elementfinder' is 'state-elementfinder' Verify at page 'Page title' // passes if current page title (case-insensitive) Verify at page 'site.com/page' // or url contains this text Verify at page '/page' Verify at page 'Page .*' // passes if current page title Verify at page '(.*)page' // or url matches this regex Verify at page '^http(.*)page.+$' Verify cookie 'name' contains 'value' Verify alert contains 'hello' // passes if alert is open and contains this text // Note: this step doesn't wait up to 2 secs - it's immediate |
Wait until
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | Wait until 'elementfinder' is visible Wait until 'elementfinder' is visible (up to '30' secs) Wait until 'elementfinder' is not visible Wait until 'elementfinder' is not visible (up to '30' secs) Wait until 'elementfinder' is 'state-elementfinder' Wait until 'elementfinder' is 'state-elementfinder' (up to '30' secs) Wait until every 'elementfinder' is 'state-elementfinder' Wait until every 'elementfinder' is 'state-elementfinder' (up to '30' secs) Wait until at page 'Page title' // passes if current page title (case-insensitive) Wait until at page 'site.com/page' // or url contains this text Wait until at page '/page' Wait until at page 'Page .*' // passes if current page title Wait until at page '(.*)page' // or url matches this regex Wait until at page '^http(.*)page.+$' Wait until at page 'Page title' (up to '30' secs) Wait until at page 'site.com/page' (up to '30' secs) Wait until at page '/page' (up to '30' secs) Wait until at page 'Page .*' (up to '30' secs) Wait until at page '(.*)page' (up to '30' secs) Wait until at page '^http(.*)page.+$' (up to '30' secs) Wait until cookie 'name' contains 'value' Wait until cookie 'name' contains 'value' (up to '30' secs) |
Assert
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | Verify {variable} equals 'value' Verify {variable} is 'value' Verify {variable} == 'value' Verify {variable} is greater than 'value' Verify {variable} > 'value' Verify {variable} is greater than or equal to 'value' Verify {variable} >= 'value' Verify {variable} is less than 'value' Verify {variable} < 'value' Verify {variable} is less than or equal to 'value' Verify {variable} <= 'value' |
Network conditions and throttling
Since Chrome is the only browser that supports emulating network conditions, this step only works in that browser. In all other browsers, this step quietly does nothing.
See conditional tests for ways of excluding other browsers, if it's cleaner that way.
1 2 3 4 5 | // Makes the browser emulate the given network conditions // latency is additional latency in ms // max download and upload speeds are in bytes/sec Set network conditions to offline='true' latency='200' max-download-speed='300000' max-upload-speed='400000' |
Mocking time and geolocation
Time
1 2 3 4 5 6 7 8 9 | // Makes the browser think the date and time is the one that's given (hijacks js Date) // Takes any string the js Date object can interpret Mock time to '4/1/2003' Mock time to '2011 Aug 19 4:45 pm' Mock time to '2020-09-02 19:19:45' // Where {date} contains a js Date object Mock time to {date} |
Geolocation
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | // Makes the browser think the user's current location is the one that's given Mock location to latitude='28.538336' longitude='-81.379234' // that's Orlando, FL, USA // Current pre-defined locations (case-insensitive) Mock location to 'Berlin' Mock location to 'London' Mock location to 'Moscow' Mock location to 'New York' Mock location to 'Mumbai' Mock location to 'San Francisco' Mock location to 'Seattle' Mock location to 'Shanghai' Mock location to 'São Paulo' Mock location to 'Sao Paulo' Mock location to 'Tokyo' |
Stop
1 2 | // Stops all time, geolocation, and http mocks and restores originals Stop all mocks |
Mocking APIs
String response
1 2 3 4 5 6 | Mock an endpoint { await mockHttp('GET', '/endpoint', 'canned response'); } // An XHR GET from the browser to /endpoint will always get a // 200 with 'canned response' body |
JSON response
1 2 3 4 5 6 | Mock an endpoint with a JSON response { await mockHttp('GET', '/endpoint', {key: 'val'}); } // An XHR GET from the browser to /endpoint will always get a // 200 with json body '{"key":"val"}' |
Detailed response
1 2 3 4 5 6 7 8 | Mock an endpoint and specify the response status code, http headers, and body { await mockHttp('GET', '/endpoint', [201, {'Content-Type': 'text/plain'}, 'canned response'] ); } // An XHR GET from the browser to /endpoint will always get a // 201 with the given http headers and 'canned response' body |
Function response
1 2 3 4 5 6 7 8 | Mock an endpoint with a function { await mockHttp('GET', '/endpoint', function(xhr) { xhr.respond(201, {'Content-Type': 'text/plain'}, 'canned response'); }); } // An XHR GET from the browser to /endpoint will always get a // 201 with the given http headers and 'canned response' body |
Regex endpoint
1 2 3 4 5 6 | Mock every endpoint that matches a regex { await mockHttp('GET', /\/end.*/, 'canned response'); } // An XHR GET from the browser to any matching endpoint will always get a // 200 with 'canned response' body |
Stop mocks
1 2 3 | Stop all http mocks and restore original endpoints { await mockHttpStop(); } |
Configure
1 2 3 4 | Configure the mock server { await mockHttpConfigure({autoRespond: false}); // See "Fake server options" at sinon's page for a list of configuration options } |
Check out sinon's fake xhr and server (the underlying library) for more details on what you can do.
ElementFinders
What's an ElementFinder?
An ElementFinder (or EF) is a string that matches elements on a page.
They contain one or more lines, where each line is a comma-separated list of props. A prop (or property) describes an element, or the state of an element.
1 2 3 4 5 | Click 'login button' // login button is an EF consisting of one prop, login button Click ['follow', next to 'bob'] // 'follow', next to 'bob' is an EF // 'follow' and next to 'bob' are both props // remember that [brackets] delimit strings, like "quotes" or 'quotes' |
Alternatively, a list of props may be separated by spaces, but only if each item is either 'text', an ord (see below), or a prop that's already been defined (multiple words are ok, but no 'inputs').
1 2 3 4 5 6 | Click ['Login' button] // 'Login' button is an EF, 'Login' and button are both props Click [4th 'Login' button] // 4th (the ord prop), must come first in space-separated EFs Click 'red button' // if red button is not already defined, it will try to // interpret this as a list of two props, red and button |
You should use space-separated EFs in steps (such as Click) because they sound natural and are easier to read. For example, Click [red 'login' button] sounds better than Click [button, red, 'login'].
The $() function is used to find an element given an EF. Since it throws an error if nothing is found in time, $() can also be used to verify the existence (and visibility) of an element.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | Verify login box { // Verifies the existence of at least one visible element // that matches the EF with prop login box await $(`login box`); } Get login box { // Sets elem to the first visible WebElement // that matches the EF with prop login box let elem = await $(`login box`); } Verify focused login box { // Verifies the existence of at least one visible element // that matches the EF with these 4 props await $(`login box, focused, 'text inside', .selector`); // Similar example using props separated by either spaces or commas // css selectors and props with input have to be separated with commas await $(`focused 'text inside' login box, .selector`); } // See code reference for details on $() |
Likewise, $$() finds multiple matching elements, given an EF.
To play around with EFs, run smashtest -r, open a browser, nav to a page, and type in various quoted 'EFs' (or, you can run a test in debug). The browser console (DevTools) will contain logs for every EF match.
Props
Although selectors can be valid props on their own, avoid steps like Click '#some-elem'. Instead, define a prop and use it in the step, e.g., Click 'login box'. Your tests will be easier to read and refactorable.
Setting
Props are defined with the props() function. More on that on the code reference page.
1 2 3 4 5 6 7 8 | On homepage { // Define props for all the elements on the homepage // Format: 'prop name': `EF` or function props({ 'login box': `.msgbox, enabled, 2nd`, 'about link': `selector "a[name='about']"` }); } |
Matching rules
Starting with all elements in the DOM, as props are applied (left to right), each one narrows down the list of elements. The elements that remain at the end are the ones that match the EF.
Whenever a prop is encountered, it is interpreted according to the first rule it matches in the following list:
-
'text'
- Matches elements where the given text is contained in its innerText, value, placeholder, associated label's innerText, or currently selected <option>'s innerText (for selects).
- Case insensitive, leading and trailing whitespace is ignored, and all whitespace is treated as a single space (both the 'text' and text in the DOM).
-
1st, 2nd, 3rd, etc. (ord)
- E.g., 4th = take the elements currently matched and choose only the 4th one.
- In a comma-separated prop list, you will usually list an ord last, since it narrows you down to just one element.
- In a space-separated prop list, the ord must come first (it's more natural-sounding that way), but will be applied last.
-
defined prop
- These are the definitions set by props()
-
They may take an input string, such as propname 'input string'
- You must use commas to separate these from other props
- There are several props that come pre-defined
-
css selector
- If nothing else matches, a prop is interpreted to be a css selector
- You must use commas to separate these from other props
If an EF fails to match an element, it will throw an error containing the full EF, prettified, with a --> explanation next to the lines that didn't match.
If you use a tagname selector and it actually matches the name of a defined prop, the defined prop will be used. To be safe, use selector 'tagname' as your prop.
The same is true for selectors containing commas (ORs). Always use selector '.selector1, .selector2' since commas separate props by default.
Implicit visible prop
By default, all EFs get an implicit visible prop (meaning, only match visible elements). The exception is when you explicitly use a visible, not visible, or any visibility prop. More on those here.
Not
Start a prop with "not" to find elements that don't match that prop. E.g., not fuzzy or not .selector,
Counters
Match multiple elements by preceding a line with a counter.
1 2 3 4 5 6 7 8 9 10 11 | await $$(`login box`); // match 1 or more login boxes await $$(`1 x login box`); // match exactly 1 login box await $$(`3 x login box`); // match exactly 3 login boxes await $$(`0+ x login box`); // match 0 or more login boxes await $$(`1+ x login box`); // match 1 or more login boxes await $$(`2+ x login box`); // match 2 or more login boxes await $$(`2- x login box`); // match 2 or more login boxes await $$(`2-5 x login box`); // match between 2 and 5 login boxes, inclusive |
Child elements
Simple example
1 2 3 4 5 6 7 8 9 10 11 12 | // Matches one or more .list elements that contain these 3 children, in that order: await $$(` .list .item1 .item2 .item3 `); // This is a subset matching, meaning that other elements // can exist in and around the 3 children inside .list // The top parent (.list) can start at any indentation that's a multiple of 4 |
Multi-level with counters
1 2 3 4 5 6 7 8 9 10 11 12 | await $$(` 1+ x .list // matches 1 or more .list elements that contain these children: 4 x .item // 4 .item's button[name=q] // 1 button with attribute name set to "q" // blank lines are ok .section // 1 .section that contains these children: #textbox // 1 #textbox enabled login box // 1 login box that's enabled button, contains 'click me' // 1 button that contains 'click me' `); // Note that // comments are allowed inside EFs |
Any order
1 2 3 4 5 6 7 | await $$(` #list any order // the 3 children can be in any order .item, "NYC" .item, "Tokyo" .item, "Paris" `); |
Matching children
1 2 3 4 5 6 7 8 9 10 11 | // A [] around a line will match and return those elements, // as opposed to the top parent // This EF will match 4 elements: // a 'Tokyo' item and the 3 items that follow await $$(` #list .item, 'NYC' [.item, 'Tokyo'] [3 x .item] `); |
To match the selector [attr="something"], use the prop selector '[attr="something"]'. Otherwise, the []'s will be interpreted as a matching operator.
Implicit body
1 2 3 4 5 6 7 8 9 10 11 12 | await $$(` #one // multiple lines at the top indent #two `); // implicitly equates to await $$(` body #one #two `); |
Element array
Do stricter validations with element arrays:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | // An element array is a line that starts with a * // It will match as many elements as it can, and verify that // all children listed underneath map 1-to-1 with each element matched. // The ordering and number must be the same. If not, an error occurs. await $$(` #list * .item // this is an element array .item, 'NYC' .item, 'Tokyo' 2 x .item `); // There must be exactly 4 items inside of #list: // a 'NYC', a 'Tokyo', and 2 more items of any kind // in that exact order. Otherwise you get an error. // Note: You can include the `any order` keyword, as shown before, // to allow any ordering of the children |
Default ElementFinder props
Smashtest defines these props by default. Remember, you can put not in front of any of them.
-
visible =
matches if an element is visible to the user
(width and height > 0, no hidden styles, opacity > 0, opacity of all ancestors > 0)
- This prop is implicitly applied to all EFs, unless 'any visibility', 'visible', or 'not visible' are already in that EF
-
any visibility =
matches elements regardless of visibility, disables implicit 'visible' prop
- If you define a new prop as 'new prop': `#some-invisible-elem, any visibility`, you must use that new prop as Click 'new prop, any visibility'
- Likewise, if you define another prop as 'even newer prop': `new prop, any visibility`, you must use that prop as Click 'even newer prop, any visibility', applying 'any visibility' all the way up the chain
- enabled = matches if 'disabled' attribute isn't present
- disabled = matches if 'disabled' attribute is present
- checked = matches if element.checked is true
- unchecked = matches if element.checked is false
- selected = matches if element.selected is true (used for selects)
- focused = matches if element currently has focus
- element = matches any element
- clickable = matches if element is of a clickable type (a, button, label, input, textarea, select, option) or has cursor:pointer style
- page title 'title' = causes error if the page title isn't equal to the given string
- page title contains 'title' = causes error if the page title doesn't contain the given string (case-insensitive)
- page url 'url' = causes error if the page url isn't equal to the given string (relative or absolute)
- page url contains 'url' = causes error if the page url doesn't contain the given string
-
next to 'text' =
matches the one element that's closest in the DOM to the given text
- Takes each elem and expands the container around it to its parent, parent's parent etc. until a container containing the text is found - matches that one element associated with that container (matches multiple elems if there's a tie)
- value 'text' = matches if element.value is equal to the given text
-
contains 'text' =
matches if the given text is contained in an element's innerText, value,
placeholder, associated label's innerText, or currently selected <option>'s innerText
(for selects).
- Case insensitive, leading and trailing whitespace is ignored, and all other whitespace is treated as a single space (in both the 'text' and the DOM).
- 'text' = same as contains 'text'
- contains exact 'text' = matches if an element's innerText, value, placeholder, associated label's innerText, or currently selected <option>'s innerText (for selects) equals the given text exactly
- innertext 'text' = matches if an element's innerText contains the given text
- selector 'selector' = matches if an element matches the given css selector
- xpath 'xpath' = matches if an element matches the given xpath
- style 'name:value' = matches if an element has the style with the given name set to the given value
-
position 'N' =
matches the one element from the pool of currently matched elements that's in position N
(where the first index is 1)
- Same as ords (1st, 2nd, etc.)
- textbox = matches if an element is a textbox or textarea
Screenshots
When enabled, screenshots are taken before and after every step in branches that include an Open [browser] step. They're available in the report by clicking a step.
While good for debugging failures, screenshots can take up a lot of disk space and slow down test execution. Limit screenshots with these flags:
-
--screenshots=[true/false]
- set to true to enable screenshots
-
--max-screenshots=[N]
- doesn't take any more screenshots after N are taken
- screenshots that are taken and later deleted due to --step-data don't count against N
-
--step-data=[all/fail/none]
- set to 'fail' to only retain screenshots (and other step data) for failed branches
- set to 'none' to not retain any screenshots (or other step data)
Code reference
Async js function calls should always be preceded with await. If errors are thrown but aren't captured inside a step, or if things just seem wonky, it's probably because you forgot an await.
Finding elements
$()
1 2 3 4 5 6 7 8 9 | // Finds the first matching WebElement let elem = await $('elementfinder'); // waits up to 2 secs, otherwise throws error // Can be used to verify existence of an ElementFinder (by throwing error if not found) await $(` #list .item .item `); |
1 2 3 4 5 6 7 8 9 10 11 12 | /** * Finds the first matching element. Waits up to timeout ms. * Scrolls to the matching element, if found. * @param {String or WebElement} element - An EF representing the EF to use. If set to a WebElement, returns that WebElement. * @param {Boolean} [tryClickable] - If true, first try searching among clickable elements only. If no elements are found, searches among non-clickable elements. * @param {WebElement} [parentElem] - If set, only searches at or within this parent element * @param {Number} [timeout] - How many ms to wait before giving up (2000 ms if omitted) * @param {Boolean} [isContinue] - If true, and if an error is thrown, that error's continue will be set to true * @return {Promise} Promise that resolves to first WebDriver WebElement that was found * @throws {Error} If a matching element wasn't found in time, or if an element array wasn't properly matched in time */ await $(element, tryClickable, parentElem, timeout, isContinue); |
$$()
1 2 | // Finds all the matching WebElements let elems = await $$('elementfinder'); // waits up to 2 secs, otherwise throws error |
1 2 3 4 5 6 7 8 9 10 11 | /** * Finds the matching elements. Waits up to timeout ms. * See $() for param details * If element is an EF and a counter isn't set on the top element, sets it to 1+ * @return {Promise} Promise that resolves to Array of WebDriver WebElements that were found * @throws {Error} If matching elements weren't found in time, or if an element array wasn't properly matched in time */ await $$(element, parentElem, timeout, isContinue); // Note: If you want this function to return an empty array and // not throw an error, pass in an EF with counter `0+ x element` |
not$()
1 2 | // Ensures no matching elements exist await not$('elementfinder'); // waits up to 2 secs, otherwise throws error |
1 2 3 4 5 6 7 | /** * Throws an error if the given element(s) don't disappear before the timeout * See $() for param details * @return {Promise} Promise that resolves if the given element(s) disappear before the timeout * @throws {Error} If matching elements still found after timeout */ await not$(element, parentElem, timeout, isContinue); |
str()
1 2 3 4 5 6 7 | // str() escapes strings for use in ElementFinders: let s = "string\'\n"; await $(` #list .item .item, contains '${str(s)}' `); |
ElementFinder props
props()
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | // Define props with ElementFinders or functions // If a prop listed already exists, it is overridden // Prop definitions exist for all future steps in the branch, until overridden props({ // ElementFinder definitions: 'message box': `.msgbox, enabled`, 'search results': ` #list .result, 'one' .result, 'two' `, 'groovy': `'contains this groovy text'`, 'retro button': `selector '.button', groovy`, // Function definitions: 'fuzzy': function(elems, input) { // This function will be injected into the browser // elems is an array of DOM Elements // input is the input string (e.g., fuzzy 'input here'), undefined if not set // return an array of Elements from elem that match this prop return elems; } }); |
propsAdd()
1 2 3 4 | // Same as props(), but adds to a definition if it already exists propsAdd({ 'message box': `.msgbox` }); |
1 2 3 4 5 6 7 8 9 10 11 12 13 | Define a message box { props({ 'message box': `.msgbox` }); } Add to the definition of a message box { propsAdd({ 'message box': `#msgbox` }); } // Now, an element will be a 'message box' if it matches .msgbox OR #msgbox |
propsClear()
1 2 | // Clears the definitions of the props listed propsClear(['message box', 'groovy']); |
Executing JS in browser
executeScript()
1 2 3 4 | await executeScript(function() { // executes js in the browser // see webdriverjs's executeScript() }); |
1 2 3 4 5 6 7 8 9 | let arg1 = 'one'; let arg2 = 2; let v = await executeScript(function(arg1, arg2) { // arg1 and arg2 are accessible here return arg1 + arg2; }, arg1, arg2); // v is "one2" |
executeAsyncScript()
1 2 3 4 5 6 | await executeAsyncScript(function(done) { // executes js in the browser // must call done() callback at the end // see webdriverjs's executeAsyncScript() done(); }); |
1 2 3 4 5 6 7 8 9 | let arg1 = 'one'; let arg2 = 2; let v = await executeAsyncScript(function(arg1, arg2, done) { // arg1 and arg2 are accessible here done(arg1 + arg2); }, arg1, arg2); // v is "one2" |
browser
browser
1 2 3 4 5 6 7 8 9 10 11 | browser; // BrowserInstance object that represents the open browser // Browser details browser.params.name; browser.params.version; browser.params.platform; browser.params.width; browser.params.height; browser.params.deviceEmulation; browser.params.isHeadless; browser.params.testServer; |
browser.driver
1 | browser.driver; // webdriverjs WebDriver object that represents the open browser's driver |
Mocking
mockTime()
1 2 | let date = new Date(); await mockTime(date); // set the browser's date to this one |
1 2 3 4 5 6 | /** * Mock's the current page's Date object to simulate the given time. Time will run forward normally. * See sinon's fake timers for more details * @param {Date} time - The time to set the browser to */ await mockTime(time); |
mockLocation()
1 | await mockLocation(28.538336, -81.379234); // sets the browser's location to the given latitude and longitude |
mockHttp()
1 2 3 | // Responds with 200 'canned response' when an XHR GET in the browser tries to hit /endpoint on the current domain // See mocking APIs for more details await mockHttp('GET', '/endpoint', 'canned response'); |
1 2 3 4 5 6 7 8 9 10 11 12 13 | /** * Mocks the current page's XHR. Sends back the given response for any http requests to the given method/url from the current page. * You can use multiple calls to this function to set up multiple routes. If a request doesn't match a route, it will get a 404 response. * See sinon's fake xhr and server for more details * @param {String} method - The HTTP method ('GET', 'POST', etc.) * @param {String or RegExp} url - A url or a regex that matches urls * @param response - A String representing the response body, or * An Object representing the response body (it will be converted to JSON), or * an array in the form [ [status code], { header1: "value1", etc. }, [response body string or object] ], or * a function * See server.respondWith() from sinon's documentation */ await mockHttp(method, url, response); |
mockHttpConfigure()
1 | await mockHttpConfigure({autoRespond: false}); |
1 2 3 4 5 6 7 | /** * Sets configs on the currently mocked XHR * @param {Object} config - The options to set (key value pairs) * See fake server options for details on what config options are available * Fails silently if no mock is currently active */ await mockHttpConfigure(config); |
mockTimeStop()
1 | await mockTimeStop(); // stops time mocks and restores original time |
mockLocationStop()
1 | await mockLocationStop(); // stops geolocation mocks and restores original geolocation |
mockHttpStop()
1 | await mockHttpStop(); // stops http mocks and restores original endpoints |
mockStop()
1 | await mockStop(); // stops all time, geolocation, and http mocks, restores originals |
injectSinon()
1 2 3 4 | // sinon becomes accessible in browser via global var 'sinon' // automatically called when one of the mocking functions above invoked // does nothing if sinon is already available inside browser |
API Testing
One of Smashtest's built-in packages handles HTTP API testing. This section discusses the js functions that this package makes available.
Here are two examples of API testing:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | Book room for '0' nights Book room for '1' nights Book room for '2' nights Verify success Book room for '100' nights Verify error * Book room for {{n}} nights { await api.get(`https://api.com/book/${n}`); } * Verify success { response.verify({ statusCode: 200 }); } * Verify error { response.verify({ statusCode: { $min: 400, $max: 499 } }); } |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | Make a request { await api.post({ url: `https://${host}/endpoint`, headers: { 'content-type': 'application/json' }, body: { id: 123, user: 'jerry' }, timeout: 1500 }); } Verify response { response.verify({ statusCode: 200, headers: { 'content-type': 'text/html; charset=utf-8' }, error: null, body: { id: 123, user: 'jerry', results: [ '$anyOrder', // loose-matching of a JSON response body { cost: { $min: 10 }, items: 6, name: { $contains: 'apples' } }, { cost: { $min: 15 }, items: 7, name: { $typeof: 'string', $contains: 'berries' } } ] } }); } |
Also, check out this API test example.
Request
request()
1 | await request('https://api.com/endpoint'); // GET by default |
1 2 3 4 5 6 7 | await request({ method: 'GET', // GET, POST, PUT, PATCH, DELETE, HEAD, or OPTIONS url: 'https://api.com/endpoint', timeout: 1500 // setting timeout (in ms) is recommended // otherwise the default OS TCP timeout applies, // which may be longer than the 60 sec step timeout }); |
get()
1 | await get('https://api.com/endpoint'); |
1 2 3 4 5 6 7 | await get({ url: 'https://api.com/endpoint', headers: { 'content-type': 'text/plain' }, timeout: 1500 }); |
post()
1 2 3 4 5 6 7 8 | await post({ url: 'https://api.com/endpoint', headers: { 'content-type': 'text/plain' }, body: `body goes here`, timeout: 1500 }); |
1 2 3 4 5 6 7 8 9 | // JSON body await post({ url: 'https://api.com/endpoint', body: { something: true }, json: true, // converts body to json and sets content-type header timeout: 1500 }); |
put()
1 2 3 4 5 6 7 8 | await put({ url: 'https://api.com/endpoint', headers: { 'content-type': 'text/plain' }, body: `body goes here`, timeout: 1500 }); |
patch()
1 2 3 4 5 6 7 8 | await patch({ url: 'https://api.com/endpoint', headers: { 'content-type': 'text/plain' }, body: `body goes here`, timeout: 1500 }); |
del()
1 2 3 4 5 6 7 8 | await del({ url: 'https://api.com/endpoint', headers: { 'content-type': 'text/plain' }, body: `body goes here`, timeout: 1500 }); |
head()
1 | await head('https://api.com/endpoint'); |
options()
1 | await options('https://api.com/endpoint'); |
api.defaults()
1 |
Cookies
1 2 3 4 5 6 7 8 9 10 | // create cookies let jar = api.jar(); let cookie1 = api.cookie('key1=value1'); let cookie2 = api.cookie('key2=value2'); let url = 'http://site.com'; jar.setCookie(cookie1, url); jar.setCookie(cookie2, url); // make a request that includes the cookies await get({url: 'http://site.com/endpoint', jar: jar}); |
Response and verify
Simple example
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | Make a request { await get('https://site.com/endpoint'); } Verify the response { // The global variable 'response' is automatically filled with the last response // response.verify() checks that the actual response object matches // the expected response object that's passed in response.verify({ // you can list zero or more of these expected keys: statusCode: 200, // expected status code headers: { // expected headers 'content-type': 'application/json' }, error: null, // expected error object from request library body: { // expected body (js obj if body is json, string otherwise) one: 'two' }, rawBody: '{"one":"two"}', // expected raw response body responseObj: {} // expected response object from request library }); // To access the response details: response.statusCode; // status code response.headers; // js obj where keys are header names response.error; // error obj from request library response.body; // js obj if body is json, string otherwise response.rawBody; // string containing unparsed body data response.responseObj; // response obj from request library } |
A good pattern for developing API tests is to make a request step followed by a response step that just does c(response); (similar to console.log(response))
Then, manually verify the response in the console and copy it into a response.verify(); in the response step.
Matching in response.verify()
Basic rules
- A value in the expected object must == the corresponding value in the actual object, or an error will occur.
-
Expected arrays are exact match by default.
- E.g., expected [ A, B, C ] means actual must be [ A, B, C ] exactly.
-
Expected objects are subset matching by default.
- E.g., expected { one: 1 } will match { one: 1, two: 2 } but not { one: 3 } or { two: 2 }.
- If matching fails, an error is thrown containing the entire actual object, prettified, with --> explanation next to each line that didn't match.
Special matching
Replace values in the expected object with { $key: value } for special loose matching, as described below:
-
{ $typeof: "type" }
- Makes sure the corresponding value in the actual object is of this type (uses js's typeof).
- You can use "array" to match arrays.
-
{ $regex: /regex/ } or
{ $regex: "regex" }
- Makes sure the corresponding value in the actual object is a string that matches this regex.
-
{ $contains: "string" }
- Makes sure the corresponding value in the actual object is a string that contains this string.
-
{ $max: N }
- Makes sure the corresponding value in the actual object is a number that isn't greater than N.
-
{ $min: N }
- Makes sure the corresponding value in the actual object is a number that isn't less than N.
-
{ $code: (actual) => { return actual == 'something'; } }
{ $code: "return actual == 'something'"}
{ $code: "actual == 'something'"}- Makes sure the corresponding value in the actual object causes this code to evaluate to true or return true.
-
{ $length: N }
- Makes sure the corresponding value in the actual object is an array, string, or object (with a length property set) whose length is N.
-
{ $maxLength: N }
- Makes sure the corresponding value in the actual object is an array, string, or object (with a length property set) whose length isn't greater than N.
-
{ $minLength: N }
- Makes sure the corresponding value in the actual object is an array, string, or object (with a length property set) whose length isn't less than N.
-
{ $exact: true, a: A, b: B }
- Makes sure the corresponding value in the actual object is an object that matches exactly (every expected key exists and has the expected value, and no other keys exist).
- E.g., { $exact: true, one: 1 } will match { one: 1 } but not { one: 1, two: 2 }, { one: 3 }, or { two: 2 }.
-
{ $every: A }
- Makes sure the corresponding value in the actual object is an array where every item matches A.
- E.g., { $every: 'Q' } would match [ 'Q', 'Q', 'Q' ].
- E.g., { $every, { $contains: "foo" } } would match [ "foobar", "barfoo", "foo" ].
-
[ "$subset", A, B, C ]
- Makes sure the corresponding value in the actual object is an array that contains A, B, and C in any order (and it could have more items).
-
[ "$anyOrder", A, B, C ]
- Makes sure the corresponding value in the actual object is an array that contains A, B, and C (and nothing else) in any order.
- You can use $subset and $anyOrder together such as [ "$subset", "$anyOrder", A, B, C ] to match an array containing A, B, and C (and potentially more items) in any order.
-
You can have multiple criteria in the same object.
- E.g., { $typeof: "string", $length: 10, $regex: /[A-Z]+/ }
- E.g., [ "$subset", "$anyOrder", A, B, C ]
- When the expected value is an object, you can include both regular keys and $-keys in the { expected object }.
-
Undefineds
- { one: undefined, two: 2 } will match { one: undefined, two: 2 } or { two: 2 }
- To validate that one: undefined is actually there, use one: { $typeof: 'undefined' }
- To validate that one: undefined is actually not there, use { $exact: true, two: 2 } with no "one" key
Comparer
Verify JS objects in general by using the built-in Comparer object. For example:
1 2 | // Same functionality as response.verify(expectedObj); Comparer.expect(actualObj).to.match(expectedObj); |
1 2 3 4 5 6 7 8 9 10 11 | /** * Compares the actual object against the expected object * @param {Object} actualObj - The object to check. Must not have circular references or multiple references to the same object inside. Could be an array. * @param {Object} expectedObj - The object specifying criteria for actualObj to match * @param {String} [errorStart] - String to mark the start of an error, '-->' with ANSI color codes if omitted * @param {String} [errorEnd] - String to mark the end of an error, '' with ANSI color codes if omitted * @param {String} [errorHeader] - String to put at the top of the entire error message, '' if omitted * @param {Boolean} [jsonClone] - If true, compares using the rough clone method, aka JSON.stringify + JSON.parse (which handles multiple references to the same object inside actualObj, but also removes functions and undefineds, and converts them to null in arrays) * @throws {Error} If actualObj doesn't match expectedObj */ Comparer.expect(actualObj, errorStart, errorEnd, errorHeader, jsonClone).to.match(expectedObj); |
REPL
What's a REPL?
REPL stands for "read–eval–print loop". It's a way of driving Smashtest from the command-line by entering text commands.
Try it yourself. Run smashtest -r or smashtest --repl, or debug a file with the ~ modifier.
Command list
-
Entering a step to run it
- One-line steps ok
- Multiple lines ok if step has code block (first line is Step name { and last line is })
- Just type in { to start an anonymous step with a code block
- No function declarations, hooks, or step blocks allowed
- When browser is open, a convenient step is [<ElementFinder here>]. It will print the elements found to the browser's console and the number of elements found to the console you're typing into.
- Ctrl + C = exit
- Enter key = run next step or exit (if no more steps ahead)
- s = skip next step
- p = repeat previous step
- r = resume running
- x = exit
- .break = sometimes you get stuck, this gets you out
- .clear = break, and also clear the local context
- .editor = enter editor mode
- .exit = exit
- .help = print a help message
- .load = load js from a file into this REPL session
- .save = save all evaluated commands in this REPL session to a file
Packages
Make a package
Share your steps and functions by distributing a package:
- Implement functionality inside a js file.
- Distribute that file as an npm package via the npm registry.
-
Distribute simple function declarations that users can download directly from you or
copy into their .smash files:
123* My awesome function {i('my_awesome_npm_package').myAwesomeFunction(); // i() is similar to require()}
Use a package
To install a package:
- npm install the package globally (via npm -g install), in the same directory as the .smash files that will use it, or in any directory above that.
- Copy the above function declaration into one of your files, or download a .smash file with that declaration already inside it. Make sure it runs with the other .smash files when you run your tests.
-
Now you can use the step anywhere in your tests:
1My awesome function
Promote
We'll help promote your package! Simply contact us.
Contact Us
Say hello! We're very friendly :)