Gleb Bahmutov PRO
JavaScript ninja, image processing expert, software quality fanatic
Fast Testing Using Cypress For Free 💸💸💸
Gleb Bahmutov
📺 video at https://youtu.be/1idlr9IE0oU
Global temperature anomaly 2022 https://climate.nasa.gov/vital-signs/global-temperature/
survival is possible* but we need to act now
- change your life
- join an organization
Speaker: Gleb Bahmutov PhD
C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing
Gleb Bahmutov
Sr Director of Engineering
glebbahmutov.com/blog testing posts
Lots Of Testing
A typical Mercari US Cypress E2E test
Plus internal web application E2E tests
Now up to 800 tests!!!
Running Lots of E2E Tests: Problems
- Slow
- Flake
- Expensive
Cypress Cloud
Cypress Dashboard Cloud
cypress run --record
Cypress Dashboard Cloud
cypress run --record --parallel
every it(...) executed = test result
10 tests x 1 per day = 300 test results
10 tests x 2 per day = 600 test results 🛑
This Presentation
-
How to run Cypress E2E specs in parallel
-
How to save test artifacts
-
How to debug failed Cypress tests
for free 🎉
for free 🎉
for free 🎉
Example repo: https://github.com/bahmutov/fast-free
// spec-a.cy.js
it('works A', () => {
cy.wait(10_000)
})
// spec-b.cy.js
it('works B', () => {
cy.wait(60_000)
})
// spec-c.cy.js
it('works C', () => {
cy.wait(10_000)
})3 specs: 10 seconds, 1 minute, 10 seconds
$ npx cypress run
Let's Run Tests On CI!
name: ci
on: push
jobs:
e2e:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
# Install npm dependencies, cache them
# and run all Cypress tests
- uses: cypress-io/github-action@v6.github/workflows/ci.yml
Workflow run UI
Cypress GH Action details
Caching is important
Waiting for 1 CI machine...
spec-b
spec-a
spec-c
Can't we use 2 CI machines?
Split Specs Manually
name: ci
on: push
jobs:
e2e-1:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
with:
spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
e2e-2:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
with:
spec: cypress/e2e/spec-c.cy.jsSplit Specs Manually
2 parallel containers
cypress-split plugin
name: ci
on: push
jobs:
e2e-1:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
with:
spec: cypress/e2e/spec-a.cy.js,cypress/e2e/spec-b.cy.js
e2e-2:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
with:
spec: cypress/e2e/spec-c.cy.jsautomatically compute the spec lists for each machine
const { defineConfig } = require('cypress')
// https://github.com/bahmutov/cypress-split
const cypressSplit = require('cypress-split')
module.exports = defineConfig({
e2e: {
setupNodeEvents(on, config) {
cypressSplit(on, config)
// IMPORTANT: return the config object
return config
},
},
})npm i -D cypress-split
cypress.config.js
name: ci
on: push
jobs:
e2e-1:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT: 2
SPLIT_INDEX: 0
e2e-2:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT: 2
SPLIT_INDEX: 1GitHub Actions Workflow
cypress-split algorithm
-
find all Cypress specs
-
split into N chunks
-
take chunk k
The First CI Machine
The first machine's summary
The second machine's summary
The GHA Job Matrix
name: ci
on: push
jobs:
e2e:
runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
containers: [1, 2]
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT: ${{ strategy.job-total }}
SPLIT_INDEX: ${{ strategy.job-index }}
Matrix Jobs UI
The specs are split the same way
Want To Go Faster?
name: ci
on: push
jobs:
e2e:
runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
containers: [1, 2]
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT: ${{ strategy.job-total }}
SPLIT_INDEX: ${{ strategy.job-index }}Split specs across 2 machines
name: ci
on: push
jobs:
e2e:
runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
containers: [1, 2, 3, 4, 5]
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT: ${{ strategy.job-total }}
SPLIT_INDEX: ${{ strategy.job-index }}Split specs across 5 machines
name: ci
on: push
jobs:
e2e:
runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
containers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT: ${{ strategy.job-total }}
SPLIT_INDEX: ${{ strategy.job-index }}Split specs across 10 machines
About that split...
spec-b
spec-a
spec-c
Is this the best we can do?
💻 1
💻 2
About that split...
spec-b
spec-a
spec-c
💻 1
💻 2
We know the spec-b takes a lot longer
Idea: use spec durations
$ SPLIT_FILE=timings.json SPLIT=1 SPLIT_INDEX=0 npx cypress run
run this command locally
{
"durations": [
{
"spec": "cypress/e2e/spec-a.cy.js",
"duration": 10073
},
{
"spec": "cypress/e2e/spec-b.cy.js",
"duration": 60090
},
{
"spec": "cypress/e2e/spec-c.cy.js",
"duration": 10075
}
]
}commit timings.json to the source code repo
name: ci
on: push
jobs:
e2e:
runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
containers: [1, 2]
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
env:
SPLIT_FILE: timings.json
SPLIT: ${{ strategy.job-total }}
SPLIT_INDEX: ${{ strategy.job-index }}.github/workflows/ci.yml
cypress-split timings algorithm
-
find all Cypress specs
-
look up timings
-
sort specs by duration from longest to shortest
-
fill N machine chunks
-
put each spec into the chunk with smallest sum of durations
-
💻 1
💻 2
spec-a
spec-b
spec-c
spec-b
spec-a
spec-c
spec-b
spec-a
spec-c
the first machine
the second machine
Update timings.json nightly
name: nightly
on:
# run this workflow every night at 3am
schedule:
- cron: '0 3 * * *'
# or when the user triggers it from GitHub Actions page
workflow_dispatch:
jobs:
e2e:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cypress-io/github-action@v6
with:
# we don't need Cypress own runner summary
publish-summary: false
env:
SPLIT_FILE: timings.json
SPLIT: 1
SPLIT_INDEX: 0
- name: Commit changed spec timings ⏱️
if: github.ref == 'refs/heads/main'
# https://github.com/stefanzweifel/git-auto-commit-action
uses: stefanzweifel/git-auto-commit-action@v4
with:
commit_message: Updated spec timings
branch: main
file_pattern: timings.json.github/workflows/nightly.yml
Trigger the workflow manually
GitHub Actions integration with GitHub = 💪💪💪
Make It Simple
name: split
on:
workflow_dispatch:
jobs:
split:
uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
with:
nE2E: 2.github/workflows/split.yml
name: split
on:
workflow_dispatch:
jobs:
split:
uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
with:
nE2E: 2
split-file: 'timings.json'
name: split
on:
workflow_dispatch:
jobs:
split:
uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
with:
nE2E: 2
split-file: 'timings.json'
commit-updated-timings:
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-22.04
needs: split
steps:
- uses: actions/checkout@v4
# pretty-print json string into a file
- run: echo '${{ toJson(fromJson(needs.split.outputs.merged-timings)) }}' > timings.json
- name: Commit changed spec timings ⏱️
uses: stefanzweifel/git-auto-commit-action@v4
with:
commit_message: Updated spec timings
branch: main
file_pattern: timings.jsonCommit the changed timings
Reporting
Slow tests
Slow debugging
Fast tests
Slow debugging
Slow tests
Fast debugging
Fast tests
Fast debugging
🥇
Example repo: https://github.com/bahmutov/test-artifacts
beforeEach(() => {
// clear the backend data
cy.request('POST', '/reset', { todos: [] })
})
it('clears the completes todos', () => {
cy.visit('/?addTodoDelay=1000')
cy.get('.loaded')
cy.get('.new-todo').type('first todo{enter}')
cy.get('li.todo').should('have.length', 1)
cy.get('.new-todo').type('second todo{enter}')
cy.get('li.todo').should('have.length', 2)
cy.get('.new-todo').type('third todo{enter}')
cy.get('li.todo').should('have.length', 3)
cy.log('**complete the third todo**')
cy.get('li.todo').eq(2).find('.toggle').click()
cy.get('li.todo').eq(2).should('have.class', 'completed')
cy.get('li.todo').eq(0).should('not.have.class', 'completed')
cy.get('li.todo').eq(1).should('not.have.class', 'completed')
cy.log('**clear completed items**')
cy.get('[data-cy="filter-completed"]').click()
cy.location('hash').should('equal', '#/completed')
cy.get('li.todo').should('have.length', 1)
cy.contains('button', 'Clear completed').click()
cy.log('**see all todos**')
cy.get('[data-cy="filter-all"]').click()
cy.location('hash').should('equal', '#/all')
cy.get('li.todo').should('have.length', 2)
})- add 3 todos
- mark 3rd todo completed
- clear completed todos
Example repo: https://github.com/bahmutov/test-artifacts
npx cypress open
E2E Test Artifacts
-
Console logs 📝
-
Screenshots on failures 🖼️
-
Browser videos 📼
-
Test commands logs 🖨️
-
Code coverage 🗺️
-
Traces 👀
Server logs...
name: split
on:
workflow_dispatch:
jobs:
split:
uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
with:
nE2E: 1
start: npm start
store-artifacts: true.github/workflows/split.yml
zip for each CI machine
📺 cypress/videos/spec.cy.js.mp4
A test error on purpose
failure screenshot
Better Logs
// cypress.config.js
const logOptions = {
printLogsToConsole: 'always',
}
require('cypress-terminal-report/src/installLogsPrinter')(on, logOptions)
Every Cypress command with its arguments
App console.log calls
Failed assertions
Awesome Logs Reports
// cypress.config.js
setupNodeEvents(on, config) {
require('cypress-mochawesome-reporter/plugin')(on)
require('cypress-terminal-report/src/installLogsPrinter')(on)
},
// spec or support file
require('cypress-mochawesome-reporter/register')
afterEach(() => {
cy.wait(50, { log: false }).then(() => {
cy.addTestContext(Cypress.TerminalReport.getLogs('txt'))
})
})
require('cypress-terminal-report/src/installLogsCollector')()Save HTML report
E2E Test Artifacts
-
✅ Console logs 📝
-
✅ Screenshots on failures 🖼️
-
✅ Browser videos 📼
-
✅ Test commands logs 🖨️
-
Code coverage 🗺️
-
Traces 👀
Code Coverage
"Full Code Coverage For Free"
name: ci
on: [push]
jobs:
tests:
uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
with:
nE2E: 4 # use 4 containers for E2E tests
nComponent: 2 # use 2 containers for component tests
start: npm start
# merge E2E and component code coverage into a single report
coverage: true
Merge coverage workflow job
Merged coverage summary
Merged coverage reports as test artifacts
Traces
same install as terminal reporter, then:
Except...
Cypress-the-company now disables plugins that compete with its cloud
https://cypresstips.substack.com/p/cypress-tips-november-2023
☢️☢️☢️☢️☢️☢️☢️
Summary
- Run specs in parallel on CI 🤖🤖🤖
- Save artifacts 💾
- screenshots, videos
- awesome reports
- traces perhaps
Resources
https://github.com/bahmutov/cypress-split
https://glebbahmutov.com/blog/cypress-parallel-free-based-on-timings/
https://github.com/bahmutov/cypress-workflows
https://github.com/archfz/cypress-terminal-report
https://github.com/LironEr/cypress-mochawesome-reporter
🎉 Fin 🎉
Gleb Bahmutov
Thank you 🙏
📺 video at https://youtu.be/1idlr9IE0oU
Fast Testing Using Cypress For Free
By Gleb Bahmutov
Fast Testing Using Cypress For Free
In this presentation, Gleb will explain how to speed up Cypress test execution locally and on CI using free open-source solutions that do not require paying for 3rd party dashboards or services. We will explore using API calls, data caching, and app actions to speed up each test. We also will see how to configure CI specs to split the entire test suite into multiple machines running in parallel. This presentation will benefit anyone who wants their end-to-end and component Cypress tests to finish quickly. Video at https://youtu.be/1idlr9IE0oU
