Let's Explore

Exploratory Testing

@MichaKutz

I would consider it a red flag if a team isn't doing exploratory testing at all - even if their automated testing was excellent.

Even the best automated testing is inherently scripted testing - and that alone is not good enough.

November 18th 2019

Martin Fowler

@martinfowler

Exploratory Testing

that the software meets expectations

for additional risks

tested = checked + explored

Simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next.

interesting things to vary and interesting ways in which to vary them

as soon as you think of a test

look past what you expect

to push even harder

Elisabeth Hendrickson
@testobsessed

Can a user actually pass the checkout process?

What happens when I change my basket in a different tab, right before finishing checkout?

What happens when I use app and web in parallel? Are baskets synced?
Is there a reasonable error page displayed, when the system is on overload?
What happens when I click the basket button really fast?
What happens when I change the delivery time slot in a different tab, right before finishing checkout?

Think about questions for your product…

Automation

Exploration

Load

Exploration

Exploration

Exploration

Design

Explore <target>


with <resources>


to discover <information>

Where are you exploring?

a feature

a requirement

a module

What resources will you bring with you?
a tool
a data set
a technique
a configuration
an interdependent feature

What kind of information are you hoping to find?

security

performance

reliability

capability

usability

consistency of design

violations of a standard

Charter Template

from "Explore It!"
by Elisabeth Hendrickson

My mission is to test <risk>


in the context of <coverage>.

What am I testing?

a feature

a requirement

a module

What could go wrong?

functionality is wrong

usability is bad

not accessible

inconsistency

Alternative Template

from "Tips for Writing Better Charters for Exploratory Testing Sessions"
by Michael D. Kelly

<timeframe>

How long is this supposed to take?

often implicit

Explore the behavior of the basket button


with various interaction types and speeds


to discover unintended side effects

Explore every input field in the shop


with every security tool you can find


to discover security issues

Explore the address form


with the name "Søren Anderson"


to discover if scandinavial letters are handled correctly

Too narrow

actually a test case

Too broad

you will never be finished

Explore the registration form


with common XSS injection strings


to discover XSS attack vulnerabilities

My mission is to test SQL vulnerabilities

 

in the context of the search form.

Find ways that a valid order modification might fail.

Experiment with invalid values when updating customer addresses.

Good Charters,
Bad Charters

Bad Charters Feel Like…

from "Tips for Writing Better Charters for Exploratory Testing Sessions"
by Michael D. Kelly

What was I supposed to focus on?!

I need more time!

Boooooooooring!

Where can I find the data needed for this?!

Didn't somebody test this already?!

Is this really important?!

We are going to offer same day delivery.

What's the latest time to order for same day?

1pm.

What happens when 1pm passes while the user is already in checkout past the time slot selection?

Can we throw the user back to the selection?

Shouldn't be any problem.

Explore checkout past time slot selection around same day cutoff


with a same day time slot

 

to discover problems with throw back mechanism.

Requirement Discussions

I have to think about that…

Users can reserve time slots for up to 30 minutes.

Should we prevent reservations past 12:30?

Explore reservations


with same day time slots past 12:30

 

to discover if reservations allow users to place same day orders past 1pm.

Requirement Discussions

Continued

Security Model

Performance Requirements

Reliability

Usability

Accessability

Scalability

Implicit Expectations

// I don't know why this works…
// …but it does
// DO NOT TOUCH!
if (deliveryAddress != null) {
    …

Other sources for charters…

What happens when I change my basket in a different tab, right before finishing checkout?

What happens when I use app and web in parallel? Are baskets synced?
What happens when I click the basket button really fast?
What happens when I change the delivery time slot in a different tab, right before finishing checkout?

Think about charters for your product…

Explore the checkout process


with background changes of the basket

 

to discover consistency problems.

Explore the basket


with background changes of the basket via app

 

to discover consistency problems.

Explore delivery fee calculation after time slot selection


with changing the time slot in background

 

to discover surprises.

Explore the behavior of the basket button


with various click speeds

 

to discover surprises.

Execute

Did the installer run without errors?

Can I start the software?

Ask deeper questions

Was the add-to-basket success message displayed?

Did the product actually appear in the basket overview?

Is the logout button displayed?

Can I edit my profile?

Look past what you expect or hope to find

unexpected activity

JavaScript errors

unexpected system load

unexpected filesystem or database activity

$> watch ls
$> psql

subtle UI changes

unexpected data transfer

Forms

Browser type

Cookie values

URL & parameters

Input fields

Browser plugins

Variables

  • Number of products shop
  • Number of search results
  • Number of users logged in
  • Uptime of the system

Countable Things

→ 0, 1, many

→ too many

→ too few

Positions

→ beginning, middle, end

Files & Storage

→ make inaccessible

→ use different then default location

Geographic Locations

→ far away, close by

→ different time zone

Formats

→ dates, decimals, binary, IP addresses

→ file formats

Size

→ huge, small

Timing, Frequency & Duration

→ at midnight

→ fast, slow

Input & Navigation

→ mouse, keyboard

→ type, paste

→ touch

Variable Types

Think about variables in your product…

  • Format: try to add "a"
  • Input: try pasting, typing, clicking
  • Position: type/paste at beginning, middle, end
  • Countable: 0, 1, (too) many
  • Speed: click fast
  • Input: click, press enter, press space
  • Timing: add to basket and wait for September 13th

Ensemble

Evaluate Results

Who will use this software and for what purpose?

Why would someone choose this software over competitors?

Always & Never

Can you give me an elevator pitch for the software?

If nothing else is working, what has to function?

What is your greatest fear regarding the software?

What are our most important quality factors?

Accuracy? Reliability? Availability? Usability? Accessibility? Security?

\left. \begin{array}{l} \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \end{array} \right\}

Core Capabilities

\left\{ \begin{array}{l} \\ \\ \\ \\ \\ \end{array} \right.

"Non-Functional"
Requirements

\left\{ \begin{array}{l} \\ \\ \\ \end{array} \right.

Risks

Alternative Verification

Internal Consistency

Standards

Comparables

Plausibility

Reverse Test

\sqrt{a^2}=a

Checklists

e.g. https://www.ministryoftesting.com/dojo/lessons/checklist-for-testing-web-page-functionality

Think about Alwayses and Nevers of your product…

Always bill!

Never bill twice!

Never display wrong ingredients for a product!

Always provide at least one deliver time slot within the next week!

Never display wrong data at order confirmation page!

Always allow to cancel the checkout!

Never display different prices for the same product!

Always deliver valid HTML according to W3C standards!

Always reader page withing 1 second!

Always have a reasonable alt attribute for all images!

Learn

Design

Execute

Learn

My mission is to explore the checkout process


for surprises.

My mission is to explore the checkout process


for side effects caused by browser navigation.

My mission is to explore the checkout process


for side effects caused by parallel activity in a different tab.

actual time for testing

confidence factor
on coverage of the stated risk

found bugs

possible new charters

open questions

things to be automated

things to be documented

setup steps

My mission is to explore the checkout process


for side effects caused by browser navigation.

My mission is to explore the checkout process


for side effects caused by parallel activity in a different tab.

What did you learn about your product?

Continuous Exploration

Sprint

Planning

Review/

Retro

Coding

1st story done 🎉

2nd story done 🎉

last story done 🎉

Daylies

Backlog Refinement

"Explore It!"
by Elisabeth Hendrickson
https://pragprog.com/book/ehxta/explore-it

"Tips for Writing Better Charters for Exploratory Testing Sessions"
by Michael D. Kelly

https://www.slideshare.net/EuroSTARConference/mike-kelly-euro-star-webinar

https://www.youtube.com/watch?v=dOQuzQNvaCU

"A Heuristic Approach to Test Charters"
by Adam Howard

https://solavirtusinvicta.wordpress.com/2014/12/18/a-heuristic-approach-to-test-charters/

Sources & Acknowledgments

Lisi Hocke

@lisihocke

For her great feedback & encouragement on this talk

Alex Schladebeck

@alex_schl

For a lot of high value shared knowledge & feedback

https://youtu.be/9FKY1Is0lgs

Adding Dimensions

Nouns & Verbs

Mix randomly

Randomize your navigation

mouse only, short cuts only, search only, menu only…

Personas

how would Chuck Norris use this product?

Let's Explore Exploratory Testing

By Michael Kutz

Let's Explore Exploratory Testing

An hands-on exploratory testing workshop for development teams. Based on "Explore It!" by Elisabeth Hendrickson

  • 4,292