Given/When/Then? How Long? How Many? What is Required? Inputs/Outputs/Trends?
Decisions and Artifacts
Automated metric grabber (per hour, per day); Side effects
Example
title: WHAT - “How might we increase the number of orders?”
STEP 1: Change Request
title: WHY: "We need business to grow."
`````ad-info
title: WHAT: "Increase the number of sales per week."
````ad-question
title: HOW:
color: 200, 200, 200
```ad-info
title: Assumptions: Results of the brainstorm
- [x] A1: Web traffic can be converted to sales
- [ ] A2: Customer reviews can trigger sales
- [ ] A3: Discount with an expiration date can trigger sales
- [ ] A4: Payback with an expiration date can trigger sales
- [ ] A5: Instagram train/giveaway can trigger sales
- [ ] A6: ...continue...
```
```ad-note
title: Voting Results: selected A1 as the primary action
```
````
`````
STEP 2: Confirmation
title: Confirmation
``````ad-faq
title: METRICS:
- [ ] M1: number of page views/impressions
- [ ] M2: number of orders
- [ ] M3: $$$ order amount
- [ ] M4: number of drops/exits/cancelations
- [ ] M5: $$$ cost of the customer
- [ ] M6: conversion rate (%)
``````
``````ad-info
title: Side effects/artifacts:
- [ ] E1: Why was the order canceled? (examples: Price too high, not proper quality, ...)
- [ ] E2: Where does the user exit from the "making order" sequence? (examples: too many options in order, on payment step, on personal details, ...)
- [ ] E3: Session duration
- [ ] E4: Bounce Rate
- [ ] E5: New vs. Returning visitor
- [ ] E6: Average pages/steps per session
- [ ] E7: Cost of Execution
``````
`````ad-faq
title: Success/Failure criteria
- [ ] Fail. Customer cost is too high (>15$ per customer)
- [ ] Success. Conversion rate >10%
- [ ] Success. Customer cost is low (<5$ per customer)
- [ ] Success. Too Many orders (stop the experiment if needed, reduce the size)
`````
`````ad-question
title: Should we execute it?
- [ ] is the experiment atomic?
- [ ] are risks of failure identified (roadblocks)? What can we change to reduce risks?
- [ ] is implementation time short?
- [ ] can we score the test's priority/value/impact?
`````
STEP 3: Execution
title: Execution
``````ad-important
title: Required for execution: select one option only
- [ ] Organic traffic
- [ ] Purchased traffic (ads networks)
- [ ] Purchased traffic (3rd party, freelance)
- [ ] Purchased traffic (social networks ads)
``````
``````ad-info
title: Duration: 1 week (or till we reach 10'000 impressions)
``````
``````ad-info
title: Size: 10'000 impressions
``````
``````ad-cite
title: Test
`````ad-info
title: GIVEN
- [ ] website is ready to accept traffic in such an amount
- [ ] business is ready to be shortly overloaded (too many orders)
- [ ] website proper functioning in "order sequence"
- [ ] metrics collected by analytics from all pages (M1-M...)
- [ ] can we divide the experiment into two equal groups? (One group **without** experiment (clean group), another with experiment (test group))
`````
`````ad-info
title: WHEN
- [ ] clean run (no other experiments executed)
- [ ] metrics extracted every hour and saved to the database
- [ ] tested something that can trigger sales (texts, graphics, layouts, colors, etc.)
`````
`````ad-info
title: THEN
- [ ] collected metrics are visualized with calculated trends
- [ ] assumption on trend defined for detecting anomaly behavior
- [ ] comparison to the group clean group results visualized
`````
``````
STEP4: Decisions and Artifacts
title: Decisions (Stop|Adjust|Scale up)
`````ad-question
title: Is Success? How can we apply this in scale?
- [ ] convert experiment to a business rule
- [ ] identify failure criteria that should switch off this business rule
`````
`````ad-question
title: Is Failure? What is the reason of failure?
color: 255, 0, 0
- [ ] clarify that no outside forces are involved in test (for example: infrastructure failure)
`````
`````ad-question
title: Results are unclear? What influenced the experiment? Can we eliminate the side risks and repeat the experiment?
color: 255, 251, 45
- [ ] test is not atomic
- [ ] test does not produce a statistically significant difference
- [ ] test results are under the influence of outside force
`````
`````ad-info
title: Artifacts
color: 48, 96, 255
- [ ] Metrics collection tech should become a part of MAIN code
- [ ] Any findings should be documented and become a part of the backlog
`````
