You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Given/When/Then? How Long? How Many? What is Required? Inputs/Outputs/Trends?
Decisions and Artifacts
Automated metric grabber (per hour, per day); Side effects
Example
title: WHAT - “How might we increase the number of orders?”
STEP 1: Change Request
title: WHY: "We need business to grow."
`````ad-info
title: WHAT: "Increase the number of sales per week."
````ad-question
title: HOW:
color: 200, 200, 200
```ad-info
title: Assumptions: Results of the brainstorm
- [x] A1: Web traffic can be converted to sales
- [ ] A2: Customer reviews can trigger sales
- [ ] A3: Discount with an expiration date can trigger sales
- [ ] A4: Payback with an expiration date can trigger sales
- [ ] A5: Instagram train/giveaway can trigger sales
- [ ] A6: ...continue...
```
```ad-note
title: Voting Results: selected A1 as the primary action
```
````
`````
STEP 2: Confirmation
title: Confirmation
``````ad-faq
title: METRICS:
- [ ] M1: number of page views/impressions
- [ ] M2: number of orders
- [ ] M3: $$$ order amount
- [ ] M4: number of drops/exits/cancelations
- [ ] M5: $$$ cost of the customer
- [ ] M6: conversion rate (%)
``````
``````ad-info
title: Side effects/artifacts:
- [ ] E1: Why was the order canceled? (examples: Price too high, not proper quality, ...)
- [ ] E2: Where does the user exit from the "making order" sequence? (examples: too many options in order, on payment step, on personal details, ...)
- [ ] E3: Session duration
- [ ] E4: Bounce Rate
- [ ] E5: New vs. Returning visitor
- [ ] E6: Average pages/steps per session
- [ ] E7: Cost of Execution
``````
`````ad-faq
title: Success/Failure criteria
- [ ] Fail. Customer cost is too high (>15$ per customer)
- [ ] Success. Conversion rate >10%
- [ ] Success. Customer cost is low (<5$ per customer)
- [ ] Success. Too Many orders (stop the experiment if needed, reduce the size)
`````
`````ad-question
title: Should we execute it?
- [ ] is the experiment atomic?
- [ ] are risks of failure identified (roadblocks)? What can we change to reduce risks?
- [ ] is implementation time short?
- [ ] can we score the test's priority/value/impact?
`````
STEP 3: Execution
title: Execution
``````ad-important
title: Required for execution: select one option only
- [ ] Organic traffic
- [ ] Purchased traffic (ads networks)
- [ ] Purchased traffic (3rd party, freelance)
- [ ] Purchased traffic (social networks ads)
``````
``````ad-info
title: Duration: 1 week (or till we reach 10'000 impressions)
``````
``````ad-info
title: Size: 10'000 impressions
``````
``````ad-cite
title: Test
`````ad-info
title: GIVEN
- [ ] website is ready to accept traffic in such an amount
- [ ] business is ready to be shortly overloaded (too many orders)
- [ ] website proper functioning in "order sequence"
- [ ] metrics collected by analytics from all pages (M1-M...)
- [ ] can we divide the experiment into two equal groups? (One group **without** experiment (clean group), another with experiment (test group))
`````
`````ad-info
title: WHEN
- [ ] clean run (no other experiments executed)
- [ ] metrics extracted every hour and saved to the database
- [ ] tested something that can trigger sales (texts, graphics, layouts, colors, etc.)
`````
`````ad-info
title: THEN
- [ ] collected metrics are visualized with calculated trends
- [ ] assumption on trend defined for detecting anomaly behavior
- [ ] comparison to the group clean group results visualized
`````
``````
STEP4: Decisions and Artifacts
title: Decisions (Stop|Adjust|Scale up)
`````ad-question
title: Is Success? How can we apply this in scale?
- [ ] convert experiment to a business rule
- [ ] identify failure criteria that should switch off this business rule
`````
`````ad-question
title: Is Failure? What is the reason of failure?
color: 255, 0, 0
- [ ] clarify that no outside forces are involved in test (for example: infrastructure failure)
`````
`````ad-question
title: Results are unclear? What influenced the experiment? Can we eliminate the side risks and repeat the experiment?
color: 255, 251, 45
- [ ] test is not atomic
- [ ] test does not produce a statistically significant difference
- [ ] test results are under the influence of outside force
`````
`````ad-info
title: Artifacts
color: 48, 96, 255
- [ ] Metrics collection tech should become a part of MAIN code
- [ ] Any findings should be documented and become a part of the backlog
`````