Skip to content

Instantly share code, notes, and snippets.

@johnpapa
Created February 15, 2022 03:12
Show Gist options
  • Save johnpapa/f134a1a9dd1298a428f418f7f554b93a to your computer and use it in GitHub Desktop.
Save johnpapa/f134a1a9dd1298a428f418f7f554b93a to your computer and use it in GitHub Desktop.
Thoughtful-Hypothesis

Context

We are conducting an experiment to refactor 10 Workshops. We'll update those and then measure for 3 months. The goal is to find ways to increase the module completion rate.

The information below explains my approach. This is simply one direction an evaluation can take, but feel free to take inspiration or an entirely different approach in the experiment.

Hypothesis

if we reduce or remove these "leave moments", completion rate will increase.

Data

People are leaving at 3 key moments:

  1. they leave after they succeed at publishing to Azure, especially if the next step is not clear on why it should interest them
  2. they leave when encountering a knowledge check
  3. they leave after completing the knowledge check and before the summary

Notice the drop-off rates in red for the units over a five month period.

image

Changes

Topic Category Before After
Intro Instructional Design scenario existed, but no mention of what you wold do without Azure Static Web Apps updated intro to show the  problem, what we did to solve it, what you would need to do without Azure Static Web Apps, what you get with Azure Static Web Apps
Summary Instructional Design summary did not recap the problem well updated the summary to recap the problem, what we did and how SWA helps you
2 Fallback units Product The product requires we add fallback routes to prevent web frameworks from getting 404s on browser refreshes of client side routes removed 2 units on Fallback routes, and instead baked them into the starter code. I also moved a few inches of explanation of the fallback routes in the conceptual unit 3, as it is necessary to work right
Time Instructional Design 22 minutes 18 minutes
Units Instructional Design 8 units 5 units
knowledge check Instructional Design 3 question knowledge check Removed the knowledge check as per the data above on drop-off rates, as an experiment. Still passing gold standard 10 our of 10 as we have an exercise and Sandbox
Interactivity Instructional Design we removed the interactive knowledge check we now have neither of the requirements: knowledge check or a task validation. I'd like to run an experiment to see if this hurts or helps completion rate. We can add one of these in later if needed.

Open items

  1. starter code will be updated when we release this, as it needs to include the fallback route. This can be found in this PR on the sample code

Gold Standard checklist

Data fetched using this link https://mslearnmetricportal.azurewebsites.net/

And by entering the URL for the Learn module in GitHub https://github.com/MicrosoftDocs/learn-pr/blob/master/learn-pr/azure/publish-app-service-static-web-app-api/

Standard Description Before After
Introduction Unit The first unit of a module is title "Introduction". Yes Yes
Summary Unit The last unit of a module is title "Summary". Yes Yes
Prerequisites The module contains at least 2 prerequisites. Yes Yes
Learning Objectives The module contains 2 - 5 learning objectives. Yes Yes
Unit Count The module contains 4 - 9 units. Yes Yes
Module Time The module time is between 15 and 60 minutes. Yes Yes
Alternating Units The units in the module alternate between conceptual and exercise. Yes Yes
Exercise Validation or Knowledge Check The module contains at least 1 exercise validation or 1 knowledge check. Yes Yes
Image Count The module contains at least 3 images. Yes Yes
Header Count Each unit contains at least 2 header tags. Yes Yes

The value

  1. cuts the time to about 18 minutes
  2. largest drop offs are on the units I'm getting rid of (30 to 40% drop off rate on those compared with 5 to 10 on the others)
  3. completion rate will increase.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment