We are conducting an experiment to refactor 10 Workshops. We'll update those and then measure for 3 months. The goal is to find ways to increase the module completion rate.
The information below explains my approach. This is simply one direction an evaluation can take, but feel free to take inspiration or an entirely different approach in the experiment.
if we reduce or remove these "leave moments", completion rate will increase.
People are leaving at 3 key moments:
- they leave after they succeed at publishing to Azure, especially if the next step is not clear on why it should interest them
- they leave when encountering a knowledge check
- they leave after completing the knowledge check and before the summary
Notice the drop-off rates in red for the units over a five month period.
| Topic | Category | Before | After |
|---|---|---|---|
| Intro | Instructional Design | scenario existed, but no mention of what you wold do without Azure Static Web Apps | updated intro to show the problem, what we did to solve it, what you would need to do without Azure Static Web Apps, what you get with Azure Static Web Apps |
| Summary | Instructional Design | summary did not recap the problem well | updated the summary to recap the problem, what we did and how SWA helps you |
| 2 Fallback units | Product | The product requires we add fallback routes to prevent web frameworks from getting 404s on browser refreshes of client side routes | removed 2 units on Fallback routes, and instead baked them into the starter code. I also moved a few inches of explanation of the fallback routes in the conceptual unit 3, as it is necessary to work right |
| Time | Instructional Design | 22 minutes | 18 minutes |
| Units | Instructional Design | 8 units | 5 units |
| knowledge check | Instructional Design | 3 question knowledge check | Removed the knowledge check as per the data above on drop-off rates, as an experiment. Still passing gold standard 10 our of 10 as we have an exercise and Sandbox |
| Interactivity | Instructional Design | we removed the interactive knowledge check | we now have neither of the requirements: knowledge check or a task validation. I'd like to run an experiment to see if this hurts or helps completion rate. We can add one of these in later if needed. |
- starter code will be updated when we release this, as it needs to include the fallback route. This can be found in this PR on the sample code
Data fetched using this link https://mslearnmetricportal.azurewebsites.net/
And by entering the URL for the Learn module in GitHub https://github.com/MicrosoftDocs/learn-pr/blob/master/learn-pr/azure/publish-app-service-static-web-app-api/
| Standard | Description | Before | After |
|---|---|---|---|
| Introduction Unit | The first unit of a module is title "Introduction". | Yes | Yes |
| Summary Unit | The last unit of a module is title "Summary". | Yes | Yes |
| Prerequisites | The module contains at least 2 prerequisites. | Yes | Yes |
| Learning Objectives | The module contains 2 - 5 learning objectives. | Yes | Yes |
| Unit Count | The module contains 4 - 9 units. | Yes | Yes |
| Module Time | The module time is between 15 and 60 minutes. | Yes | Yes |
| Alternating Units | The units in the module alternate between conceptual and exercise. | Yes | Yes |
| Exercise Validation or Knowledge Check | The module contains at least 1 exercise validation or 1 knowledge check. | Yes | Yes |
| Image Count | The module contains at least 3 images. | Yes | Yes |
| Header Count | Each unit contains at least 2 header tags. | Yes | Yes |
- cuts the time to about 18 minutes
- largest drop offs are on the units I'm getting rid of (30 to 40% drop off rate on those compared with 5 to 10 on the others)
- completion rate will increase.