mosaico-ab-planning.md

• Plan A1:

  • Synopsis: Update the Mosaico UI/extension to include A/B testing. Update other layers as necessary (in accord with general vision).
  • User Experience: To send an A/B test in Mosaico, create a regular Mosaico mailing -- and add multiple subjects.
  • Data Model: Each A/B test is stored as one record in civicrm_mailing.
  • Major Tasks:
    • (civicrm-core) Update CiviMail data-model to allow:
      • Storing optional variants of a mailing, as defined by overrides in the civicrm_mailing (e.g. variants.a.subject, variants.b.subject, variants.a.body_text)
      • Storing optional AB sizing (i.e. deliver 100 copies of variant A)
      • Storing optional variant ID for each delivered message
    • (civicrm-core) Update CiviMail API to support these new fields
    • (flexmailer) Update FlexMailer's ComposeBatch pipeline to:
      • Assimilate overrides during composition
      • For applicable mailings, assign a variant ID to each recipient
    • (mosaico) Update Mosaico Extension UI to:
      • Accept/display multiple subject lines, storing each as an override
      • Accept/display AB delivery params (number of recipients per test)
    • (civicrm-core) Update CiviMail reports to:
      • Allow comparison of stats for each variant
  • Strengths:
    • Addresses existing debt about experimental correctness (delivery ordering/timing)
    • Lined-up well to support conditional blocks (per-variant) in Mosaico mailings
  • Weaknesses / Debts / Potential Stretch Topics:
    • For a user, the navigation-path is confusing/ambiguous -- you have to understand more subtlties among "New Mailing (Mosaico)", "New Mailing (Traditional)", and "New A/B Test"
    • If someone tries to send a mailing with variants with BAO Delivery Engine (aka v1 delivery system), then that's not supported. Report some kind of error/warnings.

• Plan A2: (As above, but with data-mapping/decomposition on submit)

  • Synopsis: Update the Mosaico UI/extension to include A/B testing. Map one mailing to autogenerated AB record.
  • User Experience: To send an A/B test in Mosaico, create a regular Mosaico mailing -- and add multiple subjects.
  • Data Model: Mosaico UI initially generates one civicrm_mailing (with AB options under template_options); however, after submission, it is materialized into a few distinct mailings.
  • Major Tasks:
    • (mosaico) Update Mosaico Extension UI to:
      • Accept/display multiple subject lines, storing each in template_options
      • Accept/display AB delivery params (number of recipients per test), strong in template_options
    • (mosaico) Hook into Mailing.submit. Examine template_options and conditionally generate AB and related mailings. Maybe some API or hook updates in civicrm-core to facilitate this.
  • Strengths:
    • Less need to change core reports and data model
    • Less ambiguity/conflict in the navigation than Plan A1. (Still some, but less.)
  • Weaknesses / Debts / Potential Stretch Topics:
    • Still using less-correct experimental protocol.
    • Strange-loop in the model (mailing=>AB=>3x mailing) might look odd to others
    • Does not address existing debt about experimental correctness (delivery ordering/timing)

• Plan B:

  • Synopsis: Update the core CiviMail UI ("New Mailing (Traditional)" and "New A/B Test") to make ckeditor and Mosaico more interchangeable
  • User Experience: To send an A/B test in Mosaico, create an A/B test by "Subject" -- and choose to compose in Mosaico (rather than CKeditor).
  • Data Model: Each A/B test is stored as one record in civicrm_mailing_abtest with multiple relations to civicrm_mailing.
  • Major Tasks:
    • (civicrm-core) Expand or replace the API for richtext email (ckeditor/tinymce) to allow loading/storing Mosaico templates (extra metadata)
    • (civicrm-core) Define new UX/API for browsing/selecting templates (in which ckeditor-style temlates and Mosaico-style templates are interchangeable)
    • (civicrm-core) Implement that new UX
    • (mosaico) Integrate with the expanded/replaced API for richtext email
    • (mosaico) Deprecate the standalone composition screen
  • Needful But Not Fully Resolved Requirements:
    • Potentially sets up conflict over pageflow/layout in the composition screen -- existing users have conflicting expectations in "New Mailing (Mosaico)" and "New Mailing (Traditional)".
  • Strengths:
    • Addresses existing debt in navigation-path (confusion/ambiguity about "New Mailing (Mosaico)" and "New Mailing (Traditional)")
    • Reduces future maintenance in keeping "New Mailing (Mosaico)" and "New Mailing (Traditional)" in sync
  • Weaknesses / Debts / Potential Stretch Topics:
    • Lined up poorly for conditional (per-variant) blocks in Mosaico.
    • Still using less-correct experimental protocol.
    • Likely to change existing user-experiences in significant ways, so it seems more important to engage other stakeholders. This, in turn, can make it more expensive to settle the design and/or get through review.

• Other Streth Topics:

  • FlexMailer Phase-in
    • (Development) Cleanup/expose option to deliver messages for traditional mailings
    • (Discussion) Bundle with core
    • (Admin) Bundle with core
    • (CI) Run Flexmailer tests with core PRs