newsletter-spikes.md

Wagtail newsletter | Research and spikes

Repository for spikes code: https://github.com/mgax/newsletter-spike-mailchimp

Mailchimp API

Verified that these steps work:

• Use the mailchimp-marketing Python library to send a ping to the API.

• Create a campaign; send a test email.

• Send the campaign.

• Set a custom HTML body for the campaign.

• If the body does not contain the required footer fields (an unsubscribe link using the *|UNSUB|* interpolation; the physical mailing address using the *|HTML:LIST_ADDRESS_HTML|* interpolation; and, for free accounts, a Mailchimp referral link), Mailchimp appends a standard footer.
  TODO Are there other links required by Mailchimp besides unsubscribe?
  TODO configure a rich text footer via Wagtail site settings? Add validation that it does, in fact, contain an unsubscribe link? Will draftail complain about a URL with the value of *|UNSUB|*? The integrator will add the footer as part of the template, and they can render it dynamically as needed.

• Links in custom HTML are wrapped for click tracking by Mailchimp when it sends out emails.

• API returns available options for audience segmentation. One segment can be set for a campaign.

• Fetch campaign performance metrics.
  TODO where do we present this information, if at all?

• Set custom to_name, subject_line.
  TODO do we make these configurable site-wide or per-email?

• The from_name and reply_to campaign fields are mandatory.
  TODO configure via Wagtail site settings?

Things that don’t work:

• Audience exclude rules (Do not send to …): although it’s present in the Mailchimp web app, the selection is not reflected in the API, and neither is the resulting change in number of recipients. It very much feels like Do not send to operates on a different level.
  As an alternative, it’s possible to express simple exclusion rules through the API, in POST /campaigns, recipients > segment_opts > conditions.
  TODO this option is part of the current Figma design. Do we really need it? FPF does use the exclusion feature, but it’s possible to reconfigure segments to account for that.

• Filtering the audience by interests/groups: not possible either through the Mailchimp web app or the API.
  TODO I suspect the FPF Mailchimp account already has audience segments configured to reflect interests/groups. We could then use those segments for audience filtering. Confirmed, there are segments configured.

Things I wasn’t able to test:

• The API supports scheduling a campaign for later (including batch delivery and timewarp). A paid account (or 30-day trial) is required.
  TODO get a paid/trial account to try out campaign scheduling.

• A/B tests and Multivariate tests. A paid/trial account is required.
  TODO get a paid/trial account to try out A/B and Multivariate tests?
  TODO can we support A/B and Multivariate testing by creating a Mailchimp campaign and not sending it, thus giving the editor an opportunity to configure the test, and then send it?

MJML

I’ve tested with the mrml library (official Python package for  MRML, a Rust port of MJML, which is small, fast and memory-thrifty). The documentation warns that it’s missing the mj-style[inline] feature which I don’t think we’ll need.

• Rendering of a sample MJML source works; Mailchimp accepts the HTML and sends it out; and the email looks right on quick inspection.

• It’s possible to include links like <a href="*|UNSUB|*"> in the MJML source, which will get interpolated by Mailchimp when it sends out the emails.

• MJML does not support plain-text output, but Mailchimp can generate a plain-text version of the content, to include in the email. This issue suggests using html2text.

• MJML can incorporate snippets of HTML, so Wagtail Rich Text mostly works.

  • Wagtail rich text generates link and image URLs as path-only URLs by default.
    TODO find a way to either generate fully-qualified URLs, or convert them after rich text rendering. done.

  • The MJML library fails if it spots an <iframe> tag, so embeds don’t work out of the box.
    TODO do we need embeds? If so, we could e.g. include them as thumbnails with links. Need to figure out how to tell the rich text rendering machinery to do this.I think we should use mrml by default, and make it easy for people to swap it out for another MJML implementation, or something else entirely.

Wagtail admin

• Can preview MJML rendered output in the Wagtail admin page editor.

• Prototype implementations of ChooserViewSet for audience and segment.
  TODO clear the segment field if the audience is changed or cleared.
  TODO setting edit_handler on Page causes JS error in admin editor: Comments app failed to initialise. Missing HTML element.

• Enhancing the page history report to show “Newsletter sent!” next to a revision: this would require overriding the template for the whole page, or injecting the label using JavaScript, none of which seem great options.
  TODO change the design.

Newsletter sending APIs

Comparison of newsletter sending APIs.

The SendGrid analysis is based on their Legacy Marketing Campaigns API, because the New Marketing Campaigns API has no documented methods to create campaigns, only “Single Send”.

	Mailchimp	Mailjet	SendGrid
Sender parameters	from_name, reply_to	sender, sender_email	sender_id
Audience	audience (single choice)	contact_list (multiple choice)	list (multiple choice)
Segmentation	belongs to an audience (single choice)	independent filters (multiple choice)	independent or belonging to an audience (multiple choice)
Update draft campaign metadata	✅ yes	✅ yes	❌ no
Update draft campaign body	✅ yes	✅ yes	✅ yes
Generate text from html	✅ yes	❌ no	❌ no
Send test email	✅ yes	✅ yes	✅ yes
Click tracking	✅ yes	✅ yes	✅ yes
Get performance metrics	✅ yes	✅ yes	❌ no

Django settings

• NEWSLETTER_CAMPAIGN_BACKEND: Dotted import path of the class that implements the campaign backend (e.g. wagtail_newsletter.campaigns.mailchimp.MailchimpCampaignBackend).

• NEWSLETTER_MAILCHIMP_API_KEY: Mailchimp API key.

• NEWSLETTER_MAILCHIMP_FROM_NAME: Email sender name.

• NEWSLETTER_MAILCHIMP_REPLY_TO: Email sender address.

• NEWSLETTER_MAILCHIMP_CACHE_TIMEOUT: How long should audience and segment data be cached for the page editor.

• NEWSLETTER_RENDERING_BACKEND: Dotted import path of the class that implements the rendering backend (e.g. wagtail_newsletter.rendering.mrml.MrmlBackend).

Campaign backend class

The library will provide a Mailchimp backend class. The integrator can subclass it and override functionality, or provide their own separate implementation.

Methods

• get_audiences

• get_audience_segments
  TODO Might this be too mailchimp-specific? Perhaps other providers need other ways of segmenting the audience, e.g. by specifying a list of filters? (although Mailjet seems to have a similar layout; so do SendGrid, Campaign Monitor and ActiveCampaign.)

• save_campaign (create, update, or recreate; returns new campaign ID, which might be different from the old ID, if the backend can’t update it in place)

• send_test_email

• send_campaign

• get_campaign_metrics

Rendering backend class

Methods

• render_message_html: given the output of the page’s get_newsletter_content method, returns an email-friendly HTML document.

Model mixin

Provide a page mixin (NewsletterPageMixin).

Fields

• newsletter_audience: ID of the audience.

• newsletter_audience_segment: ID of the segment.

• newsletter_campaign: ID of the campaign.

• newsletter_revision: Which page revision was uploaded.

Methods

• get_newsletter_subject: Returns a string that will be the message subject. Defaults to returning title.

• get_newsletter_content: Must be implemented by the page. Its return value should include the footer and will be passed to the rendering backend’s render_message_html method. If the MrmlBackend is enabled, it should return an MJML string.

Mail content

Where each of the bits of email content comes from. It will be possible to override any of these values by subclassing and overriding methods in the newsletter backend class.

	Django settings	Page model	Sending form
API key	configure
From (name and address)	configure
Recipients (audience and segment)			select
Subject		get_newsletter_subject	customise
Content		get_newsletter_content	preview

“Newsletter” page editor tab

• Chooser for audience.

• Chooser for segment.

• Text field for subject (defaults to page title).

• Buttons:

  • Save draft campaign (disabled after the campaign is sent).

  • Send test email (opens a dialog asking for an email address).

  • Send now (disabled after the campaign is sent).

• Link to campaign in Mailchimp web app.

• Performance metrics from the API:

  • Delivery status (e.g. “delivering”).

  • Date and time the campaign was sent.

  • Total number of emails sent.

  • Bounce summary (hard, soft, syntax errors).

  • Number of abuse reports.

  • Number of unsubscribed members.

  • Unique opens.

  • Unique clicks.

• Revision that was uploaded to the API  (link to revision in page history).

Other questions

• Mailchimp RSS: this is a hands-off approach where you configure Mailchimp to send a campaign whenever a new article pops up in an RSS feed. An editor can set this up once and it will just work, no need for any Wagtail integration.

• Wagtail Page type integration: we’ll need to add fields to the page model and an editor panel. A mixin seems like the right approach.

Nice to have (ideas for later)

• Replace media embeds with a thumbnail.

• Allow for subscriber-only content.

• Load the performance metrics asynchronously, as they require an API call to Mailchimp.