Format: 6 ten minute talks, five minutes of Q&A each. Most of this is directly quoted so no amazing insights. 😞
By Elliot, VP of Product
The lessons:
- If you don't understand why your numbers are moving, you are not doing your job
- It's easier to build on a strength than to improve a weakness - find your good acquisition/retention channels and double down on those before trying something else
- Growth wins are never certain. Find multiple ways to achieve your goals.
By Kelsey, Consumer Analytics
- Retention is important. 😐
- Set reasonable expectations of engagement (e.g. is WAU the right way to look at your users, or is MAU?)
- “Resurrected users” who previously churned but returned are a key focus of Instacart retention analysis
- Goal is to have a framework that answers three questions:
- How is the business doing? (Active user retention)
- Will it continue being healthy? (New user retention, returning user retention, engagement)
- What's driving change? (Platforms, cohorts, regions)
- Analytics are in-house tools and also redshift, tableau
- They require that all experiments run long enough to measure their core metrics in addition to what the experiment is targeting
- Found more effective retention from email than push notifications
- Early adopters usually retain better, but more noisy data because of smaller sample sizes
Ben, Growth PM, and Justin, Growth Engineer
Told a story of how Lyft went crazy giving tons of coupons to users (which drained their funding), but now that they've toned it down they've still been able to create successful coupon campaigns that increase engagement.
Tension of long term projects vs experimental attitudes
- Multiple experiments showed customers were very price sensitive, but their growth marketing team went crazy with coupons and lost lots of money. There were stacking and conflicting coupons that exacerbated this.
- Had to decide between one-off programs with custom incentives or slowing down and building something scalable; lots of pressure to do the first because of very aggressive growth goals
- Finally were able to commit to the later and built internal tools for marketers to easily design their own custom campaigns. The tools provide functions for very specific targeting and even specifying the allotment of spend. It rate limited user usage of coupons and prevented them from stacking. The tools had lots of reporting functionality that made everyone happy that they could quantify their effects.
- Effects:
- Day 1: nothing
- Month 1: Ramped up org and slightly lowered CPA
- Month 6: found new use cases that they didn't design the tool for and now have much lower CPA with more efficient incentives from testing
- Lessons learned:
- Look at ground truths and pain points to identify key infrastructure to build
- Build infrastructure to accelerate A/B testing... don't A/B test to support infrastructure
- You can't measure infra results on Day 1
- Can't imagine all use cases, but if it's part of your core competency (e.g. Pricing), then new functionality will probably emerge
- Growth at Lyft is 80 engineers
- They plan allocations of growth resources on a 12 month basis and divide cost by four quarters
- “Too much information is dangerous to executives”
By Frost Li, Growth Lead at Wish frost@wish.com
- Initially only in English speaking countries, now top shopping app in multiple continents. How?
- Looked at competitors' acquisition channels, keywords, user communication, retention
- Found that users are ready to give phone numbers in signup stage
- Asked users: “what is the best way to reach you?” Users said they didn't want to be reached, but they turned out to be most receptive to text message shopping deals
- Finding: Users in USA are picky because they have lots of options.
- Found that there is a need to redefine your acquisition funnels for different regions
- Warning: Don't project your purchasing habits onto customers!
- Found lots of problems with their translations/localizations
- 80/20 rule is real
- Q: Why didn't you just make local offices/officers to figure out localizations instead of running into all these localization problems to begin with?
- A: Establish region local offices only after starting initial growth strategies in those regions.
- Q: Free gift on signup?
- A: There is an algorithm to determine that people are potentially valuable users based on region and social signup referrals. If this can be determined near signup, users are bucketed into this onboarding flow instead.
- Q: If the above is true, how many signup flows do you have?
- A: We want different flows for each region that minimize friction... except when we're able to assess that they are valuable per above
- 60 engineers overall?
Cindy Lin, Engineering Manager
-
< 5% of world speaks English as first language
-
i18n Framework
- Sort content by tier
- Real time translation for web using memcached
-
Copy experimentation framework
- “Please share your itinerary to you coworkers and anyone in Paris” → “Please share your itinerary to you coworkers or anyone” resulted in triple the itinerary shares
- They have an internal tool to switch out translation string values without engineers
-
User Generated Content (listings)
- Restrict content using structured text
- Bad results from machine translations, use third-party contractors and users instead
-
Culture inclusivity
- Highlight reviews based on determination of shared language or cultures with users - this ranking is what produces the most success for their international growth
-
Q: How do you measure experiment success in other languages with different effects?
- A: separate experiments per region