ML PM Interview Notes Written for Swanand on 2022-04-06

DataSciencePMInterview.md

• Focus Areas
  • Metrics
    • Ability to understand how does a machine learning model performance tie back to product performance e.g. does reducing ML error improve NPS? Retention?
  • Errors
    • Candidate should be able to navigate a confusion matrix, without being confused
  • Experimentation
    • Most PMs coming from the feature/problem solving nature of the universe are often fumbling around when they realise that experiments have high failure rate — even after everything worked on your local tests
    • The ask here is not for patience, but for the ability of the PM to call out/rank different errors. Here is an incomplete example:
      • Credit card fraud: Is it okay if you blocked emergency medical payment while on travel?
        • Payments in new geographies often get blocked
      • This is equal parts Machine Learning and "process" — maybe don't block, but approve the transaction and call the customer in next 30 minutes to 7 days till you get a confirmation if this was a valid one or not
  • Common Blockers
    • ML PMs need to have a good sense of what good data label annotation schemes look like e.g. inter-annotator agreements, hard label mining and so on
    • PMs should be willing to dig deep and write their own SQL to pull both data: training and predictions/errors
      • This helps them get a sense of what biases in the training data are getting amplified by the model
• Things I'd ask (in addition to what the Product Org usually asks)
  • Begin by giving them my actual problem and ask how they'd solve without Machine Learning but without humans e.g. what would their Excel/off the shelf solution look like?
  • Have them talk to an ML engineer with 0-3 years of experience and collaborate on how they'd scope a sprint?
    • This can be a high signal idea for both skill and humility
    • ML Scoping is hard and most PMs push the engineers in the wrong direction e.g. use "less data" to reduce training time instead of improving checkpointing tooling
  • Someone senior e.g. CTO/VP Product asks them how they'd work with a ML EM or equivalent there of — ask how they'd organise demo to begin with. ML typically doesn't have a lot of visual changes like traditional product, so shallow PMs don't have anything beyond metrics.