debugging-best-practices.md

Debugging Best Practices

Preparation

• Establish a clear understanding of the issue
  • Ask issue reporter/lead about issue if needed
• Reproduce the issue
  • Is it a workflow that is causing the issue
  • Is it particular data that is causing the issue
  • Is it a combination of the above
  • Is there a timing issue or a race condition issue
  • Is there a system constraint we are hitting
• Establish high level mental model of what the system is doing
  • What is the system currently doing
    • eg A system sends a message via kafka to our application, it processes the message and stores it in the database
  • What should the system be doing
    • eg The system should be storing the name in the kafka message we recieve

Before you look at Code

• Read error messages carefully
  • Check for exceptions inside exceptions
    • Investigate the root exception
  • For stack traces with multiple error messages find the most useful error messages
    • eg /etc/truststore.jks not found is more useful than Auth failed
  • Try to understand what the system is telling you with the error message
  • Unfortunately sometimes error messages can not very useful or even worse misleading
  • Search most useful error messages and try other error messages
    • Carefully evaluate solutions to errors
      • They may solve your error/problem, but create other problems

Debugging

• Start from a point that works and work your way to a point that doesn't
  • eg The UI sends a message to the backend that is correct, but db doesn't have the correct values
    • debug from the backend api endoint to storing in the db
  • Use a debugger!
• Test different sets of changes
  • Bisect small code changes
  • Bisect commits
  • Bisect versions
  • Narrow down which set of changes are causing the issue
• For harder to reproduce bugs read through the code carefully
  • Consider Race conditions
  • Consider scope of code
  • Issues with assumptions on database/frameworks/libraries