Pysangamam.md

Problems come in different forms

• The website you use doesn't sort based on two or more fields.
• You want to get an alert based on a particular condition.
• You want to visualize the data in a different way to take a decision.

Data rich world and we can engineer solutions.

• Most of the data you want to manipulate is structured.
• Getting hold of the data helps you expand on the solution rather than being constrained on the current drawbacks.
• Adds value to the data in hand.

Example use cases

• Scraping HSC cut-off results to perform queries on the cut-off.
• Scraping market data to find funds that fit your investment choices.
• Writing a bot that remind you to pick up umbrella based on weather.

Making sense out of data

• Analyzing 10k modules in Clojure ecosystem for JDK 9 incompatibility issues .
• Finding security issues in Rust ecosystem.
• Analyzing 30k bugs in Python bug tracker.

Good things

• You feel powerful and sharing code helps others who have the same problem.
• Personal problems are fun to hack and expand your horizon.
• Python with requests is fairly good for most of the use cases before you jump to scraping frameworks.

Not so good things

• Decide if it's worthy enough to automate something. (https://xkcd.com/1205/)
• Don't grow bad code habits in the name of automation and quick prototyping.

Closing quote

When you don't create things, you become defined by your tastes rather than ability. Your tastes only narrow and exclude people. So create. - Why The Lucky Stiff, Author of Poignant Guide to Ruby