innovation.txt

Why are you not innovating?

Sorry about the title. Some backstory on it though: over a watercooler conversation
with Alain by the coffeemaker robot, I was discussing the boring topic of "problem
solving" and he suggested I did a "lunch and learn". Which I later realized was may be 
his way of telling "just shut the fuck off".

Now, about "lunch, and learn" - the first part is motivating to most animals like us.
However, when you mix "learn" with "lunch", the latter can easily spoil the charm of the former. 
Specially, if the topic is as boring as "problem solving".

Alain followed up a few weeks later. He suggested if I was still willing to run the "lunch and learn".
Now, based on my previous interpretation of his interest on this topic, and on me in general,
I was a bit surprised to be honest. 
I said "yes" and he asked if the title should be "problem solving". I thought about that for a 
moment, and to be honest, it sounded awful. Bounced a few ideas, and came up with this new 
title instead, "Why are you not innovating?". In Alain's words, it is: "Short. Catchy. leaves me wanting more."

Well, the "innovation" behind this title goes like this. To put a charming spin on the rather boring topic, 
I made an effort to learn from the clickbait gurus.
Visited BuzFeed, the king of all clickbait articles, and found a lot of their titles had the word "You" and ended in a "?". 
Then, followed a few more articles about the characteristics of a clickbait title, and here you are. The "true story" bit
is straight from Steve Jobs, what a story teller he was. We may not like to "learn over lunch", but who can deny a "story time"? 
While we have eating habits just like animals, we do much better than other animals
in our ability to connect over stories. 

If you haven't already realized it, I spent a great deal of time so far
to tell a story about the buildup of this talk. I hope with this little story, we've now made good friends.

My PhD story begins in 2012. We had a baby coming. My wife and I worked downtown and often took a long walk in the Prince's Island
park. We'd go home by 6:30 and basically find things to keep us busy until midnight. Netfix had a limited selection, and 
I was never into gaming.

Out of profound boredom, one day I decided to email my professor from MSc saying I wanted to meet. He responded right away and we 
met the day after. I asked him if he'd be ok with me doing a PhD as a part-timer. He accepted it, and told me that
the University had no such program as a part-time PhD. However, we built a good relationship from my masters days.

The suggested I did a systematic literature review on a topic of my interest and try to publish it before actually getting admitted
to the university to see how I felt about the workload. At that time, I also switched jobs to join SourceFire. New job was exciting
and it was a welcome relief from the traveling that was a large part of my job at ThoughtWorks.

So, I read a few papers about systematic literature review and started a review on the topic of application integration. I just came off
of a project for AT&T, where it was hugely painful to integrate with another team as both of us were building our software at the same time. The wounds were still raw from the horrors of that project and I wanted to do some research to learn better ways to handle this in future.

Systematic literature review is essentially a well-documented repeatable review process where you say why you're interested on a review and document the actual review process so anyone can follow it. Lacking a better understanding, this is what I thought the systematic literatire review meant.

I submitted the paper to a good venue in early 2013 feeling quite good about its chance of being accepted. My baby boy was born. On a bright sunny day, me and my wife went for a walk around the neighborhood for the first time with our little one. About 5 minutes in, I even took a photo of my wife with our son curiosly looking at the outdoors. Then, I heard a ding on the phone. The much anticipated email had arrived from the paper review committee. "Dear Mr. Sohan, We regret to inform you that..."

Fast forward to 2015, and I had sharpened the focus of the review and widened the reach of this research by looking at a larger amount of papers. I had submitted it to three more venues, and all of them had the exact same results. Like I get from the Tim Horton's rollup offers, "Try again."

I felt quite hopeless. The idle times I had before the baby, was suddenly gone. And Netflix brought shows like "House of Cards" and "Breaking Bad". We had baby number two on the way. And, the SourceFire job had turned into quite a busy undertaking. I decided to call it a quit. Because, one thing for sure is, if you can't publish, you can't get a PhD.

Being a south-asian guy who grew up with good grades, I couldn't bring this shame of quitting to my family. So, I didn't.

Instead, I started working with Craig. Craig is now a professor at a university in New Zealand. He was then working as a post doctoral researcher with my professor and he was a godsend. He read my systematic literature review paper and clearly showed me what I was doing wrong - I was describing the process and not the reason or why behind the review. So, the ideas I surfaced on the paper were lost in the dull details of how, instead of shining with a purpose.

I realized that, to innovate, I had to understand my problem myself. That means, I needed to be able to set the context very clearly so anyone could connect with it to be interested in the details that followed. Once I myself understood the problem, I had a 100% success rate in publishing 4 papers in good venues, venues where I got repeated rejections before. I managed to graduate in four years, alongside my full time workload at Cisco, and two small kids with their exhausted mum. Oh, now I can finally binge watch all kinds of shows! I have a lot to catch-up on Netflix and Prime video.

From looking back at the timeline, I realized that the first few years were just the years of ignorance. I was trying to innovate on a problem space that I didn't fully understand. No wonder, the peer reviewers found it confusing and dull. In retrospect, I feel great about learning a valuable and deceptively complex lesson about innovation that I'm about to share with you here.

Let's try to solve the abstract "circle problem". Now, given this problem, we can simply draw a circle and be done with it.

However, how often do you see that a problem is not what it seemed in the beginning?

If you truly want to innovate, you need to clearly understand the problem. Ask and write down answers to questions such as "what", "why", "why not", "who", "where", "when" - you'll see that as you unwrap these answers the problem space changes from a circle to a pig to a bear to a cookie to a waffle. It's important to first recognize that most problems intially appear as a circle when they actually mean something different such as a waffle. If you want to innovate, the most important step is to find your waffle given a circle.

Once the right problem is defined, it's very easy to use a solution that you're already familiar with. Sometimes that's the best way to solve it. For any significant problem, it's very likely there's more to it. For example, if you wrote a custom XML parser in your past project to read some configuration, it's tempting to do it the same way. This is why most people don't innovate even when a great opportunity appears in front of them.

To innovate, the idea is to break your bubble. So, force yourself to think about all possible solutions to a given problem. Most problems aren't unique, and there's a high chance that some existing solutions just work. Even when that's not the case, you should consider solutions, often from other domains, that solve part of the problem. The idea then is to leverage known good solutions for those parts, and curve out a part of the problem for yourself. That's where you innovate.

Innovating while solving a large problem holistically is a rarety because your focus and efforts get diluted. So, innovate on a slice of the problem where you found no existing solutions to meet your need.

At this step, you've done the hard part of innovation - solved a well-understood unsolved problem.
 
This isn't considered "innovation" until it's published. Because, when you publish you have to think about externalizing the concepts so that they are applicable as a general solution to an unsolved problem space. There are many forms of publishing. Consider this list:

Open source, blog post, video, talks, podcasts, patents, or even building a full-on business.

Unless you publish, others including yourself will reinvent the solution a few days later. That's how we spend years as engineers without being able to show what we have innovated.

I'll share a couple of examples of innovation from my own experience.

First, going back on that PhD research:

What started as a losely defined problem of application integration ended up being "Automated example-oriented REST API documentation". However, the solution wasn't a wholistically new idea. It was built on the ideas of automated tests, HTTP interception using proxies, and automated versioning / structuring by type inference of JSON data.

We use it for our own API documentation. When you publish about this innovation, you can claim a PhD degree. Also, you may get some offers from companies like Google to hire you.