I'm looking for some advice.
Traditional web companies (Google, Twitter, Facebook) develop interesting infrastructure problems because they reach scaling challenges. Their current stack cannot support the amount of users they're seeing (one way or another), or their advanced feature set to support their product doesn't really work well in their stack, so they write more software to help handle these problems. This is cool! I like infrastructure problems!
I feel more and more at work that, even though we're growing still (ish), we're not growing so fast that we're running into scaling problems. There's no type of cluster at work that can't grow 10x or even 100x for some of them comfortably at this point (with a few caveats).* This makes it incredibly difficult to choose a new problem at work to work on, in my opinion. On top of this, we don't have great feedback loops with the product developers that develop on top of our stack. IMO this is because the stack is "good-ish enough" to ship features rapidly. They don't feel like they need anything more. (Which is not to say that there isn't something we could build that could enable faster development, or more confidence, etc., but it's not like people are CLAMMORING for it or something. Also, like, let's say we build something? How do we know we made things better? What's the feedback loop?)
So, how do you choose what you work on under these conditions? How do you choose what you work on without externalities like scaling? What the hell does the scaling team do when the company stops scaling?
I can reasonably think of a few things that would be interesting to work on for the next yearish or so (mostly in areas in which I think it's difficult to get work done today, like monitoring/logging/alerting and gaining confidence in production / debugging production), and maybe under normal circumstances that'd be sufficient, but I'm eyeing prioritizing getting a green card, which means sticking at the same employer for roughly 3 more years.
Also, maybe I should just choose the most interesting of the problems we have? Also, there are plenty of interesting and impactful non infrastructure problems. Maybe I should work on one of those?
Yours in career ugh,
*There's a few that I can think of, but the solutions to them are already in flight.