Requiring ICs to demonstrate business impact for promotion is, at best, imprecise, and, at worst, disingenuous. Instead, two more valuable and precise measures are:
- measuring project management & technical skills (can the engineer effectively/efficiently complete large, unscoped projects?), and
- evaluating the engineer's contribution to the team's roadmap (can the engineer identify high-value projects within the team's responsibilities? do they push their team to evaluate the prioritization of their work?)
Engineering ICs cannot plan to affect business impact in a foolproof way -- even if an IC had the means to evaluate the impact of their project, it's rare that they are empowered to select projects. Impactful projects are driven partly by luck: whether the project was timely/actually important, whether you get assigned that project, and whether you are given the resources to make the project successful. The influence of luck on impact often pushes engineers to do short-term/unrisky work, when long-term maintenance or ambitious projects might be more valuable; or to work on the newest shiny buzzwords, without properly evaluating the suitability of a containerized microserviced serverless opsfree architecture for the current problem.
Judging engineers based on impact is especially problematic for SWE-1s, who do not have the freedom (or experience! or vision!) to select work. In most cases, line engineers are handed projects that are scoped and selected by someone else, with more authority or visibility into organizational requirements. There's little they can do to affect the impact they have, unlike a senior IC who may have the team and organizational support to evaluate projects before working on them. (Here, my concerns reduce to the previous incentives to work on the new shiny.)
This is not to say that "impact" isn't an important requirement for an employee! It is valuable for engineers to evaluate how their work helps move the company forward. As an alternative, you could ask an IC to exercise judgement that will maximize their impact within the scope of decisions they are empowered to make. Thus, impact can be reframed in terms of (1) can the engineer demonstrate success given a fairly unscoped project, and (2) does the engineer contribute to the team roadmap?
These account for some of the following issues with impact:
- teams have roadmaps/mandates that they are responsible for,
- projects are assigned to engineers, who only have some freedom in choosing what to work on, and
- engineers don't have visibility into the business requirements as a whole,
while capturing the important skills of:
- translating vague business requirements into efficient/effective projects
- (ex: "running recurrent tasks is difficult" --> instrumenting and refactoring the cron wrappers, "incident management is cumbersome" --> building better automation for the process)
- anticipating areas for improvement and adding to the team roadmap
An example for evaluating a project on these axes vs. impact axes: a senior engineer spends 6 months on a project that was technically impressive, but was vastly overengineered for the actual requirements of the project. This is poor execution, not low impact. They could still take impact into consideration (what is the cost vs benefit of this particular solution, compared to other solutions of the same problem? did they consider the tradeoffs in choosing a solution?), but is different from ranking projects on a team and promoting someone for having worked on the biggest project.
(note: this is a compilation of opinions/advice from several people!)