Planeta @osantana

These are speaking notes for my October 4th, QCon talk in San Francisco.
Slides for this talk.

Over the course of my career, I’ve frequently heard from colleagues, team members and random internet strangers with the same frustration: the company doesn’t have an Engineering strategy. I don’t think this problem is unique to Engineering: it’s also common to hear folks complain that they’re missing a strategy for Product, Design or Business. But, whereas I don’t feel particularly confident speaking to why so many companies are missing a clear Business or Product strategy, I’ve come to have some clear opinions about why so many engineering organizations don’t have a written strategy.

I’ve been fortunate to be involved in architecture at many companies, including designing several iterations of Stripe’s approach to architecture (which taught me some lessons). From that experience, I’ve tried writing about this topic quite a few times:

• Magnitudes of exploration documented a public version of Stripe’s Engineering strategy
• Write five, then synthesize presents a methodology to drive Engineering strategy while operating as an individual contributor
• Writing an engineering strategy describes how Engineering executives can lead Engineering strategy at their company
• I also collected engineering strategy resources written by others into Staff Engineer’s appendix

In this talk, I hope to pull those ideas together, into a unified theory of Engineering strategy, with a particular emphasis on how you can drive strategy even if you’re not the company’s CTO. Another way to think about this talk, is that I hope to “Solve the Engineering Strategy Crisis” that so many people keep emailing me about.

What I will talk through

In this talk, I’ll work through five topics around engineering strategy:

1. Eng strategy is honest diagnosis + practical approach
2. It’s useful (↑dev velocity, ↓friction)
3. It’s everywhere, although rarely written
4. Written strategy is much more effective
5. You can advance strategy at your company

What is Engineering strategy?

Whenever I think about strategy, I start from Richard Rumelt’s Good Strategy, Bad Strategy, which three pillars of effective strategy:

1. Diagnosis - a theory describing the nature of the challenge. This is trying to identify the root cause(s) at play, for example “high work-in-progress is preventing us from finishing any tasks, so we are increasingly behind each sprint” might be a good diagnosis
2. Guiding policy - a series of general policies which will be applied to grapple with the challenge. Guiding policies are typically going to be implicit or explicit tradeoffs. For example, a guiding policy might be “only hire for most urgent team, do not spread hires across all teams.” If a guiding policy doesn’t imply a tradeoff, you should be suspicious of it (e.g. “working harder to get it done” isn’t really a guiding policy, the relevant guiding policy there might be “work folks hard and expect high attrition”)
3. Coherent actions - a set of specific actions directed by guiding policy to address challenge. This is the most important part, and I think the most exciting part, because it clarifies that a strategy is only meaningful if it leads to aligned action

I’ve found that definition extremely useful, and Rumelt’s views have shaped how I think about Engineering strategy. In particular, I believe that Engineering strategy comes down to two core components:

1. Honest diagnosis that engages with the reality your organization’s current needs and challenges
2. Practical approach to move forward while addressing the circumstances raised in the diagnosis

Sure, that sounds nice, but what does that mean? To clarify that a bit, let’s work through an example scenario. This is a scenario that many folks have experienced in their career:

1. You join a new company
2. Your team works in a Python monolith to build the Widget product
3. Your CTO hates monoliths, mandates service migration
4. You join a team building a brand new Hammer product in a new service
5. 2 years later, your old team and Widget are still in the monolith
6. You have no idea how to share code between Widget and Hammer

I believe this sequence of events keep reoccuring because of bad strategy, and is preventable with good strategy. Lets work into the components of strategy to look at how strategy could cause and prevent this scenario from happening.

Starting with “honest diagnosis” and in particular, looking at what a bad honest diagnosis would look like for this scenario. (For the record, I don’t think “dishonest” is the opposite of an “honest” diagnosis, they tend to be “bad” rather than “dishonest.”)

Here’s a bad diagnosis:

1. “We can migrate from our monolith to services in three months.”
2. “We’ve derisked our approach by moving a meaningfully complex component out of our monolith.”
3. “We’re willing to invest heavily in migrating to services, even if it means slowing down product velocity in the short term.”
4. “We are willing to expand our Developer Tools team to build new tools for services in addition to supporting our existing monolith.”

OK, but then let’s briefly consider what a good diagnosis might look like:

1. “We can migrate from our monolith to services in three months.”
2. “We’ve derisked our approach by moving a meaningfully complex component out of our monolith.”
3. “We’re willing to invest heavily in migrating to services, even if it means slowing down product velocity in the short term.”
4. “We are willing to expand our Developer Tools team to build new tools for services in addition to supporting our existing monolith.”

Disappointingly, this is the same list in both cases. In a small startup with only one simple product, you probably can migrate from a monolith to services in a few months, maybe even less. In a larger startup, that’s almost certainly impossible.

An honest diagnosis is a reality-based assessment of your circumstances. Nothing is universally honest. (Neither is anything universally bad.)

Once you find a reality-based assessment to inform your honest diagnosis, the second half of your strategy, a practical approach. The most important thing to keep in mind is that a practical approach makes explicit tradeoffs that acknowledge your real constraints, for example, here are some good approaches, even if they are a bit painful to write:

• “We want to migrate to services, but are unwilling to staff Dev Tooling more, so the migration will happen in 12 months after tooling gets finished.”
• “We don’t adopt additional programming languages, even if we prefer them, because we don’t have capacity to support them.”

What makes these good is not that they’re beautiful, ambitious statements of how we work. These are not loft “engineering values”, they are specific acknowledgments of how you’ll navigate your constraints.

Thinking back to our scenario with Hammer and Widget products, our practical approach might look like:

1. Expand Developer Tooling team by 2 engineers for next year
2. Those additional engineers will focus on tooling for services
3. Before committing to our services migration, we’ll validate by moving the Widget product to a service, and operating it as a service
4. If we can’t exceed monolith productivity within Widget, we’ll migrate back
5. No other products are allowed to spin up new services until we’ve validated the Widget migration was successful and a significant improvement (as measured by % of product eng team’s time spent on features combined with number of major Widget product ships relative to last year)

Once again, tragically, a practical approach depends on your company and your circumstances. You could write the same exact practical approach and have it go very badly indeed, which is why senior leaders often fail when they reapply familiar strategies at new companeis.

Hopefully you’ll accept the definition of “engineering strategy = honest diagnosis + practical approach”. Next, is to try to convince you that this definition is actually useful.

Engineering strategy is useful

Let’s start making the case for engineering strategy by talking through some practical examples of enginering strategy that I’ve encountered in my career.

Stripe – “We run a monolith in a monorepo.”

Diagnosis:

1. We work in a business with dynamic external forces–regulators across each country, numerous financial partners like banks, and growing enterprise customers–that change frequently and unexpectedly
2. We integrate with thousands of external financial infrastructure that are filled with bad, inconsistent, buggy technology and numerous human-driven processes
3. We have a meaningfully complex financial platform (e.g. money movement) internally that our other products (e.g. Stripe Connect) are built on

Approach:

1. We need our entire risk budget to respond to external changes
2. We reduce technology risk by running a Ruby monolith in a monorepo
3. Our developer tooling team invests heavily in running Ruby and our monorepo at scale
4. Exceptions to the above are narrow and rare (data engineering, tokenization environment)

Impact of Stripe’s strategy:

1. Innovation budget (mostly) went into product, not infrastructure
2. Avoided the decade-long journey into (micro)services that distracted most contemporaneous technology companies
3. Narrow technology landscape made it possible to concentrate investment into technologies like the Sorbet (static typing for Ruby) without an outsized investment with developer tooling

Calm – “We’re a product engineering company.”

Diagnosis:

1. We’re spending a lot of time arguing about adopting new technologies
2. We seem to be adopting new technologies out of interest in using and learning about new technologies
3. We have a long-running services migration, but only small infrastructure and platform components have been moved out. All product engineering code remains in our monolith
4. Our developer tooling team is split between supporting monolith and service workflows

Approach:

1.. We are a product engineering company 2. We adopt new technologies to create valuable product capabilities 3. We do not adopt technologies for other reasons 4. We write all code in the monolith unless there is a functional requirement that makes it extremely difficult to do so 5. Exceptions to the above are granted exclusively by the CTO, who will approve in writing in the #engineering channel

Impact of Calm’s strategy:

1. We stopped arguing about technology investments
2. We exited several engineers who didn’t want to follow our strategy
3. Combined, this meant we could consolidate our tooling investments into our TypeScript monolith
4. We started spending our innovation chips on product enhancements, culminating in ML-powered algorithm to determine best content for each user based on their behavior, UI to allow content team to self-service content management rather than require engineering support, and so on
5. This was initially viewed, by some, as making it “less fun”, but ultimately meant we spent a lot more time having doing fun work that both stretched us as engineers and helped our users

Uber – “We run our own hardware.”

Diagnosis:

1. Uber was going through a period of rapid geographic expansion
2. Some of those geographies lacked a meaningful cloud presence
3. We were operating at a scale, X0,000s of servers, where economic impact of 20-30% lower cost of ownership from managing our own hardware was meaningful
4. We were willing to incur the cost of not having access to useful cloud

Approach:

1. Run exclusively on our own hardware in dedicated colo space
2. Do not store data or compute in the cloud
3. It’s OK to do networking (e.g. TLS termination) on cloud, along the line of a Point of Presence (POP)
4. Any cloud experiments beyond POPs will require CTO approval

Impact of Uber’s strategy:

1. We were able to enter, and remain within, regions that cloud-reliant competitors would be unable to maintain operations within in the case of shifting data locality regulatory changes
2. Concretely, we were able to spinup datacenter in China in ~6 months, without colocating our US or EU data
3. (Aside – this was very painful, I don’t recommend it)
4. We did a lot of Not Invented Here (NIH) to replace common cloud tooling
5. (Life is tradeoffs: even good strategies have undesirable consequences!)

Why do these strategies work?

These strategies are effective for a few reasons:

1. Many interesting properties only available through universal adoption (“we run our own hardware”)
2. Concentrate tooling investment onto smaller space (“we run in a mono repo”)
3. Reduce energy lost on conflict (“we are a product engineering company”)
4. Control your innovation budget (all three)
5. New hires, especially senior new hires, forced to engage explicitly with strategy rather than having option of ignoring it (all three)

This is the power of making explicit, consistent tradeoffs across an entire organization.

Absence shows value as well

In addition to arguing the value of strategy from these positive examples, it’s easy to find negative examples where a missing or inconsistent strategy caused a great deal of pain:

1. Digg’s 3+ year migration to V4, onto a 100% new codebase with a new database, new frontend, new backend, and new algorithms. Honest diagnosis about challenges, but highly impractical approachs
2. Stripe’s introduction of Java had unclear evaluation criteria, took years to assess effective. Rooted in inaccurate diagnosis about problems at hand
3. Uber’s invested heavily in competing routing technologies, causing significant friction. Rooted in simultaneous following conflicting approaches without aligning on approach

I’m sure you can think of examples from your careers as well!

Strategy is everywhere. Written strategy is rare

Interestingly, Uber and Stripe are well-known technology companies, and I wrote a bit above about their technology strategies were, but neither were particularly proactive at writing their strategies down.

I’ve come to believe that:

• Most companies do have an engineering strategy
• Awareness of that engineering strategy is often inconsistent
• It’s very rare for a company to have a written engineering strategy

This is the first really important takeaway from this talk: you can solve half the engineering strategy crisis by just writing stuff down.

We’ll get to solving the other half in a second.

Written strategy is more powerful

There are probably an infinite number of reasons why written strategy outperforms implicit strategy, but a few that I’ve seen matter in particularly important ways are:

1. You can get feedback on it
2. You can make updates to it
3. You can explain why you made updates to it!
4. You can clarify points of confusion
5. Nuance is important, and almost impossible in unwritten strategy
6. It democratizes technical decision making beyond a small caste of architects
7. You can hold people accountable for not following it
8. New hires can learn proactively rather than “fail their way into learning”

You can drive Engineering strategy

Two primary ways:

1. From below: how you can rollout strategy without being the CTO engaging
2. With above: how you can rollout if the CTO’s bought in

Top-down

This strategy is a modified version of the one describes in Writing an engineering strategy. At it’s core, the thing to recognize is: it’s easy to get CTO buy-in if you write the strategy that the CTO wants.

To do that:

1. Align up frequently, and take time to debug their feedback
2. Be trustworthily curious: folks know you’ll listen hard to understand their point
3. Be pragmatic rather than dogmatic
4. Have a track record of Doing The Work to build buy-in
5. Frame it as a low-risk experiment, “We’ll try for 3 months then reevaluate”
6. Let CTO decide how to break ties

If you’re reading this and your biggest thought is, “My CTO will never let me do this”, then 7 out of 10 times, I promise you that either you’re not writing the strategy that the CTO wants. The other 3 out of 10 times, there’s some internal conflict that the CTO just isn’t willing or able to resolve, which is a bit trickier, but you can approach via the next strategy.

Bottom-up

The approach to bottoms-up rollout is described in Write five, then synthesize:

1. Write 5 design docs
2. Synthesize those design docs into a “narrow strategy”
3. Do the above five times, until you have 5 “narrow strategies”
4. Synthesize those five into a “broad strategy”
5. You just wrote a really good engineering strategy

This approach definitely takes a long time, but I’ve seen it work a number of times. Even if your current strategy has some gaps in it, birthing it into an explicit strategy document will always make it much easier to address those gaps.

Recap

Here’s what we talked about:

1. Eng strategy is honest diagnosis + practical approach
2. It’s useful (↑dev velocity, ↓friction)
3. It’s everywhere, although rarely written
4. Written strategy is much more effective
5. You can advance strategy at your company

Within those topics, the two disappointingly straightforward steps that you can talk to solve the engineering strategy crisis are:

1. Writing down the existing strategy
2. Using either tops-down or bottoms-up approach to improve the quality of your existing strategy

This might not be what you were excited to do when you wrote about getting more strategic in your annual goals, but it’s what actually works.

Back in late April, I mentioned that I was working on a new book, The Engineering Executive’s Primer, with O’Reilly. I wanted to share a few notes on progress!

First, there’s a cover, shown above in this post’s image, and also in the right rail (or bottom footer if you’re reading on a smaller device). I’m quite excited about the cover, which is simple and imperfect. There is nothing pure about being an executive; it’s mostly about balancing opposing forces to the best of your ability, and I think the cover captures some of that. The map underneath the cracks is an early map of San Francisco’s Golden Gate Park (if you want further proof, try searching for “Stow Lake” whose label you can see peeking through in the crack on the right side).

Second, I’ve done a lot of writing. I’ve been sharing early chapters with the “executive” tag, which now has 28 posts, all except one of which are from this year. Every one of those is an idea that I intended for the book. Some will be in the book exactly as is (well, almost exactly, they all still need some editing), others have been trimmed down to asides to include within other chapters, and just a couple of them didn’t end up fitting (e.g. the post on creating executive LinkedIn profiles was top of mind for me as I was reworking mine for the job search that helped me connect with Carta, but there’s no advice I can write about any tool that’s truly evergreen advice–tools change too often).

At this point, I am nominally done writing, although what I really mean is that I’ve finished the first draft. There’s still quite a bit of editing, including incorporating feedback from an amazing group of tech reviewers (ty Jasmine, Julia, Kevin, Tanya, Uma, and Virginia), which I hope to finish over the course of September.

From there, there’s copy editing, perparing the book for printing, actually printing the book, and so on, but most of that won’t require much direct involvement from me. That means, we should be on track for the digital version being complete by the end of this year, and the physical release by June, 2024.

This is my third book, and I’d say that I have a pretty clear sense of how to write this sort of book, so it hasn’t been a particularly tortured experience pulling it together. It certainly helped that I had a couple months winding down at Calm before starting at Carta, which gave some space to focus on outlining and writing the book. I’m pretty sure I couldn’t have written this while ramping up at a new job if so much of it hadn’t already been pulled together. In particular, the chapters that I think are exceptionally good were all written by the time I started, including Writing an engineering strategy, which I hope will be the enduring piece from this book. (Perhaps that’s wishful thinking, as it’s a topic I’ve been trying to land for a long time now.)

Alright, now I’m off to edit, prepare for a talk on engineering strategy at QCon San Francisco in October, and continue my work at Carta.

Uber’s original performance process was called “T3B3” and was remarkably simple: write the individuals top 3 strengths, and top 3 weaknesses, and share the feedback with them directly in person. There was a prolonged fight against even documenting the feedback, which was viewed as discouraging honesty. On the other side of things, there are numerous stories of spending months crafting Google promotion packets that still don’t get their authors promoted. Among those who’ve worked within both Uber and Google’s promotion processes, there are advocates and detractors, and absolutely no consensus on what an ideal performance process looks like.

Compensation is a subtly different set of problems, but similarly there are no universally appreciated compensation processes out there. Highly structured, centrally orchestrated compensation systems often converge on most folks at a given level receiving similar compensation, even if their impact is quite different. More dynamic compensation systems disproportionately reward top performers, which introduces room for bias.

Because there’s no agreement on what performance or compensation process you should use, you’ll likely end up working within a variety of systems. This post digs into:

• The conflicting goals between those designing, operating, and participating in performance and compensation processes
• How to run performance processes, including calibrations, and their challenges
• How to participate in a compensation process effectively
• How often you should run performance and compensation cycles
• Why your goal should be an effective process rather than a perfect one

Every one of these systems is loaded with tradeoffs and traps that you’ll need to be wary of, and after finishing this post, you should be prepared to plot the right course for your organization through them.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

Conflicting goals

Going back to Uber’s T3B3 performance process–where you told someone their top and bottom three areas for a given half–what’s most remarkable was its radical simplicity. It was focused exclusively on providing useful feedback to the recipient. To this day, I find that clarity of purpose very remarkable, and genuinely rare.

Most performance and compensation systems have far less clarity of purpose, because they try to balance many priorities from many stakeholders. Your typical process at a given company is trying to balance all of these goals:

• Individuals: want to get useful feedback to grow, want to get promoted as soon as possible, want to maximize their compensation, and they want to do it quickly.
• Managers: want to provide fair and useful feedback to their team, want to promote their team as appropriate (and in alignment with the various commitments they’ve made to their team), want to provide appropriate compensation (and, once again, do so in alignment with the various commitments they’ve made to their team). Oh, and they also want to do it quickly as well!
• People (or Human Resources) Team: want to ensure individuals receive valuable feedback, create a “floor” for quality of feedback that individuals receive, document feedback that can be used in performance management and legal scenarios later to support the company’s perspective, need to demonstrate performance process for compliance purposes (e.g. SOC2 requires annual performance reviews), and create structured input to include in calculating compensation.
• Executives: decide who to promote based on inconsistent evaluations across managers, optimize allocation of a fixed compensation budget to meet organizational objectives, and minimize the impact of inexperienced and misaligned managers on promotions and compensation.

I’ve never encountered, or heard of, a process that solves all these problems elegantly. My informed guess is that there simply isn’t any process that works with hundreds of people that isn’t a bit laborious to operate within. There’s also no way to flawlessly balance the goals of objective, consistent outcomes and recognizing exceptional individuals.

There’s a lot of room for improvement in these processes, and they can absolutely always be improved, but the tension in these process is inherent to the participants’ conflicting goals. These conflicting goals are real, fundamental, unavoidable, and must be kept in mind as you make decisions about how your process works.

Performance & Promotions

I’ll start out talking about performance processes, including promotions. Your baseline performance process is each manager providing written feedback for each of their direct reports, including a decision on whether to promote them, but there are quite a few details and variations to consider.

The first variations to consider are whether to include peer and upward feedback. Upward feedback is a constrained problem, as each person should only have one manager. In the worst case, asking for upward feedback generates low-value feedback, often because the individual doesn’t want to criticize their manager, but it doesn’t take up too much time.

Peer feedback can take up a significant amount of time, particularly for highly connected individuals who may be asked to provide peer feedback on ten or more individuals. This is usually accompanied with the advice that you can decline peer feedback requests if you get too many, but many individuals find it difficult to decline peer feedback requests, even if they know they should.

More importantly, my experience is that peer feedback is very inconsistent, and I’ve come to believe that each’s team’s beliefs are the value of peer feedback determine whether the feedback is actually useful. I’ve managed teams who feel peer feedback is too uncomfortable to give honestly, and those teams have provided useless peer feedback: in those cases, it’s not worth collecting peer feedback. I’ve also managed teams who believed feverishly in the value of peer feedback, and those teams generated insightful, valuable feedback. As such, I prefer to lean towards empowering managers to make the decision on collecting peer feedback for their team. Often this is a policy decision enacted for the overall company, and in that case it’s not a battle I’d pick.

Levels and leveling rubrics

Agreeing on performance ratings and who should be promoted is nearly impossible without written criteria that describe the dimensions of expected performance for each level. However, before we can talk about leveling rubrics, first we have to talk about levels.

Most companies have paired titles and levels, such as:

• Entry Level Engineer (Level 3)
• Software Engineer (Level 4)
• Senior Software Engineer (Level 5)
• Staff Software Engineer (Level 6)

The specific levels vary widely across companies (there many sites that show how levels differ across companies), and what is a “Level 3” at some companies might be a “60” at another, and a “601” at a third. There is no consistent leveling standard across companies. It’s fairly common for Software Engineering levels to start at “Level 3”, as companies use levels across many functions, and often reserve “Level 1” for entry-level roles in roles with fewer entry requirements.

Titles vary even more widely across the industry, and there certainly isn’t a universal standard to adopt. If you are in the position of setting titles for your company, I recommend using the fairly typical progression of Entry-Level Software Engineer, Software Engineer, Senior Software Engineer, Staff Software Engineering, and Sr Staff Software Engineer. If you’re tempted to experiment with new titles, note that the downside is that it makes your hiring process more complex since you have to explain what the titles mean, and you will lose some candidates who are worried the non-standard titles will harm their career trajectory.

Once you establish Engineering’s titles and levels, the next step is documenting the leveling rubrics that describe expectations for operating within each level (again, there are a variety of sites that collect publicly available leveling rubrics from many companies). This can be a very sizable endeavor, and I’d recommend skipping the hardest part by picking a reasonably good one that’s available online, creating a working group to tweak the details, and then refining it after every performance cycle to address issues that come up.

Additionally, I’d emphasize a few things that I’ve learned the hard way over time:

• Prefer concise leveling rubrics over comprehensive ones: there’s a strong desire for leveling rubrics to represent the complete, clear criteria for being promoted. The challenge, of course, is that many folks are exceptionally good at gaming specific criteria. For example, Stripe’s promotion criteria included mentorship, and I encountered folks who claimed to mentor others because they scheduled a meeting with that person, unrequested, and said that constituted mentorship.

  Concise rubrics require more nuanced interpretation, but attempts to game rubrics mean that all options in practice require significant interpretation. You can respond to each attempt at gaming with even more comprehensive documentation, but your rubrics will quickly become confusing to use, more focused on preventing bad behavior than providing clear guidance for the well-intentioned.

• Prefer broad job families over narrow job families: a classic executive decision is whether Site Reliability Engineers and Software Engineers should have different leveling criteria. Let’s say you decide that yes, separate criteria would be more fair. Great! Shouldn’t you also have separate rubrics for Data Engineers, Data Scientists, Frontend Engineers, and Quality Assurance Engineers?

  Yes, each of those functions would be better served by having its own rubric, but maintaining rubrics is expensive, and tuning rubrics requires using them frequently to evaluate many people. Having more rubrics generally means making more poorly tuned promotion decisions, and creating the perception that certain functions have an easier path to promotion. I strongly recommend reusing and consolidating as much as possible, especially when it comes to maintaining custom rubrics for teams with fewer than ten people: you’ll end up exercising bespoke judgment when evaluating performance on narrow specializations whether or not you introduce a custom rubric, and it’s less expensive to use a shared process.

• Capture the how (behavior) in addition to the what (outcomes): some rubrics are extremely focused on demonstrating certain capabilities, but don’t have a clear point of view about being culturally aligned on accomplishing those goals. I think that’s a miss, because it means you’ll promote folks who are capable but accomplish goals in ways that your company doesn’t want. Rubrics–and promotions–should provide a clear signal that someone is on the path to success at the company they work in, and that’s only possible if you maintain behavioral expectations.

My final topic around with levels and leveling rubrics is that you should strive for them to be an honest representation of how things work. Many companies have a stated leveling and promotion criteria–often designed around fairness, transparency and so on–which is supplemented by a significant invisible process underneath that governs how things actually work. Whenever possible, say the awkward part out loud, and let your organization engage with what’s real. If promotions are constrained by available budget and business need, it’s better to acknowledge that than to let the team spend their time inventing an imaginary sea of rules to explain unexpected outcomes.

Promotions and calibration

With leveling criteria, you can now have grounded discussions around which individuals have moved from one level to another. Most companies rely on managers to make a tentative promotion nomination, then rely on a calibration process to ratify that nomination. Calibration is generally a meeting of managers who talk through each person’s tentative rating and promotion decision, with the aim of making consistent decisions across the organization.

In an organization with several hundred engineers, a common calibration process looks like:

1. Managers submit their tentative ratings and promotion decisions.
2. Managers in a sub-organization (e.g. “Infrastructure Engineering”) meet together in a group of 5-8 managers, including the manager responsible for all of the sub-organization (e.g. “Director of Infrastructure Engineering” who the other managers report into), to discuss each of the tentative decisions within their sub-organization.
3. Managers reporting to the Engineering executive meet together with the Engineering executive, and re-review tentative decisions for the entire organization. In practice, this is too many folks to review in detail, so this round typically focuses on promotions, top performers, and bottom performers.
4. The Engineering executive will review the final decisions with the People team, and then align with other executives to maintain some degree of consistency across organizations. They’ll also review how the proposed ratings and promotion decisions will impact the current company budget.

The above example has three rounds of calibration (sub-organization, organization, executives), and each round will generally take three to five hours from the involved managers. The decisions significantly impact your team’s career, and the process is a major time investment.

The more calibrations that I’ve done, the more I’ve come to believe that outcomes depend significantly on each manager’s comfort level with the process. One way to reduce the impact of managers on their team’s ratings is to run calibration practice sessions for new managers and newly joined managers, to give them a trial run at the process before their performance dictates their team’s performance outcomes.

Another way is for you, as the functional executive, to have a strong point of view on good calibration hygiene. You will encounter managers who filibuster disagreement about their team, and you must push through that filibuster to get to the correct decisions despite their resistance. You will also find managers who are simply terrible at presenting their team’s work in calibration meetings, and you should try to limit the impact on their team’s ratings. In either case, your biggest contribution in any given calibration cycle is giving feedback to your managers to prepare them to do a better job in the subsequent cycle.

While most companies rely on the same group to calibrate performance ratings and decide on promotions, some companies rely on a separate promotion committee for the later decision, particularly for senior roles. The advantage of this practice is that you can bring folks with the most context into the decision, such that Staff-plus engineers can arbitrate promotions to Staff-plus levels, rather than relying exclusively on managers to do so. The downside is that it is a heavier process, and often generates a gap between feedback delivered by the individual’s manager and the decision rendered by the promotion committee, which can make the process feel arbitrary.

Demotions

The flipside of promotions are demotions, often referred to via the somewhat opaque euphemism, “down leveling.” Companies generally avoid talking about this concept, and will rarely acknowledge its existence in any formal documentation, but it is a real thing that does indeed happen.

There are three variants to consider:

• Demotion with compensation adjustment: for example, your level goes from Senior Engineering Manager (L6) to Engineering Manager (L5), and your compensation is adjusted to be appropriate for an Engineering Manager (L5). Equity grants are, of course, particularly messy to adjust in this way.
• Demotions without compensation adjustment: as above, your level goes from Senior Engineering Manager (L6) to Engineering Manager (L5), but your compensation is not adjusted down to match the new level. This is good for you, but in most compensation systems you will exceed (or be close to exceeding) the pay band for the previous level, which means you will see very limited adjustments going forward.
• Title demotion without level adjustment: your title goes from a Senior Engineering Manager to Engineering Manager (L6), while maintaining the same level (L6). This means compensation will keep treating you in the same way, but organizationally you’ll be treated as a member of the lower level, e.g. not publicly considered a Senior Engineering Manager, not included in forums for Senior Engineering Managers, and so on.

All of these approaches are a mix of fair or unfair, and come with heavy or light bureaucratic aftereffects to deal with going forward. These bureaucratic challenges are why most companies try to avoid demotions entirely. Further, the concept of “constructive dismissal” means that demotions need the same degree of documentation as dismissals. It’s certainly not a time saving approach.

I avoided demotions entirely for a long time, but I have found demotions to be effective in some cases. First, there are scenarios where you mis-level a new hire. They might come in as a Staff Engineer (L6), but operate as a Senior Engineer (L5). In that scenario, your options are either to undermine your leveling for everyone by retaining an underperforming Staff Engineer–which will make every promotion discussion more challenging going forward–or to adjust their level down. I’ve done relatively few demotions, but few is not zero. I have demoted folks in my organizations, as well as those I directly managed, and the outcomes were better than I expected in every case where outright dismissal felt like the wrong solution.

Floor for Feedback

When you’re designing processes, I think it’s helpful to think about whether you’re trying to raise the floor of expected outcomes (“worst case, you get decent feedback once a year”) or trying to raise the ceiling (“best case, you get life changing feedback”). Very few processes successfully do both, and both performance processes focus on raising the floor of delivered feedback. This is highlighted by the awkward, but frequent, advice that feedback in a performance process should never be a surprise.

Because performance processes usually optimize for everyone receiving some feedback, it’s unwise to rely on them as the mechanism to give feedback to your team. Instead, you should give feedback in real time, on an ongoing basis, without relying much on the performance process to help. If you’re giving good feedback, it simply won’t help much.

This is particularly true as your team gets more senior. If senior folks are getting performance feedback during the performance process, then something is going very wrong. They should be getting it much more frequently.

Compensation

As an Engineering executive, you will generally be the consumer of a compensation process designed by your People team. In that case, your interaction may come down to reviewing the proposed changes, inspecting for odd decisions, collecting feedback from senior managers about the proposals for their team, and making spot changes to account for atypical circumstances.

That said, I have found it useful to have a bit more context on how these systems typically work, and I will walk through some of the key aspects of how these processes typically work:

• Companies typically build compensation bands by looking at aggregated data acquired from compensation benchmarking companies. Many providers of this data rely on companies submitting their data, and try to build a reliable dataset despite each company relying on their own inconsistent leveling rubrics. You’ll often be pushed to accept compensation data as objective truth, but recognize that the underlying dataset is far from perfect, which means compensation decisions based on that dataset will be imperfect as well.

• Compensation benchmarking is always done against a self-defined peer group. For example, you might say you’re looking to benchmark against Series A companies headquartered in Silicon Valley. Or Series B companies headquartered outside of “Tier 1 markets” (“Tier 1” being, of course, also an ambiguous term). You can accomplish most compensation goals by shifting your peer group: if you want higher compensation, pick a more competitive peer group, if you want lower compensation, do the opposite. Picking peers is more an art than a science, but it’s another detail to pay attention to if you’re getting numbers that feel odd.

• Once you have benchmarks, you’ll generally discuss compensation using the compa ratio, which expands to “comparative ratio.” Someone whose salary is 90% of the benchmark for their level has a 0.9 compa ratio, and someone who has 110% of the benchmark for their level has a 1.1 compa ratio.

  Each company will have a number of compensation policies described using compa ratios. For example, most companies have a target compa ratio for new hires of approximately 0.95 compa, and aim for newly promoted individuals to reach approximately 0.9 compa at their new level after their promotion. Another common example is for companies to have a maximum compensation of 1.1 compa ratio for a given level: after reaching that ratio, your compensation would only increase as the market shifts the bands or if you were promoted.

• Every company has a geographical adjustment component of their compensation bands. A simple, somewhat common, implementation in the United States is to have three tiers of regions–Tier 1, Tier 2 and Tier 3–with Tier 2 taking a 10% compensation reduction, and Tier 3 taking a 20% reduction. Tier 1 might be Silicon Valley and New York, Tier 2 might be Seattle and Boston, and Tier 3 might be everywhere else. Of course, some companies go far, far deeper into both of these topics as well, but structurally it will be something along these lines.

Whatever the compensation system determines as the correct outcome, that output will have to be checked against the actual company budget. If the two numbers don’t align, then it’s almost always the compensation system that adjusts to meet the budget. Keep this in mind as you get deep into optimizing compensation results: no amount of tweaking will matter if the budget isn’t there to support it.

Whatever the actual numbers end up being, remember that framing the numbers matters at least as much as the numbers themselves. A team that is used to 5-7% year over year increases will be very upset by a 3% increase, even if the market data shows that compensation bands went down that year. If you explain the details behind how numbers are calculated, you can give your team a framework to understand the numbers, which will help them come to terms with any surprises that you have to deliver.

How often should you run cycles?

Everyone has strong opinions about the frequency of their company’s performance cycles. If you run once a year, folks will be frustrated that a new hire joining just after the cycle might not get any formal feedback for their first year. If you run every quarter, the team will be upset about spending so much time on the process, even if the process is lightweight. This universal angst is liberating, because it means there’s no choice that will make folks happy, so you can do what you think will be most effective.

For most companies, I recommend a twice annual process. Some companies do performance twice a year, but only do promotions and compensation once a year, which reduces the overall time to orchestrate the process. There’s little evidence that doing more frequent performance reviews is worthwhile.

The only place I’ll take a particularly firm stand is against processes that anchor on each employee’s start date and subsequent anniversaries. For example, each employee gets a performance review on their anniversary of joining the company. This sort of process is very messy to orchestrate, makes it difficult to make process changes, and prevents inspecting an organization’s overall distributions of ratings, promotions or compensation. It’s an aesthetically pleasing process design, but it simply doesn’t work.

Avoid pursuing perfection

In The Engineering executive’s role in hiring, my advice is to pursue an effective rather than perfect hiring process, and that advice applies here as well. There is always another step to improve your performance or compensation process’ completeness, but good processes keep in mind the cost of implementing each additional step. Many companies with twenty employees provide too little feedback, but almost all companies with 1,000 employees spend most of their time on artifacts of performance that could be devoted instead to giving better feedback or on the business’ underlying work itself rather than meta-commentary about that work.

As an executive, you are likely the only person positioned to make the tradeoff between useful and perfect, and I encourage you to take this obligation seriously. If you abscond this responsibility, you will incrementally turn middle-management into a bureaucratic paper-pushing function rather than a vibrant hub that translates corporate strategy into effective tactics. Each incremental change may be small enough, but in aggregate they’ll have a significant impact.

If you want to get a quick check, just ask your team–particularly the manager of managers–how they feel about the current process, and you’ll get a sense of whether the process is serving them effectively. If they all describe it as slow and painful, especially those who’ve seen processes at multiple companies, then it’s worth considering if you’ve landed in the wrong place.

Summary

This post has covered the core challenges you’ll encounter when operating and evolving the performance and compensation processes for your Engineering organization. With this background, you’ll be ready to resolve the first batch of challenges you’re likely to encounter, but remember that these are extremely deep topics, with much disagreement, and many best practices of a decade ago are considered bad practice today.

Everyone in an engineering organization contributes to the hiring process. As an engineer, you may have taken pride in being an effective interviewer. As an engineering manager, you may have prioritized becoming a strong closer, convincing candidates to join your team. As a more senior manager, you will have likely shifted focus to training others and spending time with candidates for particularly senior roles.

As an engineering executive, your role in the hiring process will shift once again. You’ll continue to make some key leadership hires yourself, but you’ll spend more and more time designing and debugging your overall interview process.

In this post, we’ll cover:

• Establishing your overall hiring process, including job descriptions, rubrics, hiring loops, and so on
• How many executives focus so much on perfect hiring processes that their processes fail
• Your role in monitoring and debugging the hiring process
• Helping close key candidates across your organization
• Options for leveling candidates appropriately
• How managing headcount is a key part of managing hiring
• How to train your hiring managers to avoid challenges like pursuing non-existent unicorn candidates
• Calibrating hiring inside and outside of your network, along with considering company-internal candidates
• Evaluating building an engineering brand for your company
• Deciding whether to introduce a hiring committee process
• Remembering that the system serves you, not the opposite

After reading through, you’ll have a clear plan for structuring your overall hiring process, as well as your specific role within that new process. You’ve spent much of your career serving a hiring process, and now you need to create a system that serves you.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

Establish a hiring process

Unless you’re joining an extremely early-stage company, the engineering organization will already have some sort of hiring process in place. Unless there’s widespread agreement that the current process isn’t working, you should participate in the existing process to get a feel for how it works.

It’s almost always the case that you can adapt the existing process to accomplish your goals rather than starting from scratch, and incorporating what already exists will both simplify retraining the team on the new process and build good-will with the folks who built the previous process.

Regardless of where you start, your final process should include every one of these components:

• Applicant Tracking Systems (ATS): a good ATS is the core mechanism for coordinating your interviewing process. Although many early companies try, running an effective hiring process without an ATS is time intensive with limited return: don’t try it. There are enough reasonable options out there that I won’t recommend any one in particular.

• Interview loop documentation: every role should have a documented interview loop that covers the interviews, the trained interviewers for each interview, and links to each interviews’ definition and rubric..

• Leveling framework: articulate how you level candidates based on their interview performance and prior experience. In particular, describe when you level candidates in your process.

• Interview definition and rubric: define an explicit problem or set of questions to ask for each interview. Then add an explicit rubric for evaluating that candidate’s answers. My experience is that it’s preferable to be very consistent on which questions to ask. For example, using the same programming problem for all candidates’ programming interviews.
  
  A frequent pushback is that candidates will cheat by learning the problem from previous candidates, which is certainly possible. However, I’ve found the risk of cheating is still lower than the risk of poor signal due to solving inconsistent problems. (Furthermore, it’s usually pretty clear which candidates are cheating. Make sure you have additional sections for them to complete if they go fast, and note if their ability to solve those sections degrades in a surprising way.)

• Hiring role definitions: every interviewer, hiring manager, and recruiter will engage with your hiring process using assumptions built on the prior processes they’ve worked in. This will often lead to disagreement between hiring managers and recruiters about who’s responsible for the closing process, who has input on the offer’s compensation details, and so on. The best way to avoid this is being very explicit about who is responsible for what.

• Job description template: you should create a baseline template for job descriptions, with a consistent structure and background on your organization, benefits, and mission.

• Job description library: hiring managers should use the job description template to write a job description for each role they hire. These job descriptions should be aggregated in a shared folder where they can be reused rather than reinvented. This also simplifies keeping descriptions updated as you refine shared components.

• Hiring manager and interviewer training: finally, the last component of an effective hiring process is a clear mechanism for training your interviewers. The most effective process I have seen is having new interviewers shadow several trained interviewers, combined with one reverse-shadow interview where an experienced interviewer shadows (and gives feedback to) the new interviewer.

  There are certainly other approaches to consider, including training materials or classes, but I’ve found that many interviewers simply don’t listen in those training, whereas shadowing and reverse-shadowing is much harder to fake your participation.

If you’re joining a relatively scaled engineering organization, it’s likely that most of these will already exist, and that you can quickly formalize the few undocumented portions. On the other hand, if you’re joining a smaller organization, it’s quite possible that you’ll start from a place where none of these materials exist. In the latter case, I’d aim to introduce one or two components at a time over the course of a year: going too fast will overwhelm the team, but isolating changes will lead to retraining fatigue from the team as their hiring process changes repeatedly.

Pursue effective rather than perfect

The two biggest errors that executives make in designing their hiring processes are not designing a process at all—hopefully addressed by the preceding section—and designing overly heavy processes that make it ineffective to hire. The latter is particularly challenging to notice, because you’ll often believe you are optimizing the process when you’re actually slowing it down.

The three clearest indications that you’ve over-optimized your hiring process are:

• Recruiters hire fewer than five engineers per recruiter per quarter (excluding the scenario where they are constrained by headcount)
• You frequently need new interview loops or new interview questions
• It routinely takes more than two weeks from a candidate’s first interview to you making an offer to that candidate

Each of these indicate a process that’s consuming a lot of energy without generating much impact. Often the cause is an indecisive executive who adds steps to find clearer signals, which generally obscures reality rather than clarifies it. I’ve also seen this caused by well-meaning, structured thinkers who are trying to replace biased human judgment with more structured approaches. Neither of these are inherently bad ideas, and it’s through inspecting the above indicators that you can check whether you’re really improving your process or if it just feels like progress.

As the responsible executive, I recommend you require a high bar for each extension to your hiring process. Even if individually they make a great deal of sense, in aggregate they will quickly make the process too cumbersome to operate. Each specialized interview loop, each additional interview to design and train the team on, each approval step, each movement from an accountable individual to a committee–all of these will improve quality, but often in a way that leads to worse outcomes as the process grows heavy. If the current process works, even if it’s not ideal, push the team to work with it rather than extend it. You should certainly modify the process when it’s wholly broken, or when you can improve the standard path for everyone, but stay wary of specializations, customization, and the bespoke.

Monitoring hiring progress and problems

Once you’ve built the hiring process, your job as an executive is generally to monitor and debug it, rather than serve within it. This is particularly true after Engineering grows past 100 members, at which point you’ll be directly involved in the process for a small fraction of the senior-most hires.

Here are the mechanisms I’ve found effective in monitoring and debugging hiring, in the order that I’d consider rolling them out if I joined a company without much oversight:

• Include Recruiting in your weekly team meeting: your Engineering leadership team should have a weekly team meeting, and I strongly encourage including a tech recruiter in that meeting. Their presence makes it possible to troubleshoot recruiting topics quickly and transparently. Some topics may not be particularly interesting to the recruiters, but that’s true for some members of most standing working meeting.

  In particular, this is by far the easiest place to change hiring priorities without anyone feeling left out of the loop.

• Hiring review meeting: meet once a month with the Engineering recruiting lead, and talk through their priorities, as well as any problems that have come up. Keeping this meeting small, typically just the two of you, means you can troubleshoot difficult issues that may be hard to discuss in your team meeting.

• Visibility into hiring approval: although you should likely not be approving every hire in your organization, it’s extremely valuable to have a single place where you can see all the approvals. Often this is a private chat with Engineering’s hiring managers and recruiters, where each offer is approved.

• Out-of-band compensation approval: this is discussed more below, but similarly to seeing all hiring approval, it’s even more helpful to be an approver on all atypical candidate offers. This gives you visibility into the places where your standard operating process isn’t working for some reason.

• Monthly hiring statistics: have Recruiting report on hiring statistics for each role they’re currently hiring. It’s particularly helpful to understand throughput (hires per recruiter), time to hire, offer rate, and acceptance rate. Those four metrics, cohorted by each role, should be enough for you to identify the next set of questions to dig into.

There are, of course, always more meetings and tools that you can introduce. I’d recommend starting with a couple and going from there. As you’ve probably picked up by now, my experience is generally that you can go faster by making incremental changes than by introducing massive changes, even when your goal is transformation.

Helping close key candidates

Sometimes executives insert themselves as a final interview in the hiring process, even after their organization becomes quite large. Executives who do this tend to swear by it as an essential step, where only they can ensure the quality bar for hires remains exceptionally high. However, it tends to significantly slow down the hiring process, and even executives who believe in the most strongly will eventually scale back on this practice.

However, while it’s unscalable to remain as an interviewer across all loops, it is particularly valuable to remain engaged in helping close senior candidates. As an executive, you should be able to tell Engineering’s story and how it contributes to the larger company story, and why that makes for interesting work. You’re also best placed to address strategic concerns the candidate raises.

The approach I’ve found helpful here is three-fold:

• Letting the recruiting team know that you’re available for sell calls with good candidates.
• Updating the Staff-plus and Engineering Manager hiring loops to offer you as a sell call by default for candidates who ask for it. Many candidates won’t, but some will, and these are the candidates who will shape your company the most quickly after they’re hired.
• Inserting yourself into the selling process for particularly high importance roles. Even if you don’t say something new, often the same message from an executive will carry more weight, and your involvement is a clear signal to the candidate that they’re considered an important hire.

I’ve only found this counterproductive in two scenarios. First, some executives add a sell call as a mandatory part of the hiring process, which often creates more friction than it’s worth. There are candidates who are excited to accept without meeting you, and for them the additional sell call will slow things down, and executives are particularly painful to schedule. Second, there are some executives who are exceptionally bad at selling their organization. A friend once did a sell call with an executive who turned out to be watching online videos during the call, which unsurprisingly did not make them feel like a valued candidate.

Leveling candidates

Within the hiring process, the two most contentious topics tend to be determining compensation details for each candidate, and determining the candidate’s level. You can’t determine appropriate compensation for a candidate without knowing their level, so we’ll start there.

The first question to answer is when you level candidates in your process. The obvious answer is that you level candidates after seeing their interview performance, but there are a few issues with that. Most importantly, you likely want to conduct a different process to evaluate very senior candidates than to evaluate early career candidates. At a minimum, you’d want the interviewers to be different, as it’s relatively rare for a panel of mid-level interviewers to decide a candidate is Staff-plus, and you likely wouldn’t be confident in their evaluation even if they did.

Generally I recommend provisionally leveling candidates before they start the bulk of your interview process. For example, you might provisionally level them after they complete the technical phone screen, allowing you to customize the remainder of their process for that provisional level. You can then finalize the leveling decision as part of deciding whether to make an offer. I recommend relying on a simple provisional leveling heuristic such as a combination of technical phone screen performance and years of prior experience. This is far from perfect, but there’s generally enough signal there to determine the range of plausible levels.

The final leveling decision should be guided by a written leveling framework, which looks at the candidate’s holistic interview performance to determine a level. Part of that framework is handling disagreement around leveling, which is particularly common. The most common approach is that:

• Final leveling decision is made by the hiring manager then escalated for approval.
• Approval is done by the hiring manager’s manager for levels at or below your career level (also known as “terminal level” at some companies, this is Senior Software Engineer at many companies), and approval for more senior levels is done by the Engineering executive.

Some companies, particularly larger ones, rely on a committee rather than individual hiring managers for these decisions. My experience is that committees appear less biased, but generally introduce a bias of their own, and are less efficient than wholly accountable individuals. The counter-balance is that at a certain scale, it’s simply impossible to centralize these decisions without significantly slowing down your hiring process. I recommend introducing committees only after relying on individuals has proven too slow at your rate of hiring.

Determining compensation details

Compensation is a broad topic, which I’ll write about more in my next post, but a quick overview on determining compensation details for your offers. There are two particularly important questions which should be detailed in your hiring role definitions: who calculates the initial offer, and what are the approval steps once an offer has been calculated?

The approach that I’ve found effective is:

• Recruiter calculates the offer and shares it into a private chat channel with the hiring manager.
• Offer approvers are added to the channel to approve decision to make an offer, the candidate’s leveling, and the offer details.
• Offers following standard guidelines, e.g. at or below 1.0 compa-ratio, require approval from hiring manager’s manager, and those outside the guidelines require approval from both the hiring manager’s manager and the Engineering executive.
• Any escalations or modifications to the offer occur within the same private chat, to ensure all relevant parties are aware.

A centralized approach where recruiters follow a structured process to calculate offers has many benefits. First, it facilitates training on the shared process, and retraining on that process as your compensation bands adjust, you experiment with offer strategy and so on. Second, it avoids less effective hiring managers leaning on compensation, such that those hired by worse hiring managers get outsized compensation packages. (Which is surprisingly common, although of course it’s almost always framed as the weak hiring manager pursuing exceptional candidates.) Finally, you can still design a process to break bands if you want to, but with a centralized mechanism to make it easier to both manage costs and drive consistency.

Some managers argue that this approach doesn’t give them enough flexibility to make compelling offers to the best candidates. That’s true, but I’ve consistently found that there’s always another way to close a candidate other than more compensation. Further, outsized compensation packages will always create ongoing problems in your annual compensation process, which will be designed to normalize compensation across individuals with similar performance ratings at a similar level. Your broader perspective as the Engineering executive is necessary to balance these incentives, whereas an individual hiring manager is almost always incentivized to hire even if it creates a long-term mess for the wider organization.

Managing hiring prioritization

The intersection of headcount planning and hiring is discussed in How to plan as an engineering executive, but is worth mentioning here as well. In practice there are two fundamental modes of prioritizing hires:

• In rapidly growing companies, there is so much headcount that you’ll be constrained by recruiter assignment rather than headcount.
• At slower growing companies, headcount is the more likely constraint.

In both cases, you’ll frequently have teams pushing for higher priority for their roles. I’m a believer in forcing leaders to solve within their constraints rather than frequently shifting those constraints, but my preference is just one of many ways to approach these tradeoffs. The most important thing to highlight is that both recruiter assignment and headcount are global constraints that you must control as the Engineering executive.

This control can either be something you do personally, or something you delegate to one individual to do on behalf of the wider organization, but they must be made centrally. Making these decisions centrally doesn’t mean that you have to spend a lot of time on it. The simplest way to sidestep this is to determine the headcount and recruiters for each Engineering sub-organization (roughly, each area corresponding to one of your direct reports) and then allow those sub-organization to optimize within their boundaries and allocations.

The biggest trap to avoid is prioritizing recruiters based on hiring need will often steer all recruiting capacity towards your least effective hiring managers. My learned belief is that slow hiring is almost always an execution issue by the hiring manager or the recruiter, and only infrequently the consequence of limited staffing. The exception is when you’ve opened too many concurrent roles for your current recruiter staffing, which is easy to diagnose by looking at the ratio of recruiters to roles (if you have more than three open roles per recruiter, something is very likely going wrong). If you really want to help, first consider spending time training the individuals involved rather than shifting headcount or recruiter staffing.

Training hiring managers

Earlier, I mentioned shadowing and reverse-shadowing as an effective mechanism to train interviewers. That is a crucial part of an effective hiring process, but there’s a second component of training that’s often ignored: training your hiring managers.

There’s a handful of particularly common hiring problems that are usually due to untrained or inexperienced hiring managers:

• Demanding unrealistic, unicorn, candidates from the recruiting team, such that you never make offers
• Allowing any concern from any interviewer to block a candidate from getting an offer, such that you never make offers
• Pushing for non-standard compensation on every candidate rather than learning to close candidates within the standard compensation bands
• Being indecisive on candidates, asking for additional interviews until the candidate withdraws from the process
• Refusing to talk to candidates early in the process and then blaming recruiting team that senior candidates aren’t interested in finishing the process

If you identify one of these, then I do recommend running focused trainings for your hiring managers on the specific topic that’s coming up. These are all topics that I’ve devoted a session of my Engineering Managers Monthly meeting to, talking through examples of why it’s problematic, why it’s not a sign of strong hiring, explaining what reasonable pass-through rates look like for a healthy hiring loop, and recommending strategies for overcoming the issue.

Once you’ve done a training session, you and the recruiting team should point out the issue to individuals who are running into it, and hold them accountable for fixing it. Folks making these mistakes will often have conviction that they’re doing the right thing, but don’t get swayed by their conviction. Effective hiring processes hire candidates. Hiring managers are accountable for their hiring process. Any argument suggesting one of these is false is a flawed argument.

Hiring internally and within your network

When I worked at Yahoo!, our team needed another engineering manager. We didn’t run a hiring process, or even do interviews. Instead, our Director brought on a colleague he’d worked with before. That new manager soon decided he needed a tech lead on his team. We didn’t run a hiring process, do interviews, or consider candidates on the existing team. Instead, our new manager brought over one of his previous colleagues. A third previous colleague reached out to our Director, and without a single interview we’d soon hired a new Chief Architect who would ultimately never write or read a technical specification about our product, nor contribute a single line of code.

One of my teammates–one who had joined the team through the more traditional route of interviewing–described this pattern as the flying wedge, and it’s emblematic of the worst sort of network-hiring. Hiring exclusively from your network will convince your existing team that they and their networks aren’t wanted for important roles at your company.

A similar, somewhat common, scenario is one where your company exclusively fills important roles with external hires. Each individual external hire may make sense, but in aggregate the pattern of prioritizing external hires will encourage your team to seek career advancement elsewhere, draining your organization of context and continuity.

When it comes to internally or externally hiring and hiring within or without your network, the ideal path is always moderation. Hire some internally, some externally, some within and some without. Too much of any path will either isolate your culture from valuable opportunities to evolve, consolidate it onto the culture that worked at a former employer, or prevent it from coalescing to begin with.

In my experience, almost everyone agrees with the above statement, but quite a few don’t follow its advice. As I’ve dug into that, it’s generally because of a missing hiring skill:

• Inability to hire outside their network (or within it)
• Inability to fairly evaluate internal candidate (or external candidate)

Periodically look at the number of internal versus external hires for senior roles within your organization, and dig into areas where there are exclusively hires of one sort. If you find a lopsided pocket of your organization, talk with the relevant leader and push them to make one hire of the sort they’re currently ignoring. Even one will force them to acknowledge the skill gap, and start the process of fixing the imbalance.

If the person with a significant imbalance is you, then take it seriously! Don’t hide from it by justifying the unbalance with philosophical or intellectual rationales, and instead push yourself to make one hire of the other sort. Particularly for new executives, I often find there’s an underlying belief that they cannot close strong external candidates, and disproving that belief is an important part of your personal growth.

Building an Engineering brand

The details of building an engineering brand are discussed in Building personal and organizational prestige, which I’ll avoid repeating here in full. Instead, I’ll briefly repeat its conclusion regarding building Engineering brands in particular:

• Some companies do a tremendous amount of investment into their Engineering brand, those companies will generally tell you that their branding efforts are a fundamental part of their success
• However, most companies, including some very successful ones, do very little Engineering branding, and rarely find themselves stymied by its absence. This doesn’t mean you shouldn’t do any, but rather than you shouldn’t view it as a fundamental necessity
• If you choose to invest into Engineering branding, a small investment can capture most of the upside. The biggest exception is for companies who sell to engineers in addition to hiring them, where you may prioritize a larger brand effort from a lead generation perspective

In general, if you’re already finding enough top-of-funnel candidates for your hiring process, don’t spend more time here unless you can connect that time to another business objective, or have internal folks who find this work energizing enough to take it on as a side project.

Should you introduce a hiring committee?

Many companies introduce centralized (Engineering-scoped) or semi-centralized (Product Engineering or Infrastructure Engineering-scoped) hiring committees as part of maintaining a consistent hiring process. I’ve seen this happen frequently enough in Silicon Valley companies that some executives have come to believe that hiring committees are a natural or ideal landing spot.

Hiring committees are a useful tool, but I’d caution against introducing them as the obvious solution. They’re useful, but come with their own problems.

I generally dislike committees as they introduce ambiguity in who should be held accountable for outcomes. In this case, they also mean that hiring decisions are made further from the particular team, which often degrades individual decisions. These committees are also vulnerable to misaligned members. I was once in a hiring committee where a new member joined who relied very heavily on the universities that candidates attended, even when we clarified to the member that we didn’t hire that way, they refused to change and our Engineering executive was unwilling to hold them accountable to our hiring practices.

On the positive side, they are also a great mechanism for training hiring managers’ judgment on what makes a good candidate. They also introduce more consistent hiring practices across an inconsistent organization, solving a similar problem as Amazon’s Bar Raiser program. Committees are slower than a responsive hiring manager, but faster than a disengaged or very busy hiring manager.

Remember that the system exists to support you

If you came up as a rules-minded leader, you can almost certainly think of examples where your executive responsible for designing the hiring process also ignored that process to accomplish an immediate goal. Personally, I was most annoyed by executives who steamrolled the process to hire former colleagues who performed poorly in our interview process. Each time I’d complain to colleagues, “Why did we build this comprehensive hiring process if we don’t even trust its decisions?”

As is often the case, as I switched into the role of the executive responsible for Engineering hiring, I began to appreciate why perfectly following the process was difficult. When I vowed to loyally follow the hiring bands, sometimes I’d find peer executives paying far outside the bands, implicitly penalizing hires in Engineering. When I endeavored to respect each negative hiring review, sometimes I’d encounter interviewers who refused to use the stated rubrics. When I hired for a brand new role, I’d sometimes find interviewers who interpreted the role’s requirements very differently, even if I pulled together materials explaining the role.

Each of those challenges can be solved over time with better training, but as an executive you rarely control the timeline you’re working in. Sometimes your problem is urgent today. In those scenarios, the question to answer is sometimes whether the company will be better off if you solve the underlying problem (e.g. missing a leader for a key role) or if you respect the process (e.g. don’t break the rules you created). You should try to solve your problem within the process you’ve designed, but don’t get so blinded by your process that you think the process is always more important than your problem at hand. Sometimes the process is clearly less important than the current problem.

That doesn’t mean you should always ignore your process. If your interview process indicates a candidate has gaps, there is usually a valuable signal when our hiring processes decline a candidate. Even if we’re confident the negative signals are wrong, it still undermines a hired individual when their new colleagues know they performed poorly in the hiring process but were nonetheless hired. There is a cost to defying your process, just as there is a cost to following it, and as an executive you need to make that tradeoff deliberately.

Summary

This post has covered your role as an executive in your organization’s hiring, the components you need to build for an effective hiring process, and provided concrete recommendations for navigating the many challenges that you’re likely to run into while operating the hiring process. There are an infinite number of questions to dig into, but this coverage will give you enough to get started, build a system that supports your goals, and start evolving it into something exceptionally useful.