An Elegant Puzzle: Systems of Engineering, by Will Larson

October 10, 2023 (1y ago)

Here below you will find brief notes and quotes from the book. I initially took these notes for myself and I don't know how useful they may be for someone who has not read the book or does not have any management experience. Put another way, this is not a summary of the book but rather, a collection of notes about concepts that I found useful - your mileage may vary.

Warning: some notes are verbatim copies of the book. All Credits go the author.

I highly encourage you to read the book.

Introduction

Organization design gets the right people in the right places, empowers them to make decisions, then holds them accountable for their results.

Organizations

Manager of Managers

  • Ideal team size is 4 to 6 managers. When too many, you end up being a problem-solving manager, too busy to work on long-term stuff.
  • Make sure you set time aside to coach, align with stakeholders, and invest in the organisation

Four states of a team

  1. Falling behind. add people
  2. Threading waters, reduce Work in Progress (WIP)
  3. Repaying debt, add time
  4. Innovating, add slack

Teams want to climb from falling behind to innovating.

With the framework above, teams transition to new states by adopting appropriate system solutions for the current state.

#1 Hiring and onboarding are disruptive. When possible, focus on one team at a time.

#2 Help people shift from personal view to team view.

#3 Make space to work on the compounding effect of addressing technical debt

#4 Maintain enough slack to build quality into their work, operate continuously on innovation

Tip: keep innovation and maintenance teams together. Pro: higher morale.

Durable excellence

The approach of nurturing a great organization is the opposite of a quick fix. These improvements compound, creating durable excellence.

High-performing teams: they are sacred, don’t touch them.

Interruptions

Consistent availability of great documentation is one of the main significant factors in reducing interruptions.

Tools

As the organization grows you will face new issues, with alignment being one of them.

Agreeing on Strategy and vision is the most effective approach for alignment at scale.

Strategy

Strategies are documents that explain the trade-offs and actions that will be taken to address a specific challenge.

Visions are aspirational documents that enable individuals who don’t work closely together to make decisions that fit together cleanly.

Strategy

The book Good Strategy/Bad Strategy by Richard Rumelt has three sections: diagnosis, policies, and actions.

Diagnosis, describes the problem at hand, calls out the factors and constraints that define the challenge, and thorough problem statement. A well-written and detailed problem statement is essential.

Identity policies that you will apply to address the challenge. These define the general approach, and are often trade-offs between competing goals.

When you apply your guiding policies to diagnosis, you get the actions. Writing them is the easy part, the tough part is to follow through (commitment).

As Strategies are specific to a given problem, it is Ok to have several of them.

Vision

An effective vision helps people think beyond constraints and lightly aligns on progress without requiring tight centralized control. Vision can be detailed, but details should be used to illustrate a dream vividly, not to prescriptively constrain its possibilities.

  • Vision statement - aspirational statement to summarize the document.
  • Value proposition - how will it be valuable to users and/ or company.
  • Capabilities - what capabilities will the product need to deliver?
  • Solved constraints.
  • Future constraints.
  • Reference material.
  • Narrative - brief, easy-to-digest summary.

Ok to have visions for distinct areas

Tips: Test the document with your peers and iterate. Refresh periodically

Metrics and baselines

Goals decouple the "what" from the "how"

Goals are a composition of 4 numbers

  • A target - what you want to reach.
  • A baseline - where we are today.
  • A trend - describes current velocity.
  • A time frame - sets bounds for the change.

Two particularly interesting kind of goals: investments and baselines. Investments describe a future state that you want to reach, and baselines describe aspects of the present that you want to preserve.

Tip: pair your investment goals with baseline metrics. Eg: performance optimization that may impact costs or operational burden.

Scaling consistency - decentralize decision-making groups

As organizations grow, there is subtle slide into inconsistency, which is often one of the most challenging aspects of evolving from a small team into a much larger one.

The solution is often to consolidate significant authority into the hands of a few. Many will feel a significant loss of freedom, as their area of decision-making will be newly limited. Less obviously, some people will find the creation of centralized groups empowering. When faced with the decision of creating a central authority, it helps to think in terms of positive and negative freedoms.

  • A positive freedom is the freedom to do something (eg: pick a programming language)
  • A negative freedom is freedom from things happening to you - for example the freedom not to be obligated to support additional programming languages, even if others will greatly prefer them.

How are you shifting freedom, and who are you shifting them from? When ownership is particularly diffuse, cautiously authoritative groups do increase net positive freedom for individuals without greatly reducing negative freedom.

Group Design

Influence: how do you expect this group to influence the results?

Interface: how will other teams interact with this team?

Size: 4 to 6 as a real team?

Time commitment: how much time will they spend working in this group. Will it be their top priority? Will they still be working on other projects? Time commitment will be higher for areas where folks are directly impacted by the consequences of their decisions, and to be lower for scenarios with weaker feedback loops.

Identity: should members view their role as their primary identity? Should they continue to identify as members of their current team? Tip: to shift identity, you need small teams and high time commitment

Selection progress: how will you select members? Best to use a structured process, starting from requirements and skills required. Memberships often signal status , which makes having a structured process especially important.

Length of term: how long will they serve? Are these permanent assignments or fixed terms? If fixed, can they be re-elected?

Representation: should be based on the domain under control

Failure modes

Four ways these groups fail:

  • Domineering - most common when those making decisions are abstracted from the consequences of the decisions. Eg. architecture groups in which the members write little code.
  • Bottleneck - often trying to do too much, at risk of burn-out.
  • Status-oriented - more emphasis on being member of the group than group nominal purpose.
  • Inert - group does not do much of anything. Not gelled or too busy.

Tip: embed a manager into each group and have him/her iterate on the format to reduce the risk

Communication

Frame why topic is important - especially if audience is not familiar

Provide a narrative - how things are, how you got there, where you are going - keep it brief

Prepare for detours - especially with senior leadership

Derive actions from principles. Provide a mental model of how you view a topic, allowing folks to get familiar with how you make decisions. Eg: data driven

Discuss the details: be familiar with details, reframe question when necessary if too far into details

Prepare a lot practice a little. Eg: practice if company wide (little detour, time matter). Less important for senior leadership, as often they detour

Make a clear ask: go into the meeting with a clear goal. Start meeting by framing the goal

Time management

Quarterly time retro: look back at how you invested your time. Time allocation, use it to shuffle allocation for incoming quarter.

Prioritize long-term success over short term quality.

Finish small, leveraged things. If you are doing Leveraged Work, each thing you finish should create more bandwidth. https://lethain.com/building-technical-leverage/

Stop doing things - with structure. Identify critical work you don’t do, recategorize that newly unstaffed work as organizational risk. https://lethain.com/organizational-risk/

Delegation: https://lethain.com/close-out-solve-or-delegate/ When you are working “in the system”, design a path that will enable someone else to take on that work - even if it takes a long time (eg one year)

Size backward, not forward: eg: allocate x hours to skip-level meetings and meet as many ppl as possible rather than trying to meet everybody, as it does not scale.

Trust the system that you build - at some point, you have to learn to trust. Handling off responsibilities to handle exceptions.

Decouple participation from productivity.

Approaches

Every policy you write is a small strategy, built by identifying your goals and constraints that bring actions into alignment with those goals.

The cost of creating and maintaining a policy is high enough that there is not much point in writing a policy that does little to constrain behavior. In that case, write norms, which provide nonbinding recommendations. As they are recommendations, they do not require escalations to address ambiguities.

Exception Debt

Reasons why applying policy consistently is difficult

  • Accepting reduced opportunity space - good constraints made trade-offs that narrow the opportunity space.
  • Locally suboptimal - satisfying global constraints leads to local inefficient..

When we’ve picked thoughtful constraints to allow us to accomplish important goals, we need the courage to bypass these (local) opportunities and accept local inefficiencies. If we don’t, we incur most of the cost and receive few of the benefits.

Policy success is directly dependent on how we handle requests for exceptions. Granting expectations undermines people’s sense of fairness, and sets a precedent that undermines future policy. With many exceptions you create an exception debt: work the policy.

Don’t ignore requests for exceptions. Once collected enough of them, revisit the constraints, eventually merge in the challenges. This is important as it creates a release valve for folks who are frustrated with rough edge cases, while ensuring everybody is operating in a consistent fair environment. IOW, escalations will be used as input for updated policies, not handled in a one-off fashion.

Tip: declare future time when policy will be updated.

Tip: document the problem (escalation) but don’t try to fix it right away. Batch exceptions first

Velocity & priorities

Identify core constraints Focus on “finished” work, not work “done” Partial work has no value. Document and apply your prioritization guiding principles

Tip: sharing how you prioritize work allows others to understand the rationale behind your decisions.

Management

  • The Golden Rule is the principle of treating others as one would expect to be treated themselves.
  • Give everyone an explicit area of ownership that they are responsible for
  • Reward and status quo from finishing high-quality work
  • Lead from the front, and never ask anyone to do something you wouldn’t

Management is an Ethical profession:

  • How we treat a member of the team who is not succeeding.
  • How we pitch the company to candidates
  • Provide growth opportunities
  • Working hours

People over processes

Process is a tool to make people collaborate, and the process that a team enjoys is usually the right process. If a process isn’t working, look into why it isn’t working

Do the hard things

If you have a poor relationship with someone, don’t avoid it. Spend even more time with them.

If two engineers have trouble working together, don’t separate them, get them together to help them understand each other’s perspective.

Growth

Mostly dealing with new problems. Not exactly Novel, but they are problems that your company has never prioritized long enough to get a usable solution. Most problems are people problems.

Execution is the primary currency in the growth phase. Typically have a surplus of fairly obvious ideas to try, and constrained in bandwidth for evaluating those ideas.

More ideas aren't helpful - rather, supply resources to execute ideas

Outside of growth, focus on basics: relationships, gelling teams, working on career development.

Managing up

  • What your manager needs to know
  • What problems are you trying to solve
  • That you are making progress (ie not stuck)
  • What you prefer to work on
  • How busy you are
  • What is your professional goal and growth areas
  • How do you believe you are being measured

Maintain a document with this information

  • What are their current priorities
  • What are they stressed about?
  • Is there anything you can do to help?

Common management mistakes

  • Only manage down
  • Only manage up
  • Never manage up
  • Optimize Locally
  • Don't spend time building relationships
  • Define their role too narrowly
  • Forget that their manager is a human being
  • Mistake team size and/or title for impact
  • Don't trust the team enough to delegate
  • Only see the problems

Roles

Director - you manage managers

VP - you manage an organization

Close out, Solve, or Delegate

For every problem that comes your way, you must pick one of these three options:

  • Close out. Close it in a way such that this specific ask is entirely resolved. This typically means making a decision and communicating it to all involved participants.
  • Solve. Design a solution such that you don't need to spend time on this kind of ask for the next 6 months. This often requires designing a process or norm.
  • Delegate. Redirect the ask to someone who is responsible for that kind of work or delegate to someone who can work on it

Careers

Hiring

  • Be kind to candidates
    • Beware of interview burnout - give engineers a sabbatical, etc.
  • Ensure all interviewers agree on the role requirements
    • Especially true for roles that vary between companies (eg. Eng Managers)
  • Understand the signal your interview is checking for (and how to search that signal out)
    • Break down interview into slots to that cover skills and requirements
    • Eg: presentation, live coding, debugging, etc.
  • Come to your interview prepared
    • Unprepared = uninterested
  • Deliberately express interest in candidates
  • Create feedback loops for interviewers and the loop’s designer
    • Pair new interviewers with experienced ones
    • Ask candidates for feedback
  • Instrument and optimize as you would any conversion funnel

Tip: If you’ve got an approved req, you have approval to grow your team. To add new skill sets. To build more stuff. In your role as a manager, I ask: “What’s more important than growing your team?”

Tip: Start with metrics - make sure the hiring funnel is instrumented and you have a good understanding of its performance, where candidates drop off, etc.

Design test for each skill - favor tests where candidates can show their strengths (rather than weaknesses).

For each test, a rubric. How to assess performance of each test

Q: for all the candidates you have hired, how does their work performance relate to their interview performance? What elements correlate with success?

Tools

Tip: Teams and organizations have a very limited appetite for new processes; try to roll out one change at a time, and don’t roll out one until the previous change has enthusiastic compliance.

Tip: process needs to be adapted to its environment and success comes from blending it with particular context.

Sprints

  • Team knows what they are supposed to be working on
  • Team knows why their work is valuable
  • Team can determine if their work is complete
  • Then knows how to figure out what to work on nextùstakeholders can learn what the team is working on
  • Stakeholders can learn what the team plans to work on next
  • Stakeholders know how to influence teams’ plans.

As a middle-manager

  • Spend time to understand what is motivating requests for new developments
  • Spend time learning what are folks are working on to continuously validate that your team effort are valuable

As org grows, you will have duplicated efforts, misalignments, etc.

Tip: have the team write a vision document; a concise statement of the team’s goal and the strategy for accomplishing those goals.

Conclusions

This is a great Engineering Management book that I recommend to both experienced managers as well as younger managers that are on the path to becoming managers of managers.

You can learn more about the author at https://lethain.com/about/