tag:graham.posthaven.com,2013:/posts Graham says wrong things 2024-03-26T18:36:08Z tag:graham.posthaven.com,2013:Post/2029999 2023-09-30T06:34:33Z 2024-03-22T17:42:20Z Why organizational hierarchies matter

Every so often it becomes popular for a certain type of thought leader to campaign against the existence of hierarchies in tech companies. Frequently, this is coming from founders or very early employee engineers. Basically, people with significant monetary interest in not sharing leadership responsibilities. In a slightly more informed, slower world, that would probably be enough to defeat the wave of thought pieces and blog posts. But this is not that world.

The reasons why this tends to catch on each time are kinda interesting, even if the causes for its cyclical resurfacing are distinctly not. Frequent readers of this terrible blog will by now know that I'm going to start with the structural approach, so let's get to that.

Reasons people might not think hierarchies are useful

  • They don't understand the value proposition of management
  • They don't understand the skill set of management
  • They are in management and don't like to do management things
  • They are not in management and would prefer to gatekeep persons unlike themselves from influence
  • They've experienced unuseful hierarchies and not useful ones
  • They're Libertarians and unused to considering how being in a group dynamic has tradeoffs that should result in an overall positive experience

I'm not going to address each of these specifically, and I won't address the last at all. No point. Let's instead take it as read that the value in having a hierarchy isn't obvious to most people, and it usually isn't obvious because they haven't seen it actually work. This is doubly more common (and frequently the source of the thought leadership cycle starting again is directly from) execs and founders who are in management roles and have no idea how to do that job, and as such don't value the job.

Many unfortunate readers will have experience with execs that are terrible at being execs, would be terrible at being managers, and because of that constantly do a bad job at both while making it difficult for anyone else to do a good job at either. Sorry.

The point of this post is not to convince your founder/CEO that they should stop acting like management is a net negative cost at all times. Probably not possible, it would require that they understand that they've been bad at their job. For the type of person we're describing, the whole point of being a founder is that no one gets to say that.

OK, Graham, so what is the point?

How to derive value from your hierarchy

The point of a hierarchy is to make clear what level of influence, responsibility, and resourcing someone is at in your company. This is because going to the wrong level for an ask is a huge waste of everyone's time. Worse, people frequently don't quite realize that, and a whole lot of effort ends up expended trying to shift something when the chosen fulcrum for the lever was never, ever going to work.

Let's talk about the generally true relationship vectors:

For each tier in a hierarchy, there is someone in the tier above, someone in the tier below, and peers at equivalent tiers in other sub organizations. Let's say we're talking about:

  • VP
  • Director
  • Manager

Let's go through these a couple of times

  • The Director owns EXECUTION of their org's functions and assigned SCOPE of RESPONSIBILITIES.
  • Their VP owns providing necessary RESOURCES for success, and guidance for EXECUTION (how much varies depending on how far up we are in the hierarchy, at VP/Director, not very much)
  • The Manager owns EXECUTION on the sub tasks of the Director's Org's functions, and the associated sub SCOPEs of the component RESPONSIBILITIES
  • The Manager is owed necessary RESOURCES for success
  • The Manager is owed guidance for EXECUTION from the Director
  • The Manager is held accountable for success by the Director, possibly modulo resourcing
  • The Director is held accountable for success by their VP, possibly modulo resourcing
  • The Director (attempts to) hold their VP accountable for RESOURCES necessary for success of the assigned RESPONSIBILITIES

Hopefully you get the way these responsibilities and relationships stack across the levels. 

How hierarchies overlap with getting stuff done

OK so yea abrupt left turn but all the above is just context necessary to understand what this is about. It's this:

I'm usually having this conversation because someone is trying to contextualize what leadership does, how they should be interacting with a system, or there's something specific relating to career growth and responsibilities that we should cover

I also like to mention that this is sort of an idealized version of what it should look like, many of these parts are often missing or incomplete, and knowing where those voids of organizational ownership and accountability are can be a multiplier in effectiveness, because you can better understand why things are failing and look for implicit or invisible work that is helping things succeed or causing them to fail. Seeing real gaps where something isn't handled (or isn't handled well) because no one owns a tier of execution is incredibly illuminating.

This builds from the bottom up, with the rough following structure to the pyramid:

Learning Environment

This is the actual baseline work that needs to get done. Hands on keyboard, doing a thing. Usually executed as a component of learning your way around.

Technical Implementation

This is the things that surround the foundational technical work. You can't just type code into space, so there's some implementation stuff that you'll directly rely on. An example here is something like "we code in python, on an linux". 

Implementation Assumptions

These are the assumptions that are baked into the first two layers. Things like "I have a functional DNS system," or "I have access to other services that are outsourced to other teams". Roughly, it's an initial boundary layer between what is inside of your influence and what is outside of it.

Implementation Complexity

This is the meta-complexity of how those implementation assumptions interact. Someone can dramatically change the scope of work that needs to be done if they outsource a critical part of the service ownership, but are they pushing risk or concerns elsewhere that could lead to problems further up or down the chain. A common example of this is an executive saying "Well, this would be easy to do if we'd just use (other solution)," ignoring that implementation of that solution would be not viable for reasons that aren't in scope for the project. This is also known as Socializing the Risks and it's how a disturbing amount of people get ahead in life. 

Functional complexity

This is where we get into the broader scope of if the implementation serves meaningful goals - moving outside of the direct technical space and into "what function does this serve, and does it do it well?". You see examples of this being done poorly when can achieve the desired outcome, but it's incredibly toilsome to do so, or in the positive sense, when the system you're interacting with encourages a functional outcome with a minimum of overhead. 

Solution fit

Does the thing actually solve the goddamn problem?

Meta-change complexity

Does the thing solve the problem by making it someone else's a problem in a way that doesn't solve the problem? Does it just go "whelp, yeah, shitty organizational forces/tech infrastructure exist, they're too complex to really meaningfully engage with, so fuck it?" Does it really understand how it'll intersect with other product, organizational, or industry drivers? Does it do a good job of taking those things into account?

Long term vector impact

How does this change the bigger picture? Is it transformational to how people interact with infrastructure? Does it become something that is clearly the right and easy and fast way to do things? Are you eating your seed corn? 

Organizational Health Impact

You're doing all of these things, but are you doing them in a way that is sustainable, healthy, and creates an environment that doesn't intrinsically burn people out? Does the organization recognize signs of overburdening and address them? Does it change when it sees them? Organizations by default want to consume people, have you made that more or less likely?


As the real fans in the chat know, I have to switch things up frequently or my ADHD gets bored. So, at this point, I'll cover the last 3 from the top down, because they're easier to explain that way. 



Execution on vision

The company has a vision of a thing it wants to achieve. The best SaaS provider on the planet, whatever. The executive leadership team is responsible for defining this and pushing that vision forward. 

Mission vectors

From that vision, there are specific areas of focus that leadership will prioritize and drive. You can't fix every problem at once, you can't go after every opportunity, this is the strategic negotiation and prioritization that executive leadership should be doing. What are they betting on? What are they giving up? The output of those conversations will be a set of mission vectors that they want to execute on. Something like "We want to capture this percentage of this part of the addressable market for our product". 

Company Function

This is how the company achieves the mission vectors - defining organizational level responsibilities, functions, how the company operates and manages moving forward, in pursuit of those goals. The decisions for things like reorgs to better move groups solving similar problems closer together, etc.


note bene: I may come back to this one to expand it some, and am interested in how it does/doesn't make sense to people.

]]>
tag:graham.posthaven.com,2013:Post/2007629 2023-08-04T04:35:37Z 2023-08-06T09:26:53Z The one about scientists & engineers & mechanics

So this came up in a slack and then i had a long expansion and someone asked me to make it a post so they could link it to people and well ok fair enough  


It’s gonna be heck of low effort post though


Technical positions can be thought of in the following model. That's not to say that they should be, or that this model is good. It is a model, don't mistake it for the reality.



- Scientists are primarily focused on expanding the capabilities of the field and finding novel ways to cause novel things to happen.

- Engineers are primarily focused on turning theory and high-level practical guidance into functional machines and infrastructure.

- Mechanics are primarily focused on whether the machine or infrastructure produces the desired outcome.



- Scientists mostly don't care about what the Mechanics think about their efforts, and only kinda care about what Engineers think.

- Engineers frequently are frustrated at the abstract nature of Scientist efforts, and often deeply horrified at the ways Mechanics get things done.

- Mechanics mostly don't give any fucks about whatever Scientists are doing, and often think Engineers spend way too much time caring about the how instead of the result.



- Scientists all-but require formal education to perform in that lane.

- Engineers usually have formal education, but it could be a deep trade school instead of academia.

- Mechanics are often self-taught through experimentation, but may have gone through trade school or academia.



- A Scientist was involved in figuring out how dead dinosaurs can make energy (and murder us all).

- An Engineer was involved in figuring out how to use that energy source to make efficient motion happen.

- A mechanic is involved when the damned thing breaks down.



- A coding Scientist is working on new algos, new crypto, fundamentally new ways of doing things.

- A coding Engineer is working on making things (based on science both old and new) run efficiently and smoothly and predictably.

- A coding Mechanic is working on shipping the fucking feature and really doesn't care what the Engineer lead thinks about their code smell.



Unfortunate truism in this model's exhibition IRL:

- The layers tend to operate the same way drivers do about other drivers' chosen speeds.

- The layer above you is lost is lost in the fucking clouds. 

- The layer below you refuses to see the big picture.



These things are actually about APPROACH not background. My computery shit is 100% shade-tree mechanic. My management theory is fucking science. My org building is some deeply skilled engineering. In each of them I'm using different approaches because that's what mostly works well for me. These are tools, not labels to be use to confine and define people.

]]>
tag:graham.posthaven.com,2013:Post/1818475 2022-05-03T04:10:04Z 2024-03-26T18:36:08Z How to build orgs that achieve your goals, by absolutely never doing that

This is about something that I wasn't ever able to explain well until recently. I've been using the same approach for a long while, but making it understandable was difficult.  I don't know if this achieves that. 

The key that unlocked the attempt was applying something I learned a while ago, and that I say a lot about my own processes, which is the next header line. While discussing what I was trying to get across with some people, I realized that what I was actually proposing was "go forth and Harry Tuttle your company" and honestly, that's probably the best possible metaphor here. Anyway, that thing I say too much: 


The purpose of a system is what it does

This is one of the things that I say often in my professional discussions, obnoxiously frequently in political/societal discussions, and at all times it is the foundation of my analytical efforts.

I do not care about the intended purpose of a system. I care about what it does

I encourage you to feel the same. You will occasionally find people who say that it's important that we care about a system's supposed purpose instead of its demonstrable purpose. In general, those people have a vested interest in the system's current impact not changing.

This focus of concern is probably fairly obvious to you. The implications may be less obvious.

  • Most systems are tuned for intended purposes that are not the system's actual purpose
  • Many systems have stated goals that are orthogonal or hostile to their actual purpose and demonstrable impact
  • Many, if not most systems have metrics being used that aren't well-correlated with the system's actual purpose
  • Many, if not most (if not actually almost all) systems are being fine-tuned and managed in ways that do not, and other than accidentally, can not improve the observable outcomes

This, obviously, is a Problem™. The usual solution is to work harder on correcting the output. Sometimes it's a whole bunch of "let's figure out the correct metric so we can measure what is happening." Sometimes it's someone doubling down on the selected or obvious metrics. Sometimes it's "accountability" impacting people's performance reviews because the system they work in isn't doing it right. This is all completely wrong, and it is also why this problem is so common. You have this problem because you have mismanaged the system in question. Applying more management of similar intent is only going to make it worse. You can't fix bad code with more code; you can't fix bad metrics with more metrics.

Are you yelling at me?

From this point, I'm going to be fairly adversarial in tone. Try to avoid reading this as me yelling at you. I'm yelling at your execs, at my execs, at every exec either of us have ever had. I'm yelling at the industry we exist in and industry in general. I'm yelling at the forces that market capitalism focuses upon us. I'm yelling at the banquet hall ceiling about this turbulent priest and no knights are about to walk in then out again. I'm just yelling, but I do have a point.

 

OK, let's get weird 

Managing an organization to the goals of the organization is some extreme "skating to where the puck is" bullshit.  Knock it off, immediately. 

Stop trying to make complex mechanisms do what you want in ways that make sense to you.

Start trying to make complex mechanisms exhibit the foundational attributes necessary for the thing you want to happen regardless of what else the system is supposedly doing. 

It is, unfortunately, sports metaphor time. Let's stick to something that we relatively simple like "track." The athlete wants to win. Does the coach manage towards "winning"? Does the coach manage towards foundational elements, and assume that winning is a byproduct thereof? Is there a split? Now let's strain this poor metaphor. What if it's a relay race competition with 100 competitors per team? Is the coach going to manage to "go win" or is the coach going to manage to the fundamentals of what a winning team looks like?

Answers are pretty obvious, right?

So why the hell are you trying to manage a security team to "numbers of reviews" or "ticket closure rate" or "how long a consulting ticket takes to get to done"? Do you think any of those lead directly to "winning"? Do you think any of those help with retention, which is undoubtedly your actual hardest problem this decade? Are you creating metrics based on "how annoyed my exec peers are with me" instead of "how healthy this security team is"? Yes. You are. The question is: "Can you make yourself stop doing that, and actually lead your org as if it is what it is, an organization made up of humans?"

Uh wait what?

Define for me what a successful security team looks like. What it feels like. Define for me what the vibe is. Define for me how people talk about that one team they were on that just fucking worked. Yes I'm serious. I'm stone-cold, dead serious. All of your metrics are causing you to push effort and people into things that aren't actually improving the things that cause your metrics to improve. Read that again if you need to. OK fine, another metaphor:

Artificial metrics are flying by instrument. They're individual "better/worse" dials that in amalgamation are supposed to tell you which way things are going, as long you are paying attention to the correct combination of them at the correct moment, and don't over-react to the feedback loops and crash the whole thing via a PIO. Instrument-only flight is harder than visual flight, it takes extensive practice, and the mistakes have worse repercussions.

You can instead choose to just fly visually. It's easier, it's safer, and it'll get you where you're trying to go. The thing is, your entire industry thinks it's impossible, and worse, they think it is irresponsible. They're kinda right. You have to be good at the innate skill of flying, instead of the skill of navigating by instrument. Guess which one the "become a manager in tech" system produces. Bonus points: recognize how that is itself a PIO. 

Bonus Bonus Bonus points: Consider that if you've learned the skillset of visual flight poorly, and you don't use the instruments to correct yourself, how will you ever know it's going wrong in time?

Yea, the thing itself is easier to do this way. The thing itself will produce better results when done this way. You cannot hire fools to do it. You have to actually hire for emotional intelligence and the ability to connect on a personal level with all your hard-skills engineers. I do actually know how to do that but I won't ever write that one down. That's my whole secret.

OK so, confused? Is this where I sell you on the self-help book and the associated cult membership? Nah. 

Integration

You're already routinely discussing terms like "emotional intelligence" when talking about leadership and management skills, even with the most obnoxiously "i am my code" people you know. I'm asking you to take those ideas and apply them as if they're primary skillsets with important uses, instead of something that's "useful for keeping the employees feeling like you care." I'm asking you to set aside the inherent sociopathy in managing people as widgets while dealing with them as people. As is often the case in all these posts, mostly I'm asking you to reject cynicism. Emotional intelligence matters for every single thing about managing humans. Why wouldn't it matter for managing your team's collective direction and output? 

What I'm getting at is that defining how you run your org should be based on those soft skills that actually affect how the people who make up that org interact, rather than what fits cleanly in a ticketing query. The query can tell you if your org is doing the things you think it should do. It cannot tell you if your org is functioning correctly, healthily, or sustainably. If you manage to the ticket metrics instead of those things, you will lose those things, and never quite understand why.

WTF I cannot put "team vibe" on a dashboard

Not with that attitude you can't. Yea, it is gonna be hard. I'm not trying to say this is easy. Your company has processes in place to make sure you don't do something weird like this. You're going to have to work around them for long enough to prove that you have a point. This is a subversive strategy. This is the Harry Tuttle method of organizational management repair. If you're lucky you can define this well enough that your own management layer can give you a window to prove it out. You won't be. You'll need to prove it out without pre-clearing it a few times, probably in a few places, before people accept that you've got a knack for this.

You won't have people crediting that the thing you're obviously doing is what you're actually doing. In general, other managers will view this as being unfairly manipulative since obviously you're misleading your people about what the real priorities are and how you will help them achieve success through them. Ain't nobody ever going to approach you to say "hay your team metrics are great and everyone wants to work on your team and you haven't lost any engineers all year and what's your secret?" They're going to ask your boss how long they will let you get away with running a good feelings daycare. Gravitas is the currency of tech engineering management, and you have to deliberately disclaim it at every opportunity. It's hard.

What matters for your team/org's success is the fundamental human relationships, comradery, esprit de corps, support and space-curation, and especially, all of the prior while treating-em-like-adults. Those things make up the totality of why people want to work on your team and are excited about working with and supporting their peers. These are not invisible things. These are things you can pay attention to, structurally. These are not things you can quantify with numbers. You're going to have to get comfortable with forming, expressing, and defending opinions based on things besides "data." Not because you don't have data, but because you don't have quantifiable numbers that represent themselves, and our industry is poisoned into believing that only such things are data. We've got thousands of years of evolution helping us understand how group dynamics are flowing. Yes, using that is a skill set. That's my point. Build and use that skill set. Learn how to read people's reactions. Learn how to understand people's motivations. Learn how to see how people work in groups and as individuals. Do the work.

Cheery "You can do it!" motivational section

When someone says "what metrics do you manage by?" I say "mostly feedback in 1:1s, career conversations, impact feedback on my people from their peers, and to an extent, retention." When they ask "how do you quantify those things?" I ask them "why the hell would anyone try to quantify those things?" 

Even the most self-deceiving engineering orgs, insisting that they make always their decisions on quantifiable data, do most of their ratings and promotion decisions based on loosely-defined statements of support and feedback forms. Objective decision making about subjective human performance is, at best, a beneficial fiction. I submit that the benefits have long since expired.

Everyone has opinions. This leads children to think that all opinions are equal. Look around you, that is demonstrably not so. 

Someone promoted you at some point because they thought that your opinions are usually right. Take a firm grasp of that and your confidence that long-term, the people matter as much as the mission. Forcefully reject all the cynical nonsense they try to sell you as pragmatism, seriousness, managerial gravitas, and lead. Lead by demonstrating that the people are the most important thing. Lead by insisting on good, supportive, healthy culture. Lead by demanding that people treat each other and themselves fairly and respectfully. Make that team a thing that those people will talk about being the best team they were ever a part of for the rest of their lives.

The work will happen. All those outputs you were worried about will flow in droves. If you have to, you can build a dashboard of nonsense based on the quantifiable outputs. Just don't go mistaking the numbers for what matters, or you'll lose what actually matters, and then you'll lose the numbers too. 





Bonus bonus bonus bonus bonus points section:  How would the company overall perform if it treated customer health/growth the way I'm saying you should treat employee health/growth?

]]>
tag:graham.posthaven.com,2013:Post/1782064 2022-01-11T23:51:27Z 2022-02-15T23:21:12Z Maintaining a healthy work culture is the first role of every executive
This is a slightly edited version of the norms document I sent out when I took over at my current role. This was not a corrective action: I work with great, talented, kind people, and was following behind previous strong, ethical leadership. This was taking that kindness and making it an explicit expectation of the group. Culture scales automatically until it doesn't, and if you're an executive, figuring out how to keep that culture healthy and scaling should be in your top two or three priorities at all times. If you lose it, your only real option is to quit and let someone else take a clean shot at doing it right without the expectations baggage you've got. You can also just not care about it of course, but we know what those companies are like.

----------------------

Team Norms

Welcome to the team norms document for <my org>. This document refers to us collectively, sometimes to you the reader, and sometimes to the leadership layer. Depending on your place in the organization, you may have multiple expectations from this document or for you in it. This is not my document. This is our document. You have suggestion rights, and you may use them. Keep in mind that, I will maintain ownership of this doc and may show up in your calendar to talk with you at length so that I can understand your suggestions. 

This document is not finished, and is never finished.

The work we do requires that we all act as responsible people, and that we’re able to trust one another. Behave as if that is true in all things, big and small. Escalate politely if it isn’t.

Hierarchy

This is a group of peers, mentors, and mentees. It has a hierarchy anyway. 

Our hierarchy exists primarily to divide work up to persons with aptitudes and interests aligned to that work. That does not mean that we will pretend it’s all collegiate fun and no one is anyone’s boss. You shouldn’t feel hassled by the fact that you report to someone, but if we pretend that you don’t, all of the power dynamics inherent in that fact become hidden. 

Let’s keep those dynamics out on the table where they can be referred to as needed. Power dynamics matter. They influence what we say and do, they influence how we respond to ideas or requests, and they influence the feedback we give and receive. It is always OK to mention them as part of any of those things. Make the subtext into context and it loses much of its unearned power. Do not insist that people who need your approval are always free to say whatever they want. Prove that it’s true, and they will. Fail them once and you'll never get another chance.

Inclusivity

This is the part where I say something about how more diverse teams build better products, and how diversity of backgrounds, identities, and opinions leads to better decisions. That is all true. However, in this organization we value diversity and inclusivity because that is the morally and ethically correct thing to do. That it benefits us, our customers, and the company is nice. We will do it regardless of how true that is. If inclusivity fails to benefit us, our customers, or the company, we will seek to realign that conflict rather than cease being inclusive. If you have a concern with this, feel free to discuss it with your lead, or just bring it straight to me. I’ll be happy to help you understand.

We support each other

If you have a payroll execution problem (that’s “mechanics of being paid,” not “compensation plan”), it’s expected that your lead treats it as a top priority. If you have a time off/emergency problem, it’s expected that your team treats covering it as a top priority. If you need to escalate something to a lead or their lead and need to stomp on a meeting, it’s expected that you will do so and they will make time or work out a time with you. Trust batteries require recharging, and knowing that we all have each other’s backs is how we keep those batteries trickle-charging at all times.

Working Hours

We do not care about your working hours for their own sake. We will not care about your working hours for their own sake.

We do care, immensely, about your ability to communicate with your peers/customers and their ability to communicate with you. We care about you getting your work done, and people who depend on you for getting their own work done. If your working hours don’t conflict with that, then they’re OK. Approach this as if it’s an important, key part of being successful here, because it is, and seek consensus with yourself, your peers, and your lead.

It’s up to all of us to keep “respecting people’s working hours (modulo emergencies)” as a shared norm. What’s that mean? It means unless you’re working on an emergency, or are doing so to flex your time elsewhere, you shouldn’t be working late. 

Please do not send non-emergency emails or chats late. It encourages other people to work late to respond, just out of politeness if nothing else. This goes on for a while and suddenly we’re responding to chat at 3AM because that’s what people expect. Gross! Let us not do that. If you have something you’ve got to type out, well, that’s the way the brain works sometimes. Scheduled send is your friend. Send the email or chat message off during the next normal working hours. 

The counterpoint is: please do not make a habit of working late and using scheduled send to hide it.

Out of Office

It's not anyone’s business why you’re not working today, modulo that there are boundaries where HR/Legal says it has to be someone else’s business when it’s multiple days. Aside from those boundaries, if you’re taking the day to watch the kids, go to the doctor, buy a car, or sit on the porch and contemplate your infinite being, that’s all fine. 

Whatever it is, it’s fine.

Don’t tell us. 

Don’t ask people about it (again, outside of HR/Legal boundaries). 

Sending a message to <internal comms> and setting your <internal messaging> status to an OoO note is sufficient notification. Block your calendar if you have time, and keep Workday accurate. We understand that many of you will have lots of societal pressure built-up about saying why you’re out and apologizing about it. Please do not. If you want to share something you were/are doing, that’s of course fantastic. What we’re avoiding is the implication that you have to justify your OoO to the team. This norm only works if we all stick to it. Please accept that little bit of discomfort you feel not explaining your absence as your personal contribution to an environment where people don’t feel the need to explain that they have a life and sometimes it interferes with work.

Meetings

There are four types of meetings:

  1. Just hanging out - This would be a wacky thing pre-covid, but it’s OK now to just have a meeting with some people to chit-chat. If someone invites you to a hangout meeting, you are 100% empowered to decline without explanation. Don’t go bugging people to accept them. 

  2. Announcements - This is a meeting for big time announcements and possibly immediate Q&A. An invite is expected to be accepted and you’re expected to show up, modulo time off and working hours.

  3. Decisions - This is a meeting where someone will make a decision. Other people are there to influence or inform that decision. If you’re optional, feel free to not show. If you’re mandatory, please do attend. If you are going into a decision meeting and the deciding person is not there, and has not sent someone to represent them who will make the decision, the default and correct thing to do is reschedule that meeting. Decision meetings do not happen without the ability to make a decision.

  4. Discussion - This is a meeting where we are going to discuss something. Being optional or mandatory on the invite isn’t very informative. Most of the “should I go or not” comes from “are you concerned that the discussion will lead towards decisions you disagree with if you do not attend?” It is always appropriate to wait for a break in the meeting flow, and ask “does anyone expect an opinion from me in this meeting?” and if the answer is “no,” and if you don’t feel the need to be there, leave. It is OK to leave a meeting you don’t think benefits you or benefits from you being there.

Meetings do not self-justify. But also, don’t be the person constantly complaining about meetings. Find a balance. While trying to do so, remember that any decent meeting has someone running it, someone in charge of a decision (if it’s a decision meeting), and if it was complex or large, it had an agenda well before it was scheduled to begin.

“Someone in charge of a decision” does not mean “a manager/lead.” It means someone informed to make the decision. That may well be you.

Virtual Meeting Expectations

Unless you are presenting, we have no expectation that your camera remains on. If you’re presenting internally and don’t want your camera on, I will usually defer to you. We have to work with partner teams that may have different norms, so we cannot set an absolute here, but within our team, cameras are great for empathic or emotional communication, but not a requirement for all communication. For many people, it’s mentally taxing to see your face reflected back at you constantly, and we’d rather you spend those mental cycles and daily willpower getting things done, and having enough left for yourself at the end of the work day.

Focus time

Feel free to book focus time onto your calendar. Please ask people to treat it seriously. Please direct them to your lead if they make a habit of not treating it seriously. We recognize Wednesday as a standard day for this, we encourage you to expand upon that if you need to. Please don’t let more than half your week become blocked-out focus time, and if you’re in a lead or coordinating role, it probably needs to be 1.5 days or less. You are encouraged to discuss coordinating “no meeting days” as focus time inside your teams and working groups, but please get peer and lead input before finalizing a decision. 

Development Expenses

The guidelines we have in <Internal Documentation> hold true for our organization, but we want to be explicitly clear that using them to their fullest is good, proper, and correct. We will bias towards “yes” for things in this category, and not “are you sure you really need that?” That bias is subject to review if it feels like it’s being abused.

Organizational Cynicism

We reject organizational cynicism at all times, and use a broad definition of the term to do so. That doesn’t mean “don’t act like nothing is cool.” Organizational cynicism is much worse than that. Organizational cynicism is “We have an approval gate on that but it’s just a 100% rubber stamp because no one has time.” It’s “There’s probably no value in this workflow, but everyone expects us to do it.” Things like that will strangle an organization, ruin its reputation, burn out its people, and exhaust its partners’ goodwill. At all times we seek to be an organization that deals with itself and its neighbors honestly. 

That honesty can be very difficult. For example, it's never easy to tell someone that you’d like to help but you’re resource constrained and rather than giving them some (insufficient) attention, you’re focusing on things that are more important. It may well be correct though for the more important thing, for yourself, and even for the less important effort that is disappointed by the decision.

A good way to think about this is that the company collectively decides where to put resources (people, time, money, etc). The company is never completely right, and frequently quite wrong, so we flex and make our own decisions too. The combination of those is how we collectively decide what’s important. It’s good to stand by those decisions. It’s also good, when there’s time, to say “are we sure that’s still the right decision?”

Yes, that’s contradictory. Sorry. (Not sorry.)

What do we do when we don’t follow the norm?

This is a document about our collective cultural norms. We’ll find other ones we want to document, either to fix something that isn’t working, or canonicalize something we like. While these norms will change over time, our default should be to resist change, and course-correct acting outside them. 

Your key phrase is: “We don’t do that here.” There’s no judgement made or implied. Point out that that isn’t how we do things here, and move on. If someone insists on it, then talk with them separately or bring it up with their leads. Escalate that if necessary, but it shouldn’t be. 

For the most part, we’re asking people to be their honest self. Where we aren’t doing that, we’re asking them to make space for the people they work with to feel free to be their own honest selves. In both cases, politely and calmly reminding someone of our culture’s expectations is helping all of us, and is a good thing to do. The best response to being reminded of the norms is “Thank you” and moving on.

OK, but what if it’s my Lead acting outside the norms?

“We don’t do that here.” 

But power structures can make that difficult. It’s OK to not rush to be the person saying it if it’s difficult for you. It’s OK to do that in private, or ask someone else to do it if that’s easier. It’s important that it happens, but that happening can come later if need be. The goal is to remind us of our norms, not to make you individually accountable for enforcing them.


]]>
tag:graham.posthaven.com,2013:Post/1558080 2020-06-11T22:41:53Z 2021-07-20T12:28:26Z 2020 seems like it's ten years long

This was sent out to my part of the Google Security Engineering org by myself after consultation and input with my local-group management peers. If you’re at Google, it’s visible as a /2020stress short link. This version of course has some edits.

I should preface this with my own perspective. My neighborhood in Security Engineering at Google has been hitting it’s goals very, very well this year. And thus I’m certain that we’ve got people who are working way too hard, because there’s no way we should be hitting all our pre-COVID goals in the midst of <waves vaguely at everything>. Then there were some conversations with people reaching out for advice about how they felt off and weren’t sure what was going on. Some of these people haven’t ever been “burnt out” before (have I mentioned, Google Security is a great place to work). 

I realized that we had a mix of “a lot of our people are working too hard right now” and “a lot of our people are overstressed in new ways right now.” That’s a combo that causes a fragile risk. What if two TLs and two senior engineers burn out hard at the same time? Those green KRs sit there staring at me, whispering “we’re costing too much, and you can’t see it.” My management peers and senior engineers/TLs have been talking about it, and a concrete and sensible plan is being worked out. I decided it was time to do something proactive, even if a little messy, about the way people were feeling, and my peers were supportive.

I’m not claiming expertise at anything except “being a person with a hilarious amount of life-long generalized anxiety disorder” best summed up by this recent tweet:



The management team has had many conversations lately with people feeling out of sorts in ways that are new to those persons. In general, they frequently sound like people struggling with the impacts of background anxiety on their lives and work. Being that I’m pretty public about my anxiety and ADHD, I decided to talk about this. I am not a mental health professional, just someone who has walked some of this road. Seek professional advice before doing anything of importance based on any of this.

Here are some things you may be experiencing from time-to-time right now:

  • Not feeling very productive/effective, and beating yourself up about it

  • More worried than usual

  • Staying up later than you usually would, but not really doing anything

  • Finding it hard to focus when you want to

  • Irritability outside of the norm

  • Feeling like you’re burnt out, but it’s different, and you’re not sure why

  • Finding yourself resentful at the work or the interrupts

  • Finding yourself working functionally, but not really caring much about it

  • Having a hard time prioritizing work or private life, together or independently


If none of that seems familiar, this probably doesn’t apply to you directly. Consider reading on to help with empathy and support for your coworkers who are affected.


Many of you are high-functioning people who haven’t experienced deep background anxiety before. I say “before” because you’re almost certainly experiencing some of it now. We’re all currently living with:

  • A global pandemic that is being managed to highly varying degrees of success

  • A global protest to the problem of police brutality in the United States

  • Working from home in environments that may be extremely non-conducive to being effective

  • Other nation-specific troubles that may be weighing on you, China, Hong Kong, and India are all struggling with things too

  • Working In a company culture based on spontaneously communicating in all internal directions, except now we don’t see anyone outside of deliberate meeting partners.

These things require extra thought and attention to deal with, and can suddenly steal focus and intrude on our minds. This can easily replicate much of the symptomatology of generalized anxiety and/or ADHD. Those of us who have lived with either of those for a long time can recognize them in the many of you. You may have been pushed into a very difficult situation without the tools to deal with it. 

First, Google resources:

<internal links removed>

Second, I’m not going to give detailed advice on handling anxiety- or ADHD-ish symptoms in a general email. If you’d like to talk to me about it, book something, it’s off the record. I expect that for many of you, you’ll be relieved at just being informed that this is real, actually affecting a lot of people, and not something wrong with you. Your managers are all happy to talk about this with you.

Third, I am going to give general advice.

1. Take more time off. If it’s kids- or family-related, make use of our generous leave. It doesn’t have to be taken in chunks, you can book pressure valve days to spend focused on your family responsibilities. If you’re not experiencing stressors at home, you can and should still take more time off. In summary, take more time off. If you ever think “ugh, I need a break” stop what you are doing and find a slot where you can take an afternoon, or day, or week. 

2. Accept that you’re not operating at full efficiency. Stop trying. Remember that we’ve got <link about no engineering heroics> codified as a norm here. Push back on work allocation that isn’t reasonable. Hold yourself and your managers/TLs/TPMs/peers accountable for realistic workloads. Realistic means “what you can do now” and not “what you’d have gotten done in 2019.” It’s not 2019 anymore and 2020 appears to be going to be a decade or two long. Settle in for a long haul and stop pushing yourself to work 30% harder to make up for all the various impacts to your functionality. I know it’s going on all over and I’m wagging my finger at all of it, including at me.

3. Define boundaries in your life and keep them. Stop working at foo o’clock. Decline meetings that stomp over your lunch break or personal/family time. This isn’t a repeat of #2. You should hold yourself accountable for realistic work output, but you also need to hold yourself accountable for keeping the space open to handle everything else, and recharge yourself enough to do good work when you are doing work.

4. Your leadership team really actually means the above. This isn’t a corporate “health is a P0” platitude. The overall situation is going to be going on for most of us through this whole year at least. We need you to be happy, healthy, and effective when this is over too. We mean it; ask us to help with it. Your entire management team are all thinking about this, we’re happy to talk about it with you, and we want to help and be helped.

5. Last, here’s some real red flags. This isn’t an exhaustive list, but if you are running into any of these, please strongly consider professional help:

    • Thoughts of self-harm
    • Finding yourself having the same worries over and over, especially if the cycle is increasing in frequency
    • Feeling like your mind can’t stop and you can’t make it stop
    • Worrying every day
    • Worrying about why you’re always worrying
    • Consistent thoughts of despair and hopelessness
    • Unable to sleep because of worry or undefined dread
6. Again take some more time off. Create space to relax. Slow down a little bit and breathe.


]]>
tag:graham.posthaven.com,2013:Post/1238855 2018-03-29T22:37:34Z 2020-10-30T04:24:40Z Security is mostly an accountability function

This is one of those "I made this post because I keep explaining the same small concept repeatedly" things. Both to save time repeating it, and (mostly) to try to clean up the concept.


Companies, decide how secure they want to be twice*

The first time is when the exec team decides, in their own terms, just how much security they want to purchase. That purchase being made in people, software, process costs, and especially meta-costs paid by every other org/effort in the company. The second time is when the exec team decides how much they want to deviate from the previous decision just this one time for this particularly important thing. That second decision gets remade over and over, however many times necessary. You (hopefully) have a security executive because that first decision only matters if someone at the exec tier is holding their peers accountable for their strategic intents. A quick, naive primer for multi-level exec politics:

  • Bosses hold reports accountable for failing to hit goals (your security exec yells at you from here)
  • Peers hold peers accountable for failing to meet shared commitments (your security exec does most of their job here, w/ other execs)
  • Reports hold bosses accountable for failing to provide resources (your security exec does their job here, once or twice a year, with their boss. You do your job with your security exec here on a frequent basis)

In a perfectly aligned company, where no one ever tries to shirk parts of the agreed-to big picture that are temporarily inconvenient, the security team could define what the risk looks like, the execs (or more likely, delegates) could continuously make decisions aligned with their security goals, and everyone would move forward together. Nobody actually does that. So let's dismiss that potential out of hand for at least another decade or so. 

That all means that a well functioning security team & security executive are, from the perspective of the rest of your company, largely an accountability minder. Their role is to point out things that damage that first central security decision and say "you're lying to yourself if you think you can do this without it damaging your other goals" or to drive requirements into the rest of the execs/orgs that are foundational, required elements of meeting the shared security goal. You are rarely actually arguing with other people about the specific topic. You're arguing with them about meeting their commitment to the overall shared security goal.

Constantly re-centering conversations to given risk appetites is incredibly useful. Generally, with honest actors, it's fantastic -- establishing "wait this was our goal, right?" with people who are dealing with you on the level is amazing. Building this habit into your security engineers will proactively prevent a noticeable percentage of escalations, and will also help your junior security engineers not make requests of partners that aren't reasonable for those shared goals. (There's a much latter stage maturity version of this where you've built an agreed-to method for estimating risk and people are generally OK with your impacts on their timelines because they know you warning them off impacting their risk budget here means they get to do something more risky later. Don't rush to this as a goal. If you get there way before the rest of the company is ready to think about these things big-picture, you will accidentally make security a 1000lb gorilla in every "a or b" decision conversation, and you'll strangle the company's ability to take informed guesses at risk decisions -- which is what it absolutely should still be doing early on.)

Unfortunately, it's also often less useful than you'd hope. You'll frequently run into people who habitually think they're the only pragmatist in the conversation. You'll run into people who discount security goals out of years of poor experiences with them. You'll also run into people acting with direct malicious intent, but they'll be indistinguishable from the "only I think pragmatically" type unless they're extremely bad at it. If you've got a choice, it's much more efficient to just choose to not work with assholes (or people who accidentally approximate assholes) who put their personal goals well out ahead of company goals, but it is an imperfect world.

If you're living in this particular intersection of startup and maturity and (hopefully un-)intentional bad actors, it can be very difficult; even demoralizing. Guiding conversations back to common security goals will seem like it isn't helpful sometimes. It still is. Even when it isn't useful for the listener, it is at least putting your head in the right place for the argument. Make it a habit. Don't make it your only one of course. If you're having these types of discussions, you need an bag full of deescalation and empathy strategies to keep a line of communication open, even if the net result of them is "we can respect each other's position, but I'm hard blocking you." Security engagement has a non-trivial number of typical conversations that end with one of the parties extremely dissatisfied. At worst, re-centering on shared goals will at least help keep it impersonal. If you find that you're using multiple conversation strategies successfully, including this one, and still have people who refuse to accept the shared strategy - it's time to decide if you're fighting an independent bad actor, or if that person is doing what the company's culture has shaped them to do. Depending on the answer, you either help them find somewhere else to work, or do so yourself. Sometimes the right answer may be that you have to change the core culture of the company to do your job, but that is a much trickier proposition.

[A note to the reader from future Graham: I quit that job and ranted about the isolated but deeply ingrained toxic asshole culture in one particular org that had forced me to quit a great job because infinite gas lighting drives everyone crazy eventually.]

[A note to Graham from this reality Graham should other realities' Grahams stumble upon this or should time travel become widely available in this reality: That first time that you thought "I shouldn't have to put up with this and if I can't convince anyone that it's happening deliberately I should just get out now," you were right and should have gotten out then. It took you almost an entire year to get over the burnout that staying caused]




*Companies decide, in the trenches, how secure they want to be a dozens times a day, and inside the security team's prioritization decisions, non-stop.

]]>
tag:graham.posthaven.com,2013:Post/1185086 2017-08-20T06:10:38Z 2020-06-12T05:26:30Z What every computer science major should know

                       Gatekeeping is for assholes and you shouldn't do it. 

]]>
tag:graham.posthaven.com,2013:Post/1113487 2016-12-07T05:49:46Z 2022-06-15T14:48:06Z Effective Line Management

This was originally a poorly formatted word doc created when some management peers asked for tips at being effective at managing their reports. A little bit (but not much) tuning later, it's now this. I make no pretense that this is my own work, it's all received wisdom from a ton of different people and sources. 

You just realized you have no idea why your employees care what you think.

Someone put you in charge. Maybe you demanded to be in charge because you kept telling everyone what to do anyway. Maybe you got stuck with it when the last person in charge bailed and you looked like you knew what you were doing. Regardless, you now have humans expecting you to make what they do and why they do it make sense. INSIDE THEIR HEADS. Welcome to management!

A non-made up, very real statistic: Depending on study, between two thirds and three fourths of employees leave their job because of their direct manager. The odds are that over the course of your career, you will be the cause of several very important employees quitting their jobs. Every time you prevent that from happening by better management you're saving that employer a ton of cash, yourself a lot of frustration, and justifying your own existence at that employer. Remember: most management is overhead; only the very best of management starts to add value again. Only people that add value to their teams should be trusted with larger teams, more complex teams, and eventually executive positions. We all know that the valley isn't really a meritocracy, but that doesn't mean we exist outside accountability either.

Hands-on line management is made up of several things:

  • How to cause work to be done at your company
  • What motivates people
  • What prevents execution
  • How to model someone’s behavior so you understand the prior two

Things we're not going to cover here include: budget, pipeline, managing up, politics, managing the HR dept, and many other important facets of being good at managing people.

Causing work to be done

This is mostly company-specific, with a bunch of generalities that are nearly platitudes. You catch more flies with honey than vinegar, etc. Be nice and professional to people, stand your ground when you're right, insist on being treated the way you treat others (and vice-versa), and generally treat people like adults until they convincingly prove they aren't. There's also a whole host of other strategies that are only viable in dysfunctional environments. You should generally learn those as needed and not by deliberate instruction, as they're all disruptive in an environment that can still function positively.

My guiding principle in this, as in all things, is that you should deliberately reject and act away from cynicism whenever you find it. Cynicism is much bigger than just acting or talking cynically. A bureaucratic process where everyone involved knows they aren’t solving a problem is cynical. An approval loop where you don’t actually consider what is being approved and how it changes things is cynical. A promotion process that defines goals that have nothing to do with performance or aptitude is cynical. Cynicism burns people out, it causes bad decisions, and it will make good employees leave. If you spend your time as a manager doing nothing but finding and fighting cynicism, you would have done more good than many achieve.

The Basics of motivation, execution, and behavior

Let’s start with the books

This is your first-tier homework. Hopefully you’ve got a fair number of “how to manage” opinions from your own experience, quite possibly unorganized. With this you begin to make those more clear and to organize them usefully. That said, management books are a lot like self-help books. You may find something that resonates for you, but that’s more likely to be a unique emotional trigger than it is to be something that works for everyone. People are complicated. If there’s any one rule of thumb here, it’s that: people are complicated. The goal here is to give you a shotgun blast of receivable wisdom, and then a dive into the areas that drive human interaction. The most useful manager is one who can understand why someone is going to react the way they will before they know they will.

Managing Humans - Michael Lopp (get the most recent edition, currently 3rd)

This is a series of bite-sized hard-learned lessons, mostly as personal post-mortems for a management interaction that went badly. This book is a chance to learn from (a very smart) someone else’s mistakes. This is an incredibly central work in tech management.

How to Measure Anything - Douglas W Hubbard

This is a strong foundational intro to modern decision science. A major component of management is how to make rational, repeatable decisions based on rational, repeatable estimates. If I could force everyone in a company to read a book, it would probably be this one.

Other suggested topics and works

Decision Theory, Statistical Analysis, Communist Manifesto, Capitalism and Freedom by Friedman, pretty much anything in recent macro or micro economics. Every time you read through a few books on a new topic, I suggest you re-skim Managing Humans for grounding. The point of these areas is that you need to be focusing on increasing the depth and complexity of your understanding of certain very complex subjects, namely, group dynamics, how bias impacts decision making, how to make decisions repeatable, how economics impacts your employees and their place in society, how your personal economics lead to different decisions than they would make. The point of all this in total is to keep your gut feelings connected to the realities that your employees live in, so they maintain confidence in your empathic relationship with them. This is why they will trust you to make good decisions and treat them fairly. Without that why, the only tools available to you are conditioned responses to negative/positive stimuli. That's not where you want to be.

Additionally, you may want to read through some of the traditional/popular “management” books in order to be able to communicate more effectively with management peers and your superiors. This is a dangerous road though, so go into it with an extremely critical eye and an assumption that you’re learning the lingo, not accepting the arguments. You may find arguments you like, and by all means, incorporate them. Just don’t do so solely because these books are popular. Here’s a brief list of such popular works:

  • Who Moved My Cheese? – Johnson
  • Peopleware: Productive Projects and Teams – DeMarco
  • First, Break All the Rules – Buckingham & Coffman
  • Slack Getting Past Burnout… - DeMarco
  • Turn the Ship Around! – Marquet
  • The New One Minute Manager – Blanchard & Spencer

People

As a manager you have six primary things to worry about with regards to your humans:

  1. Is there something preventing them from executing on the task?
  2. Do you understand how to motivate them day-to-day?
  3. Do you understand how to motivate them long-term?
  4. Are you getting better at managing them?
  5. Can you answer all of these questions about how you handle yourself?
  6. Can you answer all these questions about your boss handles you?

Let's break these down

Is there something preventing your employee from executing on the task?

The possible reasons are:

Knowledge Gap - A knowledge gap means they don't know what to do or how to do it. You can solve this with training, education, hands-on mentoring, conferences, books, etc. If the knowledge gap is extensive, you may need a long-term plan here. This is a gap that you identify mostly by building trust with your employees. If they aren’t afraid to say “I don’t know how to do this” you’ll rarely be burned by it.

Capability Gap - A capability gap means that they know how to do it, but they can’t pull it off. There’s several possible reasons for this (execution anxiety, stress, it’s too complex for their experience level, they just aren’t smart enough on their own for this problem). The fix depends on the cause. You may need to team up with them on this, you may need to partner them on this; you may need to take tasks away from them based on this.

Motivation gap - They don't want to do it, or they don't want to do it to the necessary standard. This is the hardest to solve, but we have an entire section here on motivation. Let’s get to that.

Overall Employee Motivation

This is totally different from how to keep employees motivated day-to-day. We’ve got a whole section for that after this one. This is how you need to keep employees motivated on a big-picture, long term basis. You will eventually need to build and maintain a personal model for what motivates each employee or coworker. To start with though, you should stick to the classic reasons and then modify from there based on your interactions.

If you ask ten different managers, you’ll get ten different responses as to how they model motivation. I prefer to try to stick to Maslow’s Model of Hierarchal Needs and the 4-Drive Model. The 4-Drive is fairly new, and adheres well to the motivations you see everyday in tech and sales. Maslow’s is a classic, tried and generally still true (there’s some debate about whether “internet access” needs its own tier). Neither of these are absolutely correct. They’re models that you should always have to hand to think about why someone is reacting a certain way, or how you might try to predict their reactions to change. Explaining these in depth is well beyond the scope of this document, ample resources exist elsewhere. While I won’t say that you should only pay attention to management texts that reference them, be very critical of management texts that don’t reference well-understood engagement/motivation models.

Maslow’s


Maslow’s model can basically be thought of as “people don’t care about the higher-layer needs until the lower-layer needs are met.” This is probably obvious to you, but the model breaks it down well, and it’s a useful touch point. When someone is furious and frustrated and all but yelling at you in a one-on-one, having clear models close to hand is very useful. Think about the interaction you’re having and realize that if your motivations for that person do not resonate to the layer of need that is upset with you, then those motivations cannot affect the behavior.

4-Drive

4-Drive is a model of why people do things, assuming that there are four general reasons for action/motivation. People are rarely so simple as for only one of these reasons to apply at once, but if you can try to think of someone’s personality as a group of people with their own drives, it’s very useful to guess which of these reasons is driving the behavior you’re wanting to change or to reinforce.

As with most other things about managing your team, you need to understand how these things apply to you even more than you understand how they apply to them.

Keeping Employees Motivated Day-to-Day

Keeping your employees connected to their work requires first that you keep them generally motivated, then that you keep them specifically enthused about either the challenge, what the challenge generates for them, or how their performance interacts with their environment.  There’s no one solve for this, but rather an environment of good management will lead to day-to-day motivated employees. Maintain this through a constant lifecycle of positive actions:

Be a good coach

  • Provide specific and timely feedback
  • Balance positive and negative feedback
  • Understand unique strengths/development areas of each team member
  • Tailor coaching to the individual & situation
  • Suggest solutions
  • Have regular 1:1s

Empower execution

  • Do not micromanage
  • Balance giving freedom with being available for advice
  • Make it clear you trust the team
  • Advocate for team with others outside the team

Be a decent human being

  • Genuinely care
  • Be fair
  • Make new members feel welcome
  • Show support in the good and bad times

Be productive and results-orientated

  • Keep the team focused on the results/deliverables
  • Help the team prioritize
  • Remove roadblocks
  • Be clear about who owns what
  • Be a hard worker

Be a good communicator

  • Encourage open dialogue
  • Be available for the team
  • Explain the context
  • Tell the truth - even when the news is bad
  • Be calm under pressure
  • Listen to each team member

Help with career development

  • Provide honest, specific feedback on the next step in team members' career
  • Help team members find new opportunities
  • Talk about all aspects of career development, not just promotions
  • Balance their team members’ and company’s needs

Have a clear vision for the team and a strategy to achieve it

  • Create a compelling vision/strategy
  • Clearly communicate vision/strategy
  • Involve team in setting vision/strategy
  • Build relationships with others to help the team achieve their goals

Have and use skills relevant to the team

  • Roll up sleeves and actually conduct work
  • Understand the challenges of the work
  • Help solve problems based on technical skills

Other important factors

The rest of this is basically “extra credit,” it isn’t something that you need to be focused on to start being a good manager, but it’s extremely useful. If you find yourself feeling that you’re plateauing as a manager or in your relationships with your people or peers, look here.

Mindfulness

Clarity of thought and purpose is necessary for really strong management. We’ve all run into an executive who seems to have their shit incredibly together every time they’re asking you something. That kind of clarity just comes to some people. Most of us have to work very hard at it. Mindfulness and meditation are a strong, wide pathway to that clarity. If you find yourself wishing you had more of it, I encourage you to explore both concepts.

Experimentation

Don’t be afraid to run non-damaging experiments with how your people behave. Need someone to get better at telling you when they’re too busy? Start slowly overloading them with stuff that it’s OK if something gets dropped. Need someone to give you better feedback. Illustrate the problem for them with someone else, then push them a little till they are actually kinda mad at you. Use this to help them understand how feedback matters and when they should know to give it to you. Hopefully you get enough accidental experimentation through the vagaries of life in the workplace that you don’t need to intentionally push people for effect, but if not, do so thoughtfully and ethically. People respect being taught careful lessons.

Therapy

Yes, I am literally telling you to go to therapy. You go to the doctor to make sure your body is working to your expectations. Therapy is exactly the same thing for your mind. In most cases, improving your capabilities as a human improves your capabilities as both a manager and an employee. This is both separate from mindfulness and complimentary to it.

Etcetera

This is not a complete or exhaustive list of what’s required to be a good, or even an OK manager. It’s a primer for where to start, with some pointers on where to go next. For yourself, everyone on your team, everyone you deal with regularly, and your boss, you must always have an expectation of normal behavior, a model for how to engage with and motivate them, and an understanding of what forms of feedback are productive. You also need to still get work done. It’s hard, but ultimately if management is for you, it’s probably the most rewarding thing you can do.

]]>
tag:graham.posthaven.com,2013:Post/892769 2015-10-30T00:27:45Z 2022-09-19T08:00:12Z Rockstar shopping is inherently unsafe

I have cleaned up the intro some.

This is the (very) long form of a speech I've given to a few startups who have come calling. Sometimes they listen. My observation has been that the ones that don't listen end up repeating the process a while later. If you're working on a product where security or privacy is a feature, hopefully none of this applies to you because you're already making these decisions correctly. If you're working on a product where security is an emerging need, or you've just realized that it should have been a core feature, I hope you get some use from this.

Someone just said "we really need to hire somebody to handle all this security stuff" out loud in a meeting. 


Maybe it's because you're passing 70 engineers and not having a security person seems wrong. Maybe you're at 200 because security/privacy wasn't a core component of what you're making (maybe you should have challenged that assumption earlier). Maybe you're getting 14 emails a day from security researchers about how you do something. Regardless, you have decided it's time to start thinking about security as an essential part of the product and as a role in the organization. 

You had another quick meeting with recruiting, the CTO, and someone from OPS, and tried to figure out all the things that needed to get done. Now HR has put a job posting out that reads like a list of security function buzzwords. It includes things that are tactical, strategic, high-level, low-level, PR-related, touches every type of technical skill known to humankind, and is designed to not upset any silos in your current org. Maybe this person is going to report to OPS, maybe they'll report to one of the tech leads, or maybe to the CTO. You've got a bunch of old tickets labeled "Security Debt" and you'd like them gone.

You know it's a big list, but you're not an idiot. You've got nuanced understanding of where this role can best achieve what you want done and you've built this out of a matrix of reasonable concerns. Yes, it's a bit of a laundry list and maybe it's not all achievable, but what job posting isn't like that? You'll find someone who can do the important things and get started fixing all this security stuff. At some point you'll need more security people and you'll sort out org structure and management then, but that's not important right now.

Great! That's wonderful. Kudos to you for recognizing the importance of doing this right.

Step 1: Throw that list away. 

I know, we just talked about how it's relevant to your needs. The problem with that list is that you weren't qualified to make it in the first place. You wouldn't expect to hire Your First Marketing Person or Your First Recruiter or any other entirely new position in your company and have a good grasp on what the role should be doing. You'd have things you know you want that position to do, you'd have some rough ideas about what else might be necessary or prudent, but you'd expect that position to fill in a lot of blanks. Instead, since security is a technical role, you think you know most of what it should be doing, and want to hire someone to fix that technical problem set. You're not qualified to triage these problems, and the outcomes of trying to bring in the minimal resources to deal with them, as you understand them, are far more likely to be negative than positive.

What you've done is lay out a subset of the security problem you actually have. If you're really well informed about security, it may be mostly correct. If you think you're well informed but haven't talked to actual security people, you're wrong about that set. If your recruiter tells you they talk to people like me and we say things like "hrm, it sounds like you're making the classic rockstar mistake" then you're definitely wrong, and you're into dangerous waters.

There are four potential outcomes from dedicated rockstar/unicorn shopping:

1. Someone will decide that you don't know they don't exist, they'll glue a carrot to a horse, and you'll buy it

2. You fail to learn the right lessons from being unable to find that rockstar and make a series of "good enough" individual hires

3. You give up and decide to try again "later"

4. You do find a real rockstar, who is possibly willing to handle the ten security domains you need rather than the three they find interesting. 

Let's break those down some. Keep in mind that I'm talking about the negative end of the spectrum here.
1. Buying a fake rockstar is where a lot of startups end up. That's not to say they've hired someone bad at security, they've probably hired someone outstanding who can do a subset of what they wanted and can talk about the rest well enough to get through the hiring process. That hire is probably not a dishonest actor; interviewing is hard and people like to please. You've now got one person who is supposed to solve all your problems. Since your organization thinks those problems are now all eventually solved, that person isn't in a position to drive enough unexpected change or pull in enough resources. You're on a finite timeline to failure. 

2. You decide to try to hire a couple of different security resources to get the coverage you want. Senior people aren't likely to wade into trying to fix part of the security problem without security leadership managing the big picture, so you're going to end up with mostly junior people. This is going to leave you with good talent without enough practical experience or hands-on mentoring, and you're far more likely to end up with your security engineers having a stagnant relationship with the engineering teams and not getting anywhere near enough done. The compliance person you bring in will get absorbed into audit or legal and will, typically, defend their programs instead of the company. This is a fairly normal 'lack of good leadership' pattern and you've probably seen it before.

3. You keep shopping around and never find anyone who can do all the things you want. The typical response here is to decide that the time isn't right and kick the can down the road. Startups usually default to "Well we don't want a bunch of security people; we're doing OK. Let's just wait till another 2M DAU/20 engineers/whatever." The problem with this is that your collective management brain has already realized you're at a risky inflection point. Deciding to not act is action, and it's very high-risk action. If you're still reading this, it's because you recognize the need to act.

4. Maybe you find an actual rockstar. Someone who really is good at securing all the things your network, ops, engineering, legal, and HR teams do as well as running incidents, making long-term risk decisions, and can eventually grow and manage the security team you need. Awesome. It's unlikely, but they do exist; I know a couple. The problem here is that I've just described what's about 3-4 people's worth of work in even a relatively small startup. You're going to fry that person if they try to do all those things all the time. What you think of as "security stuff" is actually an astonishing number of different contexts and modes of thought, and someone constantly flipping in and out of them, even if qualified in all of them, is very taxing. Even finding someone willing to do that constant context switching is harder than you'd think. Being capable at everything in a domain doesn't make all those components interesting, and doing all of it as a one-person-shop isn't much more rewarding than doing the parts you enjoy.
If it seems like I'm saying that the most dangerous result is finding that rockstar, then you are paying attention. It can work out wonderfully, but that's the unusual example of an already rare case. It's the most dangerous result because it's the result that hides the most unmanaged risk away from your eyes. The rest of the results will be obviously negative within a year or two, and much sooner in a good management environment. A bad rockstar hire can take four or five years to be apparently negative. Often when you discover that they built things that became central to your operations are hard to maintain/extend and massive over-solves that consumed far more resources than they should have.

So what the hell should you do then?

Short answer: Abandon the idea of a singular security hire. Reset your expectations to hiring a team across several security sub-disciplines. Hire competent technical security management first. Interview for that role based on solving some of your technical hurdles, security management competence, and management team fit. Expect to hear projections about head count and early spend before the end of the interview process. Expect to pay more than you wanted to.

I know, you're annoyed. Stick with me, and I'll back this up.

Here's a partial sample of your actual security problem sets, current and upcoming:

  • What sensitive information do you have and where does it live?
  • How do people access your infrastructure?
  • Who can access what stored information?
  • How do you know who can authorize that access?
  • What happens when a laptop gets stolen?
  • How are you dealing with external parties that are finding bugs?
  • Who does what when you get hacked? Are you sure?
  • Are you optimizing your public services for privacy?
  • When customer information shows up on the Internet, how will you figure out what happened?
  • How you decide that code and services are safe?
  • How you decide that that code is the code you wanted to run?
  • How do you make patterns, frameworks, and library decisions that will trend towards safe code/services?
  • How are you managing customer payments?
  • How are you managing employee information?
  • How are you handling security questions from customers?
  • What does your compliance roadmap look like? Are you ready for it?
  • How are you deciding what risks to take?
  • How are you deciding how much risk is the right amount for your company?
  • Are those other startups you're sharing information with doing these things right enough?

The first four bullets on that list alone can easily take a year or two of cleanup for startups that are doing things mostly right, conscientiously, and trying very hard to get the security stuff correct. More than a few big names of this decade have cleaned them up only after an FTC Consent Decree craters into their budget. If you look at that list and think "well a bunch of that isn't needed right away" you're totally right. But a lot of it builds on other components, and most of them have groundwork that you don't want to do by playing catchup. 

What you should be hiring for first is expertise in triaging the security problems versus realistic threats, and that's the security management layer. The closest analogy you'll find in the engineering world is that of CTO vs Leads. You can hire a buncha lead engineers first, but every one you hire before the CTO is going to make hiring that CTO harder, and going to make their job exponentially harder, and going to risk deeper and more difficult tech debt for them to claw back out from under.

Right now you might be thinking that I'm telling you to immediately boil the security ocean.

No, I'm not. I'm pointing out that once you decide to solve for security drivers, you're already way past the point where you can make self-informed decisions about prioritizing the the problem space. I may know when my car is being weird, but I don't have a clue how to fix it. 

You, like almost every other startup ever, transferred the vast majority of the risk of security problems to your customers. You talked about how you take information security seriously while dedicating nearly no time at all to it, deliberately not hiring for it, and occasionally saying things like "we could bring in someone to handle this but they'll slow us down too much." That's just the way it is. This is VC-based capitalism, and you have to prove success where essential while faking success everywhere else. In any specific "I can't believe company X did that" discussion it's almost always indefensible, but in general, it's the cost of doing business with other people's money. That only works for so long though, and now you're past that point. The second you're past it, you're into dangerous territory.

Even if you didn't have those pressures, no startup builds a perfect product and then releases it. You're building a boat while bailing water out to keep it from sinking. The boat has to get larger and progressively more complex without being disassembled. At some point the boat inspectors will be showing up. Also, if your boat is about important or secret things, people are going to try to break in and steal them. It's a series of devil's bargains, and you have to do all this without perfect knowledge of the forces at work internally and externally. 

I don't expect you to begin solving every security problem you'll ever have. I expect you to hire someone who can predict what those are going to be, and build to solve them within your process of building the company and product.

How do you make the right hire?

How do you pull this off? It's tough, and you've got some tradeoff decisions to make. Here's what you need to have generally agreed upon before you try to make this hire:
1. Determine your level of urgency. There's a few ways you can go. If you want someone who has done this before but isn't looking exclusively for a CSO role as their next challenge, then you're shopping in a VERY small market. If you want that to include just nice/good to work with people, you're shopping in a microscopic one. If you're willing to go with some proven management experience, well-rounded security experience, and startup experience, then you have a reasonable (albeit still small) market to shop in. You're going to find way more chaff than wheat, but there's probably enough wheat out there at any one time that you can find a good hire. If you're willing to take a chance on a senior security person who hasn't done management before, you can do dozens of interviews every week. That's probably your best way to pull this off without it being very expensive because both the other options will bring tears to your eyes.

2. Understand where you want this role to end up. You're hiring someone in the middle of a career arc for a role that may or may not have its own arc internally at your company. You should know what you expect the role to grow into or when you'll expect the role hire its own boss. This is something that should be frankly discussed in later interviews. Are you already sure your board will insist that an outside CSO be brought in at some point? Is the candidate not comfortable with the amount of compliance wrangling necessary to be the top chair in security when it's time to go public? Are they OK handling a good bit of the IC work while they find other resources for the roles they want to hire? Do they know which roles they can handle and which they can really own and which they shouldn't touch? Understand what you want in general from someone who is going to build an internal security practice, and be sure they do too.

3. Hire based on both fit and plan. You've hired management roles before so that hopefully isn't a black box to you. Evaluate candidates based on both their fit in your management team and their default plan. The discovery work and basic staffing is going to be something they can largely predict by the end of the interview cycle. They'll be able to quote you year one and two opex and capex projections. They'll be able to talk about which consultancies they want to bring in while they look for hires. They'll be able to talk about what threat actors and vulnerability classes they want to handle first. This should all make sense to you, maybe with a little tweaking, but you should be feeling like there's already a plan for the plan.

4. Get that wallet out. This role won't be cheap, even if you go with the "take a chance on someone totally unproven" option. The security roles that they're then going to hire won't be cheap. The consulting time for fill-in in the meantime or burst capacity won't be cheap. You'll have to over-allocate recruiting resources to find talent to get this person and then for every role they want to staff, at least in the beginning. 

5. Keep that wallet out. This security team is going to drive fix cycles into your engineering teams. You expect that. It's going to drive maturity and break/fix cycles into your IT and OPS teams too. It's going to complicate things with Legal a bit as it adds some more process around how they do things. It's going to cause you a non-trivial, but hopefully correct and rational overhead increase across several cost centers. Your ability to ramp into that increase is going to be one of the greatest speed limits on how quickly the security team functionality grows and how fast it can improve your security posture. If you aren't willing to hire another IT admin to run the stuff the security team wants to use to protect you, then why are you hiring the security people?

But what does all that get you?

Security is a risk management spend where you are, hopefully, exchanging fewer dollars to prevent a bad outcome than the bad outcome itself will cost. It's not going to generate a lot of "bringing in a security team got us THIS!" showcase material. Security done right drives maturity across your organization. It improves processes and makes a lot of murky things predictable. It prevents abuse of your customer's information by persons unknown, your partners, and your own employees. If customer confidence matters, it becomes a core tenant of your whole company's competency. For a lot of you though, right now you probably just lump this all into the bucket labeled "existential threats" where you keep "earthquake" and "CEO gets hit by MUNI" If that's the case, then this still isn't for you. It probably should be, but you haven't gotten nervous enough about what's hiding in that bucket yet. That's going to make it harder to fix when you do, but that's another blog post I'll probably never write.

When you're ready to start tackling the things in that bucket, do it right. Security & privacy of information may not drive sales directly, but your customers' trust is your responsibility, and you need to deal with it honestly, earnestly, and without cynicism.

]]>
tag:graham.posthaven.com,2013:Post/824484 2015-03-14T16:09:22Z 2015-03-14T16:09:23Z Security Architecture work in the modern fullstack era

]]>
tag:graham.posthaven.com,2013:Post/675773 2014-04-10T16:10:27Z 2015-02-24T03:18:36Z Heartbleed

For Muggles:

Are you an Internet user bewildered by all this noise about "heartbleed" and "resets"?
That's cool, ain't everyone gotta be a nerd. Buncha websites have probably been leaking passwords (and other security token type things) for a while. There's no good proof (YET) that anyone has really been attacking this though. Changing sensitive passwords after the sites fix themselves is probably a good idea. Panicing is probably not warranted.

OMG I WENT TO THIS TESTING SITE AND IT SAID
Woah woah woah. A bunch of websites have thrown up "see if your favorite website is vulnerable to heartbleed" test pages. At least some of the popular ones are provably wrong. Remember, everybody selling something and some of them are selling you panic.

Be careful about password reset emails
As always, people who like to trick you out of money or sensitive information love to pivot off hot news events. If you get a password reset email from a service talking about heartbleed (or just in general) and you didn't do something to trigger the email, don't click. Go to the actual service site and reset your password there.

For Wizards:

So you're running a SSL/TLS site or mail gateway?
No? Well relax.
Yes? Keep reading

Was it on the OpenSSL 0.9.8 branch? 
OK, do nothing*

Wait what about hardware load balancer TLS termination?
Well, check and make sure that the TLS stack on it wasn't vulnerable of course, but your'e probably OK. From what I've seen all the "firmware" implementations were fine but some of the "doing TLS in software on the loadblancer" implementations may have been vulnerable.** 

Was it using something else server-side to terminate TLS that isn't OpenSSL?
Well put your feet up and enjoy not being in the fire for once.

Was it on the OpenSSL 1.0.1 branch?
Cool, you've got work to do:
  • Recompile with heartbeat disabled or upgrade to latest OpenSSL right now
  • Look up on your CA's site about how to regenerate your certificates, each has their own proper workflow
  • Regenerate your certs and use a new private key to do so (your old private key may have been leaked)
  • Doing so hopefully revoked your old certs. If not forcibly do so.***
  • Invalidate all server-side session tokens in the most expedient manner. Yes, even if you have 2FA for your service.
  • Force a password reset for your users
    Is your OpenSSL server not actually important to anyone but your cat and just a random TLS server for [reasons][which are probably cat pictures]?

    That's cool, turn off your HTTPS listener till you have a chance to patch.

    I ship client applications that connect to arbitrary websites/mail systems and use OpenSSL libraries!

    Well either you dynamically link and better hope the OSes update soon, or you statically link and you should recompile and push updates.



    *Consider updating to 1.0.1 after it becomes stable again so you can improve your TLS support.

    **Yes, yes, as always, the terms hardware, firmware, and software don't apply well to appliances (it's all software, dude, have you seen my klout score on hacker news?!?!) and you can be a wonderful pedant about them, but if you're reading this you know exactly what I mean.

    *** This barely even matters because cert revocation status checking is pretty much a broken process outside of OSCP stapling which (i think) only Opera even supports.

    ]]>
    tag:graham.posthaven.com,2013:Post/603734 2013-09-23T20:11:14Z 2022-01-16T16:08:17Z The bay area dot jay peg

    ]]>
    tag:graham.posthaven.com,2013:Post/593660 2013-08-08T23:14:25Z 2020-06-11T22:45:42Z Comments

    Comments will never be enabled.

    If you feel that comments are necessary to tell me that I'm wrong about something - I refer you to the central premise of this site: the things I am saying here are wrong.

    If you feel this limits your free speech, you're wrong. Lots of things are doing that these days though, so you should probably get better informed on the specifics there. I direct you to http://www.popehat.com/free-speech-resources/ Popehat is a great resource for things like this, frequently funny, and almost always too contrarian and deliberately centrist to have political views worth caring about. 

    If you feel that you need to speak to me directly, I encourage that. If you found this, then you can find me elsewhere on the internet easily. Twitter is generally preferable. 

    ]]>