Rockstar shopping is inherently unsafe

I have cleaned up the intro some.

This is the (very) long form of a speech I've given to a few startups who have come calling. Sometimes they listen. My observation has been that the ones that don't listen end up repeating the process a while later. If you're working on a product where security or privacy is a feature, hopefully none of this applies to you because you're already making these decisions correctly. If you're working on a product where security is an emerging need, or you've just realized that it should have been a core feature, I hope you get some use from this.

Someone just said "we really need to hire somebody to handle all this security stuff" out loud in a meeting. 


Maybe it's because you're passing 70 engineers and not having a security person seems wrong. Maybe you're at 200 because security/privacy wasn't a core component of what you're making (maybe you should have challenged that assumption earlier). Maybe you're getting 14 emails a day from security researchers about how you do something. Regardless, you have decided it's time to start thinking about security as an essential part of the product and as a role in the organization. 

You had another quick meeting with recruiting, the CTO, and someone from OPS, and tried to figure out all the things that needed to get done. Now HR has put a job posting out that reads like a list of security function buzzwords. It includes things that are tactical, strategic, high-level, low-level, PR-related, touches every type of technical skill known to humankind, and is designed to not upset any silos in your current org. Maybe this person is going to report to OPS, maybe they'll report to one of the tech leads, or maybe to the CTO. You've got a bunch of old tickets labeled "Security Debt" and you'd like them gone.

You know it's a big list, but you're not an idiot. You've got nuanced understanding of where this role can best achieve what you want done and you've built this out of a matrix of reasonable concerns. Yes, it's a bit of a laundry list and maybe it's not all achievable, but what job posting isn't like that? You'll find someone who can do the important things and get started fixing all this security stuff. At some point you'll need more security people and you'll sort out org structure and management then, but that's not important right now.

Great! That's wonderful. Kudos to you for recognizing the importance of doing this right.

Step 1: Throw that list away. 

I know, we just talked about how it's relevant to your needs. The problem with that list is that you weren't qualified to make it in the first place. You wouldn't expect to hire Your First Marketing Person or Your First Recruiter or any other entirely new position in your company and have a good grasp on what the role should be doing. You'd have things you know you want that position to do, you'd have some rough ideas about what else might be necessary or prudent, but you'd expect that position to fill in a lot of blanks. Instead, since security is a technical role, you think you know most of what it should be doing, and want to hire someone to fix that technical problem set. You're not qualified to triage these problems, and the outcomes of trying to bring in the minimal resources to deal with them, as you understand them, are far more likely to be negative than positive.

What you've done is lay out a subset of the security problem you actually have. If you're really well informed about security, it may be mostly correct. If you think you're well informed but haven't talked to actual security people, you're wrong about that set. If your recruiter tells you they talk to people like me and we say things like "hrm, it sounds like you're making the classic rockstar mistake" then you're definitely wrong, and you're into dangerous waters.

There are four potential outcomes from dedicated rockstar/unicorn shopping:

1. Someone will decide that you don't know they don't exist, they'll glue a carrot to a horse, and you'll buy it

2. You fail to learn the right lessons from being unable to find that rockstar and make a series of "good enough" individual hires

3. You give up and decide to try again "later"

4. You do find a real rockstar, who is possibly willing to handle the ten security domains you need rather than the three they find interesting. 

Let's break those down some. Keep in mind that I'm talking about the negative end of the spectrum here.
1. Buying a fake rockstar is where a lot of startups end up. That's not to say they've hired someone bad at security, they've probably hired someone outstanding who can do a subset of what they wanted and can talk about the rest well enough to get through the hiring process. That hire is probably not a dishonest actor; interviewing is hard and people like to please. You've now got one person who is supposed to solve all your problems. Since your organization thinks those problems are now all eventually solved, that person isn't in a position to drive enough unexpected change or pull in enough resources. You're on a finite timeline to failure. 

2. You decide to try to hire a couple of different security resources to get the coverage you want. Senior people aren't likely to wade into trying to fix part of the security problem without security leadership managing the big picture, so you're going to end up with mostly junior people. This is going to leave you with good talent without enough practical experience or hands-on mentoring, and you're far more likely to end up with your security engineers having a stagnant relationship with the engineering teams and not getting anywhere near enough done. The compliance person you bring in will get absorbed into audit or legal and will, typically, defend their programs instead of the company. This is a fairly normal 'lack of good leadership' pattern and you've probably seen it before.

3. You keep shopping around and never find anyone who can do all the things you want. The typical response here is to decide that the time isn't right and kick the can down the road. Startups usually default to "Well we don't want a bunch of security people; we're doing OK. Let's just wait till another 2M DAU/20 engineers/whatever." The problem with this is that your collective management brain has already realized you're at a risky inflection point. Deciding to not act is action, and it's very high-risk action. If you're still reading this, it's because you recognize the need to act.

4. Maybe you find an actual rockstar. Someone who really is good at securing all the things your network, ops, engineering, legal, and HR teams do as well as running incidents, making long-term risk decisions, and can eventually grow and manage the security team you need. Awesome. It's unlikely, but they do exist; I know a couple. The problem here is that I've just described what's about 3-4 people's worth of work in even a relatively small startup. You're going to fry that person if they try to do all those things all the time. What you think of as "security stuff" is actually an astonishing number of different contexts and modes of thought, and someone constantly flipping in and out of them, even if qualified in all of them, is very taxing. Even finding someone willing to do that constant context switching is harder than you'd think. Being capable at everything in a domain doesn't make all those components interesting, and doing all of it as a one-person-shop isn't much more rewarding than doing the parts you enjoy.
If it seems like I'm saying that the most dangerous result is finding that rockstar, then you are paying attention. It can work out wonderfully, but that's the unusual example of an already rare case. It's the most dangerous result because it's the result that hides the most unmanaged risk away from your eyes. The rest of the results will be obviously negative within a year or two, and much sooner in a good management environment. A bad rockstar hire can take four or five years to be apparently negative. Often when you discover that they built things that became central to your operations are hard to maintain/extend and massive over-solves that consumed far more resources than they should have.

So what the hell should you do then?

Short answer: Abandon the idea of a singular security hire. Reset your expectations to hiring a team across several security sub-disciplines. Hire competent technical security management first. Interview for that role based on solving some of your technical hurdles, security management competence, and management team fit. Expect to hear projections about head count and early spend before the end of the interview process. Expect to pay more than you wanted to.

I know, you're annoyed. Stick with me, and I'll back this up.

Here's a partial sample of your actual security problem sets, current and upcoming:

  • What sensitive information do you have and where does it live?
  • How do people access your infrastructure?
  • Who can access what stored information?
  • How do you know who can authorize that access?
  • What happens when a laptop gets stolen?
  • How are you dealing with external parties that are finding bugs?
  • Who does what when you get hacked? Are you sure?
  • Are you optimizing your public services for privacy?
  • When customer information shows up on the Internet, how will you figure out what happened?
  • How you decide that code and services are safe?
  • How you decide that that code is the code you wanted to run?
  • How do you make patterns, frameworks, and library decisions that will trend towards safe code/services?
  • How are you managing customer payments?
  • How are you managing employee information?
  • How are you handling security questions from customers?
  • What does your compliance roadmap look like? Are you ready for it?
  • How are you deciding what risks to take?
  • How are you deciding how much risk is the right amount for your company?
  • Are those other startups you're sharing information with doing these things right enough?

The first four bullets on that list alone can easily take a year or two of cleanup for startups that are doing things mostly right, conscientiously, and trying very hard to get the security stuff correct. More than a few big names of this decade have cleaned them up only after an FTC Consent Decree craters into their budget. If you look at that list and think "well a bunch of that isn't needed right away" you're totally right. But a lot of it builds on other components, and most of them have groundwork that you don't want to do by playing catchup. 

What you should be hiring for first is expertise in triaging the security problems versus realistic threats, and that's the security management layer. The closest analogy you'll find in the engineering world is that of CTO vs Leads. You can hire a buncha lead engineers first, but every one you hire before the CTO is going to make hiring that CTO harder, and going to make their job exponentially harder, and going to risk deeper and more difficult tech debt for them to claw back out from under.

Right now you might be thinking that I'm telling you to immediately boil the security ocean.

No, I'm not. I'm pointing out that once you decide to solve for security drivers, you're already way past the point where you can make self-informed decisions about prioritizing the the problem space. I may know when my car is being weird, but I don't have a clue how to fix it. 

You, like almost every other startup ever, transferred the vast majority of the risk of security problems to your customers. You talked about how you take information security seriously while dedicating nearly no time at all to it, deliberately not hiring for it, and occasionally saying things like "we could bring in someone to handle this but they'll slow us down too much." That's just the way it is. This is VC-based capitalism, and you have to prove success where essential while faking success everywhere else. In any specific "I can't believe company X did that" discussion it's almost always indefensible, but in general, it's the cost of doing business with other people's money. That only works for so long though, and now you're past that point. The second you're past it, you're into dangerous territory.

Even if you didn't have those pressures, no startup builds a perfect product and then releases it. You're building a boat while bailing water out to keep it from sinking. The boat has to get larger and progressively more complex without being disassembled. At some point the boat inspectors will be showing up. Also, if your boat is about important or secret things, people are going to try to break in and steal them. It's a series of devil's bargains, and you have to do all this without perfect knowledge of the forces at work internally and externally. 

I don't expect you to begin solving every security problem you'll ever have. I expect you to hire someone who can predict what those are going to be, and build to solve them within your process of building the company and product.

How do you make the right hire?

How do you pull this off? It's tough, and you've got some tradeoff decisions to make. Here's what you need to have generally agreed upon before you try to make this hire:
1. Determine your level of urgency. There's a few ways you can go. If you want someone who has done this before but isn't looking exclusively for a CSO role as their next challenge, then you're shopping in a VERY small market. If you want that to include just nice/good to work with people, you're shopping in a microscopic one. If you're willing to go with some proven management experience, well-rounded security experience, and startup experience, then you have a reasonable (albeit still small) market to shop in. You're going to find way more chaff than wheat, but there's probably enough wheat out there at any one time that you can find a good hire. If you're willing to take a chance on a senior security person who hasn't done management before, you can do dozens of interviews every week. That's probably your best way to pull this off without it being very expensive because both the other options will bring tears to your eyes.

2. Understand where you want this role to end up. You're hiring someone in the middle of a career arc for a role that may or may not have its own arc internally at your company. You should know what you expect the role to grow into or when you'll expect the role hire its own boss. This is something that should be frankly discussed in later interviews. Are you already sure your board will insist that an outside CSO be brought in at some point? Is the candidate not comfortable with the amount of compliance wrangling necessary to be the top chair in security when it's time to go public? Are they OK handling a good bit of the IC work while they find other resources for the roles they want to hire? Do they know which roles they can handle and which they can really own and which they shouldn't touch? Understand what you want in general from someone who is going to build an internal security practice, and be sure they do too.

3. Hire based on both fit and plan. You've hired management roles before so that hopefully isn't a black box to you. Evaluate candidates based on both their fit in your management team and their default plan. The discovery work and basic staffing is going to be something they can largely predict by the end of the interview cycle. They'll be able to quote you year one and two opex and capex projections. They'll be able to talk about which consultancies they want to bring in while they look for hires. They'll be able to talk about what threat actors and vulnerability classes they want to handle first. This should all make sense to you, maybe with a little tweaking, but you should be feeling like there's already a plan for the plan.

4. Get that wallet out. This role won't be cheap, even if you go with the "take a chance on someone totally unproven" option. The security roles that they're then going to hire won't be cheap. The consulting time for fill-in in the meantime or burst capacity won't be cheap. You'll have to over-allocate recruiting resources to find talent to get this person and then for every role they want to staff, at least in the beginning. 

5. Keep that wallet out. This security team is going to drive fix cycles into your engineering teams. You expect that. It's going to drive maturity and break/fix cycles into your IT and OPS teams too. It's going to complicate things with Legal a bit as it adds some more process around how they do things. It's going to cause you a non-trivial, but hopefully correct and rational overhead increase across several cost centers. Your ability to ramp into that increase is going to be one of the greatest speed limits on how quickly the security team functionality grows and how fast it can improve your security posture. If you aren't willing to hire another IT admin to run the stuff the security team wants to use to protect you, then why are you hiring the security people?

But what does all that get you?

Security is a risk management spend where you are, hopefully, exchanging fewer dollars to prevent a bad outcome than the bad outcome itself will cost. It's not going to generate a lot of "bringing in a security team got us THIS!" showcase material. Security done right drives maturity across your organization. It improves processes and makes a lot of murky things predictable. It prevents abuse of your customer's information by persons unknown, your partners, and your own employees. If customer confidence matters, it becomes a core tenant of your whole company's competency. For a lot of you though, right now you probably just lump this all into the bucket labeled "existential threats" where you keep "earthquake" and "CEO gets hit by MUNI" If that's the case, then this still isn't for you. It probably should be, but you haven't gotten nervous enough about what's hiding in that bucket yet. That's going to make it harder to fix when you do, but that's another blog post I'll probably never write.

When you're ready to start tackling the things in that bucket, do it right. Security & privacy of information may not drive sales directly, but your customers' trust is your responsibility, and you need to deal with it honestly, earnestly, and without cynicism.