tag:graham.posthaven.com,2013:/posts Graham says wrong things 2018-08-20T04:37:49Z tag:graham.posthaven.com,2013:Post/1313457 2018-08-20T04:31:31Z 2018-08-20T04:37:49Z Mastodon

I'm not really on twitter anymore. I've got the account and will get notifications about DMs, so you can still reach me (mostly) privately there. If for some reason you wish to consume my maximally trivial takes, I'm active on Mastodon at https://mastodon.social/@grahamvsworld

I'm not going to tell you that you should stop using Twitter and start using Mastodon. I am going to suggest to you that you should decide if the executive direction of Twitter of late makes for a product you wish to actively support.

]]>
tag:graham.posthaven.com,2013:Post/1238855 2018-03-29T22:37:34Z 2018-03-29T22:37:34Z Security is mostly an accountability function

This is one of those "I made this post because I keep explaining the same small concept repeatedly" things. Both to save time repeating it, and (mostly) to try to clean up the concept.


Companies, decide how secure they want to be twice*

The first time is when the exec team decides, in their own terms, just how much security they want to purchase. That purchase being made in people, software, process costs, and especially meta-costs paid by every other org/effort in the company. The second time is when the exec team decides how much they want to deviate from the previous decision just this one time for this particularly important thing. That second decision gets remade over and over, however many times necessary. You (hopefully) have a security executive because that first decision only matters if someone at the exec tier is holding their peers accountable for their strategic intents. A quick, naive primer for multi-level exec politics:

  • Bosses hold reports accountable for failing to hit goals (your security exec yells at you from here)
  • Peers hold peers accountable for failing to meet shared commitments (your security exec does most of their job here, w/ other execs)
  • Reports hold bosses accountable for failing to provide resources (your security exec does their job here, once or twice a year, with their boss. You do your job with your security exec here on a frequent basis)

In a perfectly aligned company, where no one ever tries to shirk parts of the agreed-to big picture that are temporarily inconvenient, the security team could define what the risk looks like, the execs (or more likely, delegates) could continuously make decisions aligned with their security goals, and everyone would move forward together. Nobody actually does that. So let's dismiss that potential out of hand for at least another decade or so. 

That all means that a well functioning security team & security executive are, from the perspective of the rest of your company, largely an accountability minder. Their role is to point out things that damage that first central security decision and say "you're lying to yourself if you think you can do this without it damaging your other goals" or to drive requirements into the rest of the execs/orgs that are foundational, required elements of meeting the shared security goal. You are rarely actually arguing with other people about the specific topic. You're arguing with them about meeting their commitment to the overall shared security goal.

Constantly re-centering conversations to given risk appetites is incredibly useful. Generally, with honest actors, it's fantastic -- establishing "wait this was our goal, right?" with people who are dealing with you on the level is amazing. Building this habit into your security engineers will proactively prevent a noticeable percentage of escalations, and will also help your junior security engineers not make requests of partners that aren't reasonable for those shared goals. (There's a much latter stage maturity version of this where you've built an agreed-to method for estimating risk and people are generally OK with your impacts on their timelines because they know you warning them off impacting their risk budget here means they get to do something more risky later. Don't rush to this as a goal. If you get there way before the rest of the company is ready to think about these things big-picture, you will accidentally make security a 1000lb gorilla in every "a or b" decision conversation, and you'll strangle the company's ability to take informed guesses at risk decisions -- which is what it absolutely should still be doing early on.)

Unfortunately, it's also often less useful than you'd hope. You'll frequently run into people who habitually think they're the only pragmatist in the conversation. You'll run into people who discount security goals out of years of poor experiences with them. You'll also run into people acting with direct malicious intent, but they'll be indistinguishable from the "only I think pragmatically" type unless they're extremely bad at it. If you've got a choice, it's much more efficient to just choose to not work with assholes (or people who accidentally approximate assholes) who put their personal goals well out ahead of company goals, but it is an imperfect world.

If you're living in this particular intersection of startup and maturity and (hopefully un-)intentional bad actors, it can be very difficult; even demoralizing. Guiding conversations back to common security goals will seem like it isn't helpful sometimes. It still is. Even when it isn't useful for the listener, it is at least putting your head in the right place for the argument. Make it a habit. Don't make it your only one of course. If you're having these types of discussions, you need an bag full of deescalation and empathy strategies to keep a line of communication open, even if the net result of them is "we can respect each other's position, but I'm hard blocking you." Security engagement has a non-trivial number of typical conversations that end with one of the parties extremely dissatisfied. At worst, re-centering on shared goals will at least help keep it impersonal. If you find that you're using multiple conversation strategies successfully, including this one, and still have people who refuse to accept the shared strategy - it's time to decide if you're fighting an independent bad actor, or if that person is doing what the company's culture has shaped them to do. Depending on the answer, you either help them find somewhere else to work, or do so yourself. Sometimes the right answer may be that you have to change the core culture of the company to do your job, but that is a much trickier proposition.




*Companies decide, in the trenches, how secure they want to be a dozens times a day, and inside the security team's prioritization decisions, non-stop.

]]>
tag:graham.posthaven.com,2013:Post/1185086 2017-08-20T06:10:38Z 2017-08-20T06:10:38Z What every computer science major should know

Gatekeeping is for assholes and you shouldn't do it. 

]]>
tag:graham.posthaven.com,2013:Post/1113487 2016-12-07T05:49:46Z 2018-04-10T17:39:19Z Effective Line Management

This was originally a poorly formatted word doc created when some management peers asked for tips at being effective at managing their reports. A little bit (but not much) tuning later, it's now this. I make no pretense that this is my own work, it's all received wisdom from a ton of different people and sources. 

You just realized you have no idea why your employees care what you think.

Someone put you in charge. Maybe you demanded to be in charge because you kept telling everyone what to do anyway. Maybe you got stuck with it when the last person in charge bailed and you looked like you knew what you were doing. Regardless, you now have humans expecting you to make what they do and why they do it make sense. INSIDE THEIR HEADS. Welcome to management!

A non-made up, very real statistic: Depending on study, between two thirds and three fourths of employees leave their job because of their direct manager. The odds are that over the course of your career, you will be the cause of several very important employees quitting their jobs. Every time you prevent that from happening by better management you're saving that employer a ton of cash, yourself a lot of frustration, and justifying your own existence at that employer. Remember: most management is overhead; only the very best of management starts to add value again. Only people that add value to their teams should be trusted with larger teams, more complex teams, and eventually executive positions. We all know that the valley isn't really a meritocracy, but that doesn't mean we exist outside accountability either.

Hands-on line management is made up of several things:

  • How to cause work to be done at your company
  • What motivates people
  • What prevents execution
  • How to model someone’s behavior so you understand the prior two

Things we're not going to cover here include: budget, pipeline, managing up, politics, managing the HR dept, and many other important facets of being good at managing people.

Causing work to be done

This is mostly company-specific, with a bunch of generalities that are nearly platitudes. You catch more flies with honey than vinegar, etc. Be nice and professional to people, stand your ground when you're right, insist on being treated the way you treat others (and vice-versa), and generally treat people like adults until they convincingly prove they aren't. There's also a whole host of other strategies that are only viable in dysfunctional environments. You should generally learn those as needed and not by deliberate instruction, as they're all disruptive in an environment that can still function positively.

My guiding principle in this, as in all things, is that you should deliberately reject and act away from cynicism whenever you find it. Cynicism is much bigger than just acting or talking cynically. A bureaucratic process where everyone involved knows they aren’t solving a problem is cynical. An approval loop where you don’t actually consider what is being approved and how it changes things is cynical. A promotion process that defines goals that have nothing to do with performance or aptitude is cynical. Cynicism burns people out, it causes bad decisions, and it will make good employees leave. If you spend your time as a manager doing nothing but finding and fighting cynicism, you would have done more good than many achieve.

The Basics of motivation, execution, and behavior

Let’s start with the books

This is your first-tier homework. Hopefully you’ve got a fair number of “how to manage” opinions from your own experience, quite possibly unorganized. With this you begin to make those more clear and to organize them usefully. That said, management books are a lot like self-help books. You may find something that resonates for you, but that’s more likely to be a unique emotional trigger than it is to be something that works for everyone. People are complicated. If there’s any one rule of thumb here, it’s that: people are complicated. The goal here is to give you a shotgun blast of receivable wisdom, and then a dive into the areas that drive human interaction. The most useful manager is one who can understand why someone is going to react the way they will before they know they will.

Managing Humans - Michael Lopp (get the most recent edition, currently 3rd)

This is a series of bite-sized hard-learned lessons, mostly as personal post-mortems for a management interaction that went badly. This book is a chance to learn from (a very smart) someone else’s mistakes. This is an incredibly central work in tech management.

How to Measure Anything - Douglas W Hubbard

This is a strong foundational intro to modern decision science. A major component of management is how to make rational, repeatable decisions based on rational, repeatable estimates. If I could force everyone in a company to read a book, it would probably be this one.

Other suggested topics and works

Decision Theory, Statistical Analysis, Communist Manifesto (only the first half), Capitalism and Freedom by Friedman, pretty much anything in recent macro or micro economics. Every time you read through a few books on a new topic, I suggest you re-skim Managing Humans for grounding. The point of these areas is that you need to be focusing on increasing the depth and complexity of your understanding of certain very complex subjects, namely, group dynamics, how bias impacts decision making, how to make decisions repeatable, how economics impacts your employees and their place in society, how your personal economics lead to different decisions than they would make. The point of all this is to keep your gut feelings connected to the realities that your employees live in, so they maintain confidence in your empathic relationship with them. This is why they will trust you to make good decisions and treat them fairly. Without that why, the only tools avaliable to you are conditioned responses to negative/positive stimuli. That's not where you want to be.

Additionally, you may want to read through some of the traditional/popular “management” books in order to be able to communicate more effectively with management peers and your superiors. This is a dangerous road though, so go into it with an extremely critical eye and an assumption that you’re learning the lingo, not accepting the arguments. You may find arguments you like, and by all means, incorporate them. Just don’t do so solely because these books are popular. Here’s a brief list of such popular works:

  • Who Moved My Cheese? – Johnson
  • Peopleware: Productive Projects and Teams – DeMarco
  • First, Break All the Rules – Buckingham & Coffman
  • Slack Getting Past Burnout… - DeMarco
  • Turn the Ship Around! – Marquet
  • The New One Minute Manager – Blanchard & Spencer

People

As a manager you have six primary things to worry about with regards to your humans:

  1. Is there something preventing them from executing on the task?
  2. Do you understand how to motivate them day-to-day?
  3. Do you understand how to motivate them long-term?
  4. Are you getting better at managing them?
  5. Can you answer all of these questions about how you handle yourself?
  6. Can you answer all these questions about your boss handles you?

Let's break these down

Is there something preventing your employee from executing on the task?

The possible reasons are:

Knowledge Gap - A knowledge gap means they don't know what to do or how to do it. You can solve this with training, education, hands-on mentoring, conferences, books, etc. If the knowledge gap is extensive, you may need a long-term plan here. This is a gap that you identify mostly by building trust with your employees. If they aren’t afraid to say “I don’t know how to do this” you’ll rarely be burned by it.

Capability Gap - A capability gap means that they know how to do it, but they can’t pull it off. There’s several possible reasons for this (execution anxiety, stress, it’s too complex for their experience level, they just aren’t smart enough on their own for this problem). The fix depends on the cause. You may need to team up with them on this, you may need to partner them on this; you may need to take tasks away from them based on this.

Motivation gap - They don't want to do it, or they don't want to do it to the necessary standard. This is the hardest to solve, but we have an entire section here on motivation. Let’s get to that.

Overall Employee Motivation

This is totally different from how to keep employees motivated day-to-day. We’ve got a whole section for that after this one. This is how you need to keep employees motivated on a big-picture, long term basis. You will eventually need to build and maintain a personal model for what motivates each employee or coworker. To start with though, you should stick to the classic reasons and then modify from there based on your interactions.

If you ask ten different managers, you’ll get ten different responses as to how they model motivation. I prefer to try to stick to Maslow’s Model of Hierarchal Needs and the 4-Drive Model. The 4-Drive is fairly new, and adheres well to the motivations you see everyday in tech and sales. Maslow’s is a classic, tried and generally still true (there’s some debate about whether “internet access” needs its own tier). Neither of these are absolutely correct. They’re models that you should always have to hand to think about why someone is reacting a certain way, or how you might try to predict their reactions to change. Explaining these in depth is well beyond the scope of this document, ample resources exist elsewhere. While I won’t say that you should only pay attention to management texts that reference them, be very critical of management texts that don’t reference well-understood engagement/motivation models.

Maslow’s


Maslow’s model can basically be thought of as “people don’t care about the higher-layer needs until the lower-layer needs are met.” This is probably obvious to you, but the model breaks it down well, and it’s a useful touch point. When someone is furious and frustrated and all but yelling at you in a one-on-one, having clear models close to hand is very useful. Think about the interaction you’re having and realize that if your motivations for that person do not resonate to the layer of need that is upset with you, then those motivations cannot affect the behavior.

4-Drive

4-Drive is a model of why people do things, assuming that there are four general reasons for action/motivation. People are rarely so simple as for only one of these reasons to apply at once, but if you can try to think of someone’s personality as a group of people with their own drives, it’s very useful to guess which of these reasons is driving the behavior you’re wanting to change or to reinforce.

As with most other things about managing your team, you need to understand how these things apply to you even more than you understand how they apply to them.

Keeping Employees Motivated Day-to-Day

Keeping your employees connected to their work requires first that you keep them generally motivated, then that you keep them specifically enthused about either the challenge, what the challenge generates for them, or how their performance interacts with their environment.  There’s no one solve for this, but rather an environment of good management will lead to day-to-day motivated employees. Maintain this through a constant lifecycle of positive actions:

Be a good coach

  • Provide specific and timely feedback
  • Balance positive and negative feedback
  • Understand unique strengths/development areas of each team member
  • Tailor coaching to the individual & situation
  • Suggest solutions
  • Have regular 1:1s

Empower execution

  • Do not micromanage
  • Balance giving freedom with being available for advice
  • Make it clear you trust the team
  • Advocate for team with others outside the team

Be a decent human being

  • Genuinely care
  • Be fair
  • Make new members feel welcome
  • Show support in the good and bad times

Be productive and results-orientated

  • Keep the team focused on the results/deliverables
  • Help the team prioritize
  • Remove roadblocks
  • Be clear about who owns what
  • Be a hard worker

Be a good communicator

  • Encourage open dialogue
  • Be available for the team
  • Explain the context
  • Tell the truth - even when the news is bad
  • Be calm under pressure
  • Listen to each team member

Help with career development

  • Provide honest, specific feedback on the next step in team members' career
  • Help team members find new opportunities
  • Talk about all aspects of career development, not just promotions
  • Balance their team members’ and company’s needs

Have a clear vision for the team and a strategy to achieve it

  • Create a compelling vision/strategy
  • Clearly communicate vision/strategy
  • Involve team in setting vision/strategy
  • Build relationships with others to help the team achieve their goals

Have and use skills relevant to the team

  • Roll up sleeves and actually conduct work
  • Understand the challenges of the work
  • Help solve problems based on technical skills

Other important factors

The rest of this is basically “extra credit,” it isn’t something that you need to be focused on to start being a good manager, but it’s extremely useful. If you find yourself feeling that you’re plateauing as a manager or in your relationships with your people or peers, look here.

Mindfulness

Clarity of thought and purpose is necessary for really strong management. We’ve all run into an executive who seems to have their shit incredibly together every time they’re asking you something. That kind of clarity just comes to some people. Most of us have to work very hard at it. Mindfulness and meditation are a strong, wide pathway to that clarity. If you find yourself wishing you had more of it, I encourage you to explore both concepts.

Experimentation

Don’t be afraid to run non-damaging experiments with how your people behave. Need someone to get better at telling you when they’re too busy? Start slowly overloading them with stuff that it’s OK if something gets dropped. Need someone to give you better feedback. Illustrate the problem for them with someone else, then push them a little till they are actually kinda mad at you. Use this to help them understand how feedback matters and when they should know to give it to you. Hopefully you get enough accidental experimentation through the vagaries of life in the workplace that you don’t need to intentionally push people for effect, but if not, do so thoughtfully and ethically. People respect being taught careful lessons.

Therapy

Yes, I am literally telling you to go to therapy. You go to the doctor to make sure your body is working to your expectations. Therapy is exactly the same thing for your mind. In most cases, improving your capabilities as a human improves your capabilities as both a manager and an employee. This is both separate from mindfulness and complimentary to it.

Etcetera

This is not a complete or exhaustive list of what’s required to be a good, or even an OK manager. It’s a primer for where to start, with some pointers on where to go next. For yourself, everyone on your team, everyone you deal with regularly, and your boss, you must always have an expectation of normal behavior, a model for how to engage with and motivate them, and an understanding of what forms of feedback are productive. You also need to still get work done. It’s hard, but ultimately if management is for you, it’s probably the most rewarding thing you can do.

]]>
tag:graham.posthaven.com,2013:Post/892769 2015-10-30T00:27:45Z 2018-04-12T12:40:13Z Rockstar shopping is inherently unsafe

This used to be called "unicorn shopping is inherently unsafe" but all the recent hullabaloo around unicorn startups and the pre-market valuation bubble has poisoned that already pretty murky term. So let's just say "rockstar" for now I guess.

This is the (very) long form of a speech I've given to a few startups who have come calling. Sometimes they listen. My observation has been that the ones that don't listen end up repeating the process a while later. If you're working on a product where security or privacy is a feature, hopefully none of this applies to you because you're already making these decisions correctly. If you're working on a product where security is an emerging need, or you've just realized that it should have been a core feature, I hope you get some use from reading what I like to call:

Someone just said "we really need to hire somebody to handle all this security stuff" out loud in a meeting. 

Maybe it's because you're passing 70 engineers and not having a security person seems wrong. Maybe you're at 200 because security/privacy wasn't a core component of what you're making (maybe you should have challenged that assumption earlier). Maybe you're getting 14 emails a day from security researchers about how you do something. Regardless, you have decided it's time to start thinking about security as an essential part of the product and as a role in the organization. 

You had another quick meeting with recruiting, the CTO, and someone from OPS, and tried to figure out all the things that needed to get done. Now HR has put a job posting out that reads like a list of security function buzzwords. It includes things that are tactical, strategic, high-level, low-level, PR-related, touches every type of technical skill known to humankind, and is designed to not upset any silos in your current org. Maybe this person is going to report to OPS, maybe they'll report to one of the tech leads, or maybe to the CTO. You've got a bunch of old tickets labeled "Security Debt" and you'd like them gone.

You know it's a big list, but you're not an idiot. You've got nuanced understanding of where this role can best achieve what you want done and you've built this out of a matrix of reasonable concerns. Yes, it's a bit of a laundry list and maybe it's not all achievable, but what job posting isn't like that? You'll find someone who can do the important things and get started fixing all this security stuff. At some point you'll need more security people and you'll sort out org structure and management then, but that's not important right now.

Great! That's wonderful. Kudos to you for recognizing the importance of doing this right.

Step 1: Throw that list away. 

I know, we just talked about how it's relevant to your needs. The problem with that list is that you weren't qualified to make it in the first place. You wouldn't expect to hire Your First Marketing Person or Your First Recruiter or any other entirely new position in your company and have a good grasp on what the role should be doing. You'd have things you know you want that position to do, you'd have some rough ideas about what else might be necessary or prudent, but you'd expect that position to fill in a lot of blanks. Instead, since security is a technical role, you think you know most of what it should be doing, and want to hire someone to fix that technical problem set. You're not qualified to triage these problems, and the outcomes of trying to bring in the minimal resources to deal with them, as you understand them, are far more likely to be negative than positive.

What you've done is lay out a subset of the security problem you actually have. If you're really well informed about security, it may be mostly correct. If you think you're well informed but haven't talked to actual security people, you're wrong about that set. If your recruiter tells you they talk to people like me and we say things like "hrm, it sounds like you're making the classic rockstar mistake" then you're definitely wrong, and you're into dangerous waters.

There are four potential outcomes from dedicated rockstar/unicorn shopping:

1. Someone will decide that you don't know they don't exist, they'll glue a carrot to a horse, and you'll buy it

2. You fail to learn the right lessons from being unable to find that rockstar and make a series of "good enough" individual hires

3. You give up and decide to try again "later"

4. You do find a real rockstar, who is possibly willing to handle the ten security domains you need rather than the three they find interesting. 

Let's break those down some. Keep in mind that I'm talking about the negative end of the spectrum here.
1. Buying a fake rockstar is where a lot of startups end up. That's not to say they've hired someone bad at security, they've probably hired someone outstanding who can do a subset of what they wanted and can talk about the rest well enough to get through the hiring process. That hire is probably not a dishonest actor; interviewing is hard and people like to please. You've now got one person who is supposed to solve all your problems. Since your organization thinks those problems are now all eventually solved, that person isn't in a position to drive enough unexpected change or pull in enough resources. You're on a finite timeline to failure. 

2. You decide to try to hire a couple of different security resources to get the coverage you want. Senior people aren't likely to wade into trying to fix part of the security problem without security leadership managing the big picture, so you're going to end up with mostly junior people. This is going to leave you with good talent without enough practical experience or hands-on mentoring, and you're far more likely to end up with your security engineers having a stagnant relationship with the engineering teams and not getting anywhere near enough done. The compliance person you bring in will get absorbed into audit or legal and will, typically, defend their programs instead of the company. This is a fairly normal 'lack of good leadership' pattern and you've probably seen it before.

3. You keep shopping around and never find anyone who can do all the things you want. The typical response here is to decide that the time isn't right and kick the can down the road. Startups usually default to "Well we don't want a bunch of security people; we're doing OK. Let's just wait till another 2M DAU/20 engineers/whatever." The problem with this is that your collective management brain has already realized you're at a risky inflection point. Deciding to not act is action, and it's very high-risk action. If you're still reading this, it's because you recognize the need to act.

4. Maybe you find an actual rockstar. Someone who really is good at securing all the things your network, ops, engineering, legal, and HR teams do as well as running incidents, making long-term risk decisions, and can eventually grow and manage the security team you need. Awesome. It's unlikely, but they do exist; I know a couple. The problem here is that I've just described what's about 3-4 people's worth of work in even a relatively small startup. You're going to fry that person if they try to do all those things all the time. What you think of as "security stuff" is actually an astonishing number of different contexts and modes of thought, and someone constantly flipping in and out of them, even if qualified in all of them, is very taxing. Even finding someone willing to do that constant context switching is harder than you'd think. Being capable at everything in a domain doesn't make all those components interesting, and doing all of it as a one-person-shop isn't much more rewarding than doing the parts you enjoy.
If it seems like I'm saying that the most dangerous result is finding that rockstar, then you are paying attention. It can work out wonderfully, but that's the unusual example of an already rare case. It's the most dangerous result because it's the result that hides the most unmanaged risk away from your eyes. The rest of the results will be obviously negative within a year or two, and much sooner in a good management environment. A bad rockstar hire can take four or five years to be apparently negative. Often when you discover that they built things that became central to your operations are hard to maintain/extend and massive over-solves that consumed far more resources than they should have.

So what the hell should you do then?

Short answer: Abandon the idea of a singular security hire. Reset your expectations to hiring a team across several security sub-disciplines. Hire competent technical security management first. Interview for that role based on solving some of your technical hurdles, security management competence, and management team fit. Expect to hear projections about head count and early spend before the end of the interview process. Expect to pay more than you wanted to.

I know, you're annoyed. Stick with me, and I'll back this up.

Here's a partial sample of your actual security problem sets, current and upcoming:

  • What sensitive information do you have and where does it live?
  • How do people access your infrastructure?
  • Who can access what stored information?
  • How do you know who can authorize that access?
  • What happens when a laptop gets stolen?
  • How are you dealing with external parties that are finding bugs?
  • Who does what when you get hacked? Are you sure?
  • Are you optimizing your public services for privacy?
  • When customer information shows up on the Internet, how will you figure out what happened?
  • How you decide that code and services are safe?
  • How you decide that that code is the code you wanted to run?
  • How do you make patterns, frameworks, and library decisions that will trend towards safe code/services?
  • How are you managing customer payments?
  • How are you managing employee information?
  • How are you handling security questions from customers?
  • What does your compliance roadmap look like? Are you ready for it?
  • How are you deciding what risks to take?
  • How are you deciding how much risk is the right amount for your company?
  • Are those other startups you're sharing information with doing these things right enough?

The first four bullets on that list alone can easily take a year or two of cleanup for startups that are doing things mostly right, conscientiously, and trying very hard to get the security stuff correct. More than a few big names of this decade have cleaned them up only after an FTC Consent Decree craters into their budget. If you look at that list and think "well a bunch of that isn't needed right away" you're totally right. But a lot of it builds on other components, and most of them have groundwork that you don't want to do by playing catchup. 

What you should be hiring for first is expertise in triaging the security problems versus realistic threats, and that's the security management layer. The closest analogy you'll find in the engineering world is that of CTO vs Leads. You can hire a buncha lead engineers first, but every one you hire before the CTO is going to make hiring that CTO harder, and going to make their job exponentially harder, and going to risk deeper and more difficult tech debt for them to claw back out from under.

Right now you might be thinking that I'm telling you to immediately boil the security ocean.

No, I'm not. I'm pointing out that once you decide to solve for security drivers, you're already way past the point where you can make self-informed decisions about prioritizing the the problem space. I may know when my car is being weird, but I don't have a clue how to fix it. 

You, like almost every other startup ever, transferred the vast majority of the risk of security problems to your customers. You talked about how you take information security seriously while dedicating nearly no time at all to it, deliberately not hiring for it, and occasionally saying things like "we could bring in someone to handle this but they'll slow us down too much." That's just the way it is. This is VC-based capitalism, and you have to prove success where essential while faking success everywhere else. In any specific "I can't believe company X did that" discussion it's almost always indefensible, but in general, it's the cost of doing business with other people's money. That only works for so long though, and now you're past that point. The second you're past it, you're into dangerous territory.

Even if you didn't have those pressures, no startup builds a perfect product and then releases it. You're building a boat while bailing water out to keep it from sinking. The boat has to get larger and progressively more complex without being disassembled. At some point the boat inspectors will be showing up. Also, if your boat is about important or secret things, people are going to try to break in and steal them. It's a series of devil's bargains, and you have to do all this without perfect knowledge of the forces at work internally and externally. 

I don't expect you to begin solving every security problem you'll ever have. I expect you to hire someone who can predict what those are going to be, and build to solve them within your process of building the company and product.

How do you make the right hire?

How do you pull this off? It's tough, and you've got some tradeoff decisions to make. Here's what you need to have generally agreed upon before you try to make this hire:
1. Determine your level of urgency. There's a few ways you can go. If you want someone who has done this before but isn't looking exclusively for a CSO role as their next challenge, then you're shopping in a VERY small market. If you want that to include just nice/good to work with people, you're shopping in a microscopic one. If you're willing to go with some proven management experience, well-rounded security experience, and startup experience, then you have a reasonable (albeit still small) market to shop in. You're going to find way more chaff than wheat, but there's probably enough wheat out there at any one time that you can find a good hire. If you're willing to take a chance on a senior security person who hasn't done management before, you can do dozens of interviews every week. That's probably your best way to pull this off without it being very expensive because both the other options will bring tears to your eyes.

2. Understand where you want this role to end up. You're hiring someone in the middle of a career arc for a role that may or may not have its own arc internally at your company. You should know what you expect the role to grow into or when you'll expect the role hire its own boss. This is something that should be frankly discussed in later interviews. Are you already sure your board will insist that an outside CSO be brought in at some point? Is the candidate not comfortable with the amount of compliance wrangling necessary to be the top chair in security when it's time to go public? Are they OK handling a good bit of the IC work while they find other resources for the roles they want to hire? Do they know which roles they can handle and which they can really own and which they shouldn't touch? Understand what you want in general from someone who is going to build an internal security practice, and be sure they do too.

3. Hire based on both fit and plan. You've hired management roles before so that hopefully isn't a black box to you. Evaluate candidates based on both their fit in your management team and their default plan. The discovery work and basic staffing is going to be something they can largely predict by the end of the interview cycle. They'll be able to quote you year one and two opex and capex projections. They'll be able to talk about which consultancies they want to bring in while they look for hires. They'll be able to talk about what threat actors and vulnerability classes they want to handle first. This should all make sense to you, maybe with a little tweaking, but you should be feeling like there's already a plan for the plan.

4. Get that wallet out. This role won't be cheap, even if you go with the "take a chance on someone totally unproven" option. The security roles they're then going to hire won't be cheap. The consulting time for fill in in the meantime or burst capacity won't be cheap. You'll have to over-allocate recruiting resources to find talent to get this person and then for every role they want to staff, at least in the beginning. 

5. Keep that wallet out. This security team is going to drive fix cycles into your engineering teams. You expect that. It's going to drive maturity and break/fix cycles into your IT and OPS teams too. It's going to complicate things with Legal a bit as it adds some more process around how they do things. It's going to cause you a non-trivial, but hopefully correct and rational overhead increase across several cost centers. Your ability to ramp into that increase is going to be one of the greatest speed limits on how quickly the security team grows and how fast it can improve your security posture. If you aren't willing to hire another IT admin to run the stuff the security team wants to use to protect you, then why are you hiring the security people?

But what does all that get you?

Security is a risk management spend where you are, hopefully, exchanging fewer dollars to prevent a bad outcome than the bad outcome itself will cost. It's not going to generate a lot of "bringing in a security team got us THIS!" showcase material. Security done right drives maturity across your organization. It improves processes and makes a lot of murky things predictable. It prevents abuse of your customer's information by persons unknown, your partners, and your own employees. If customer confidence matters, it becomes a core tenant of your whole company's competency. For a lot of you though, right now you probably just lump this all into the bucket labeled "existential threats" where you keep "earthquake" and "CEO gets hit by a Yahoo! Bus!!" If that's the case, then this still isn't for you. It probably should be, but you haven't gotten nervous enough about what's hiding in that bucket yet. That's going to make it harder to fix when you do, but that's another blog post I'll probably never write.

When you're ready to start tackling the things in that bucket, do it right. Security & privacy of information may not drive sales directly, but your customers' trust is your responsibility, and you need to deal with it honestly, earnestly, and without cynicism.

]]>
tag:graham.posthaven.com,2013:Post/824484 2015-03-14T16:09:22Z 2015-03-14T16:09:23Z Security Architecture work in the modern fullstack era

]]>
tag:graham.posthaven.com,2013:Post/675773 2014-04-10T16:10:27Z 2015-02-24T03:18:36Z Heartbleed

For Muggles:

Are you an Internet user bewildered by all this noise about "heartbleed" and "resets"?
That's cool, ain't everyone gotta be a nerd. Buncha websites have probably been leaking passwords (and other security token type things) for a while. There's no good proof (YET) that anyone has really been attacking this though. Changing sensitive passwords after the sites fix themselves is probably a good idea. Panicing is probably not warranted.

OMG I WENT TO THIS TESTING SITE AND IT SAID
Woah woah woah. A bunch of websites have thrown up "see if your favorite website is vulnerable to heartbleed" test pages. At least some of the popular ones are provably wrong. Remember, everybody selling something and some of them are selling you panic.

Be careful about password reset emails
As always, people who like to trick you out of money or sensitive information love to pivot off hot news events. If you get a password reset email from a service talking about heartbleed (or just in general) and you didn't do something to trigger the email, don't click. Go to the actual service site and reset your password there.

For Wizards:

So you're running a SSL/TLS site or mail gateway?
No? Well relax.
Yes? Keep reading

Was it on the OpenSSL 0.9.8 branch? 
OK, do nothing*

Wait what about hardware load balancer TLS termination?
Well, check and make sure that the TLS stack on it wasn't vulnerable of course, but your'e probably OK. From what I've seen all the "firmware" implementations were fine but some of the "doing TLS in software on the loadblancer" implementations may have been vulnerable.** 

Was it using something else server-side to terminate TLS that isn't OpenSSL?
Well put your feet up and enjoy not being in the fire for once.

Was it on the OpenSSL 1.0.1 branch?
Cool, you've got work to do:
  • Recompile with heartbeat disabled or upgrade to latest OpenSSL right now
  • Look up on your CA's site about how to regenerate your certificates, each has their own proper workflow
  • Regenerate your certs and use a new private key to do so (your old private key may have been leaked)
  • Doing so hopefully revoked your old certs. If not forcibly do so.***
  • Invalidate all server-side session tokens in the most expedient manner. Yes, even if you have 2FA for your service.
  • Force a password reset for your users
    Is your OpenSSL server not actually important to anyone but your cat and just a random TLS server for [reasons][which are probably cat pictures]?

    That's cool, turn off your HTTPS listener till you have a chance to patch.

    I ship client applications that connect to arbitrary websites/mail systems and use OpenSSL libraries!

    Well either you dynamically link and better hope the OSes update soon, or you statically link and you should recompile and push updates.



    *Consider updating to 1.0.1 after it becomes stable again so you can improve your TLS support.

    **Yes, yes, as always, the terms hardware, firmware, and software don't apply well to appliances (it's all software, dude, have you seen my klout score on hacker news?!?!) and you can be a wonderful pedant about them, but if you're reading this you know exactly what I mean.

    *** This barely even matters because cert revocation status checking is pretty much a broken process outside of OSCP stapling which (i think) only Opera even supports.

    ]]>
    tag:graham.posthaven.com,2013:Post/603734 2013-09-23T20:11:14Z 2013-10-08T17:30:28Z The bay area dot jay peg

    ]]>
    tag:graham.posthaven.com,2013:Post/593660 2013-08-08T23:14:25Z 2014-09-25T04:49:12Z Comments

    Comments will never be enabled.

    If you feel that comments are necessary to tell me that I'm wrong about something - I refer you to the central premise of this site: the things I am saying here are wrong.

    If you feel this limits your free speech, you're wrong. Lots of things are doing that these days though, so you should probably get better informed on the specifics there. I direct you to http://www.popehat.com/free-speech-resources/ Popehat is a great resource for a lot of things and often pretty funny and almost always incredibly obnoxious.

    If you feel that you need to speak to me directly, I encourage that. If you found this, then you can find me elsewhere on the internet easily. Twitter is generally preferable. 

    ]]>