How secure is your security program? The rampant issues of unauthorized access, use, and modification of customer data require a security program to further secure and protect this information. In this episode of Friendly Fire, Sean Cassidy, the Head of Security at Asana, sits with Matt to share his insights on how you can build a secure security program. A common failure from other companies is that the security team has to triage and fix all the bugs but does not have enough security engineers. Sean turns this around. Although security and engineering do not always have a harmonious relationship, Sean has built a development process and culture that helps his team work together. If you want to hear more about how Sean built a security team to build a secure security program, tune in to this episode of Friendly Fire!
—
Listen to the podcast here
Building A Secure Security Program With Sean Cassidy, Head Of Security At Asana
In this show, we will be bringing you the top experts in the industry for a chat about anything that’s interesting in keeping our world secure. Speaking of interesting and keeping things secure, we are excited to welcome Sean Cassidy to the show. Sean is the Head of Security at Asana with many years in security, including leadership positions in a couple of companies like Patreon and Defense Storm.
—
Sean, welcome to the show.
Thanks for having me on.
First question, because I had to do some research. Asana is a company that everyone may not have heard of, but it has probably been impacted by your work. Before we dig into it, do you mind telling everybody a little bit more about that to get some context of what it is you’re doing and why it’s relevant to that?
Asana is a work management platform for teams. The example I like to use is you often need to work with your fellow people at your company or you might have a project you’re working on. How do you do that? You send them an email or Slack message. How do you follow up on that? How do you make sure that that’s done? You send them another email or Slack message.
Wouldn’t it be great if you sent them a task, fit it into a project, and they could see that it laddered up to your company’s goals? It was like, “This is all part of the plan. It’s not marketing sending me something, asking me, or interrupting my important work, but it actually is important to the company that I do this thing.” That’s the thing that enables us. It’s how you know what work is being done by who and when.
Now we can get to the hero with the story doing all the exciting things. It’s why we bought the ticket. In our initial conversation about that, you said one of the big things you guys do well is updating third-party dependencies. That’s what a lot of people don’t pay enough attention to. Why we see a lot of the breaches that happen is because there’s the interactive stuff that needs to be happening among all of the stuff that’s in there. Explain to me and the audience why that is so important and where that factors in considering insider threats?
This is something I’m definitely proud of our team for doing. Everybody’s got a big vulnerability feed for their third-party dependencies. It’s something like, “Libxml or something is out of date. CVSS score at 7.8.” A common failure mode I see companies do is the security team has to triage and then sometimes even fix all of these bugs.
There are not enough security engineers at your company to do this, so what happens is you only fix the absolutely most severe bugs and you let all of that medium-low priority stuff fall by the wayside. That’s a big risk because of medium and low bugs, you can chain vulnerabilities together and then turn those into a pre-major and exploit them.
All of our industry depends on open-source software now. Every single application has millions and millions of lines in open-source software, which is not being updated. This was obviously a problem. At Asana, we had to try and figure out how to fix it. Our solution had two parts. One was anytime any engineer wanted to add a new third-party dependency, they had to pick an owner for it.
This is an individual or a singular person who had to own this, which is interesting. Sometimes, when you ask people to do that, they’re like, “I don’t want this anymore. I don’t want to be on the hook for updating it, so I’m not going to introduce this. I’ll write it.” The classy example here is a left-Pad. There was this NPM library called left-pad and it didn’t do a whole lot.
It did a very small thing. It basically padded text to the left. It’s like, “Should we depend on this because it’s this risk?” What happened was that it was sold to somebody else and then somebody put a backdoor into it. Everybody then was running leftpad and a backdoor code running. It’s like, “I’m not sure if the risk of having left-pad outweighed the benefit there. It seems like not.” Asking engineers like, “Are you going to update this,” is somewhat of a deterrent. If not, then it’s like, “You need this update.”
That’s the first part that works well. We make sure every single new third-party dependency has an owner here. We track this with an Asana task, but you could track it with a spreadsheet or whatever you use. The second part is we hook our dependency scanner up to our issue tracker, which is Asana, but you could use Jira or whatever you use.
We then look to see who the owner is like libxml. We look up who owns libxml and then when there’s a vulnerability in it. That person is assigned a test like, “Go update this.” We have a company culture here where it’s very important to make sure that if somebody assigns to you, it’s probably important. You should probably do it right.
We get a pretty high uptake here on people fixing it. We have the risk reduction of having fewer of these in advance and then the second part is we get these fixed. We saw a drastic increase in our third-party report of vulnerabilities being fixed here. We’re getting to even the mediums and lows, which is very uncommon. We’re very proud of that. I feel like it’s something that’s under report in our industry of how to get these things fixed.
Enough about the company. Let’s talk about you and then one that you are responsible for, which leads us back to the company. We had guests who had served in various roles across the board in security organizations from CISO throughout. You are head of information security for a global company with a couple of thousand employees. Where’s your role different from other leadership positions? I don’t think we’ve had anyone that is listed as head of compared to other things.
I’d say it’s comparable to a CISO role, but Asana has a company culture of not having level titles, which is interesting. You’re allowed to be a head of, but you can’t be a senior or whatever. It’s very egalitarian, but my role encompasses all aspects of information security or corporate security and product security, which we already talked about. Cloud security, also risk and compliance like my team does, SOC 2 and ISO27K, and compliance audits.
We have a function that’s like a CIO and there’s a dotted line reporting over there. There’s a dotted line reporting to legal because security has a big legal impact. I report to the head of engineering. I feel like that is a good fit at a company like this because we make one product and that product is the biggest source of our risk. Having the inside line into that organization is extremely important. If I report it to legal, the CIO, or even the CEO, it would be less effective because it’s harder to align on the thing that matters, which is fixing the product and the cloud infrastructure than it brought.
The relationship between security and engineering isn’t always cozy or harmonious. It sounds like the culture that you’ve built at Asana is exactly that. How do the teams work together from the development process? Is it literally from day one from the ground up? Do you play tennis back and forth with the issues? Are there scraps?
It depends on the project. For your average new project, we have every team fill out their own risk assessment checklist. You want to add a new feature to Asana. You fill out this checklist, asking you questions like, “Are you writing your own cryptography or doing anything with that? Yes or no?” If you say yes, we’re going to be involved because it’s like, “What are you doing with cryptography?” This is very interesting. I don’t know why it’s doing this. If it’s like, “We’re adding a button here. It’s all the same as normal. We’re changing the color of the button, but it doesn’t matter.” It’s like, “That risk assessment goes away and that team can ship the product as fast as they can.”
If the risk assessment is a high risk, then the security team does. Either asynchronous risk assessment, where we’re going to read your documents and give you feedback. Also, “We’re going to design this with you.” For instance, the Asana desktop app uses Electron and Electron is an interesting piece of technology because it’s what powers Slack.
It’s a super great user experience, but if you have a cross-site scripting vulnerability on it, it can turn into remote code execution. Imagine the Asana desktop app or Slack had a bug like that. You could take over endpoints, which is a nightmare scenario. When we were doing the desktop app, we were like, “Security is going to embed onto your team.”
We’re going to put security engineers on the desktop app team and they’re going to be team members there. They’re going to be reviewing your code and designing it with you because getting that right the first time was absolutely critical. It’s the heaviest lift we’ll do. It runs the gamut from no involvement because you’re changing the color of a button all the way to we’re going to put security engineers on your team. They’re part of your team for six months or whatever.
The color of that button needs to be secure. If we’re not paying too much attention, that’s when the insider threat can occur. Let’s talk about the team that you’re building and as you build a team, knowing that you are seconding out team members to projects going over to the engineering side. You’ve mentioned the word culture. Is that a natural outcome of the personnel that you are bringing in? Are you looking for a specific person that fits the culture or are these two stupid questions?
These are not stupid questions at all. It goes back to what my job is, actually. Some people view the CISO or the head of security’s job as like, “I get all the shots. I get to make all the hard decisions.” That’s not my job at all. That’s not what I do. My job is trying to hire the right people, give them the right structure and process, and then let them go and do their thing.
For instance, we were talking about embedding on the desktop app team. That was not my suggestion. The team got the risk assessment and they were like, “This is such a big important project. We want to be more involved than we’ve ever been before. Let’s go over there.” I was like, “What a great idea.” The team’s culture enabled them to even think that was an option because it was the first time we had done that.
It is something that I do look for when we’re interviewing people. For instance, we have this interview challenge where you’re not only supposed to find security vulnerabilities. We care that you find security vulnerabilities because we want you to demonstrate your technical skill, but we are also looking at how you deal with them.
We care that you find security vulnerabilities because we want you to demonstrate your technical skill, but we are also looking at how you deal with them.
We’ve had candidates find these vulnerabilities and be like, “The fact that these exist was embarrassing. Whoever wrote this should need to go back to school or training.” We’re like, “I’m glad you found this vulnerability, but I don’t want you on my team because I don’t need somebody who’s, to be honest, a jerk like that.” I need somebody who’s like, “There are some mistakes here. We’re going to work hard and we’re going to fix it.”
We want that positive attitude because it’s so easy to lose trust in the security team. It’s so hard to build and we’ve spent years and years building it. It could be undone in a few short months with the wrong attitude and the wrong people. That is very important to us. We have several questions where we’re like, “We’re not only looking for this one thing, but we’re also keeping our eyes open for other signals here.”
In your experience, what’s the hardest part about building a security team? We do hard-hitting journalism here at the show.
It’s a good question. Maybe one way of answering that question is trying to think about the mistakes I’ve made and why I didn’t anticipate these mistakes. I think that I’ve made many mistakes. One that comes to mind is around incident response. Incident response is hard to get right. Most companies don’t do a super solid job here. It’s hard. When do you notify? Who do you notify? Did you do a complete enough investigation here? It’s hard to find people who are very experienced in this and not burnt out by it. We, as an industry, burn out people a lot in incident response because it’s extremely stressful. It’s not appreciated a whole lot. You get a lot of negative attention.
You don’t get a lot of positive attention. That’s a recipe for burnout. I’d say building a security program with a reliable and long-term focus incident response is something that we’re working on. We’re trying hard, but it’s also something that I think is extremely challenging to get right. We were literally talking about how we could improve our incident response. I asked 8 different people and I got 8 different opinions that are completely not what I expected. They’re all different, this way and that way.
Unfortunately, you don’t have a headcount for eight to get each one of those in there.
I can’t build eight separate teams that are all separate for all these things. I’d say that’s probably the hardest thing to be great at when you’re building a security program. It’s something that seems durably hard because of the stress and the consequences.
We’ve heard so much about the skills gap. It’s been beaten to death, but I’m going to keep beating on it for at least one more question. As you talk about the difficulty in building a security program, you said incident response, but that’s almost in addition to being the title of a position. It’s a vocation and almost its own vertical. Within those types of things, are there other parts that you found that it’s like, “There’s a lot of people that are good at this, but this one thing, we wish there were more of them because we need them?”
This might be a spicy take. I wish I had more software engineer-shaped security people. This is a spicy topic because there’s this question in our industry of like, “How much do you have to code to be a security person?” I think the answer is, “No, you don’t have to.” There are tons and tons of great jobs out there for people who don’t code.
For instance, Asana has tons of custom infrastructure with tons of custom software, and to interact with it and to build what we need from the security controls perspective, you need to write software. We sometimes have this tricky problem where we are looking at hiring people and say, “We want to hire this many people. Should we hire that many software engineers or that many security engineers?” The security engineers are less good at coding.
They can script a little bit, but they’re not as good as software engineers. What do we do here? We need this big performant, complex software built and we also need this security engineer expertise. It’s hard to get these in the same person. If this is describing you, that you’re the same person, “You’re a unicorn. I would love to hire you. Please, talk to me.”
They’re very hard to find. We have been trying to hire 1 or 2 security engineers as a kernel of a team and then hire a bunch of software engineers and then send the software engineers to security training. It’s been very good because the core concepts of security are not that difficult to learn. There’s a mindset and some specific things depending on which area. If you’re in the cloud or more on the website, you can read The Tangled Web or The Web Application Hacker’s Handbook.
You can read these books and get a good sense of them if you’re a talented software developer. We’ve been trying that model and it’s been working out for us. I do wish I had more unicorn people who are great at both. We’ve had some luck with this kernel model, where we get a talented person who’s a mentor to others and then a bunch of engineers around them.
You get your Tony Stark in the middle and then you start building out with your Hulk eyes and your Hulks in the rest of them.
You have to have one person who knows a lot of this stuff. That’s been very useful on all of our teams.
How creative can you get when you’re looking at candidates? One of the great things about the industry is that there’s this big outreach to non-traditional non-industry people, people that might be older in different parts of the workforce. We’re talking about what you do, but can you get creative like that or have you got to have these specific bonafides to be in this thing because what you do is what you do?
I feel that most of the security is not particularly hard to learn. There’s a deep technical side to that it is, but that’s not what we’re talking about. That’s such a narrow slice of what security is. I feel like we focus on it a whole lot. We’re focused on the reverse engineers and the people disrupting malware botnets and everything.
Most of security is not particularly hard to learn.
It’s cool if that exists or if that’s real. There’s a huge other part of security. There is a response crisis communication. Security parts and crisis comms are totally different things. The same thing to security training and security compliance. Our compliance team is absolutely fantastic. A lot of those folks came from non-traditional backgrounds.
One person on our team, his previous job was at Home Depot. It turns out he’s super great. He came from Home Depot and he’s like, “I want to work something else.” He runs our security awareness now and he does a great job at it. I’m a big fan of people coming from other backgrounds. There’s so much to do in security that we need a diverse set of viewpoints and we’re much better for it.
You look at personal life through the insider threat prism. One of the things we talked about in the prep call was the abundance of contractors that are working now as opposed to permanent employees. I’m not sure what the right term is but let’s go with contractors versus employees. Do you need to take a different stance on that because one person has this number written on her HR form and this one has a different one on his? What’s the approach for you philosophically when you look at that thing?
Some people look at those as different and say, “Full-time employees might be compensated with equity or bonuses. They have this financial incentive to align with the company, whereas the contractor doesn’t.” I don’t necessarily agree with this. One is because those financial incentives could also exist with contractors.
I don’t know the deals of the contractor, but two, usually, people are not that tied to the financial results. They’re not like, “I have to do every single waking moment of my life. I have to make this company succeed.” They have a life and they’re busy. They’re doing other things. I’ve definitely been tracking in the industry because people have this false perception of insider threats, like, “There’s this evil person and they’re going to join my company to harm it.”
It’s like, “Sure, that could happen. What about a person who is tricked?” We saw that with the Robin Hood incident. An IT person called somebody and tricked them. That’s still an insider threat, but it’s a different thing. They were misled. A Twitter incident a few years ago where they were bribed and said, “I’ll give you this money if you do this.” You don’t know the financial situation of all your employees.
That might be $10,000 or whatever it was. It might have been the world to them. At other companies, we’ve seen threats like, “Here’s a picture of your parent’s house. I’m sitting outside of your parent’s house. Are you going to do this for me or not?” If you’re trusting your company’s equity or whatever to prevent a situation there, you have a different view of it than I do because I had a picture of my parent’s house.
I am now very motivated to make that person happy or to figure out the situation. I think what you have to do is you have to not use those things. You can’t use these signals. You can’t say , “Are they full-time employees? They have ten years of experience at our company.” Those things turn out that it doesn’t matter. What matters is can they do this? I think of it as insider trading a little bit.
The example I like to use is, imagine we were like, “We’re a public company. I’m going to give detailed financial records to every single person in the company. Why not? They should have access to it.” Some new grad is like, “Hey friends, here’s a Snapchat, TikTok video, or something from my company’s financial statements.”
They trade on this information and then they all go to jail. They didn’t mean anything or any harm by it. They were accidentally misusing this. It is true of any data. We’re super careful about company financial statements. There’s a firewall and who has access? Do you have material non-public information or not? We’re not as careful with customer data and it’s like, “Maybe we should actually treat this with the same level of data.”
You don’t want access to this. You do not want access to customer data. If you have access to it, we have all of these restrictions and qualifications around it. We’re going to do all of this auditing. That’s what I think we need to do. We need to shift from a model of employees having access to everything. We’re seeing this with the Twitter drama right now.
Almost everyone on Twitter has access to all of these super-powerful tools. It should be the opposite of almost no one having access to these tools. You have to ask for access. For example, at Asana, engineers do not have access to production. They have to ask other engineers for permission and then we audit that. We make sure that they do the thing that they said they were going to do.
For instance, you said, “I am going to reboot the database because it’s not working.” You go in and you do something else other than that that launches a security incident. You did not do the thing you said you were going to do. It’s not that we don’t trust our engineers. We love our engineers. Our engineers are great, but it’s that we understand that you both need to fix production, but also we need to protect customer data. You do not have a right to it in any respect. The fact that you’re a contractor, a full-time employee has no bearing on this. It’s like, “We would do this for anybody.”
How hard is it to implement something like that? When you bring in that type of new policy to make sure that it’s not onerous, it doesn’t get in the way, but also, you mentioned the employee who’s worked here for ten years. It’s always worked like this. This is a big pain in the ass. Why are we doing this thing? As the leader of the security organization, how do you get the buy-in on things like that?
It was a little challenging for us at first because we had the exact same feedback. We said, “I’ve worked here for so long. I don’t know why you don’t trust me anymore.” It’s like, “I understand where you’re coming from, but it’s not about that.” What we did was we shared our goals. Our goals are, “We treat customer data as this highest-level thing. You should only access it if you absolutely have a need to.”
We also said, “We post the hypothetical. Wouldn’t it be great if you could do your job and you wouldn’t have to jump all through all these hoops? You didn’t have to ask for data. You could reboot the database if you needed to, but you didn’t have to access customer data if you needed to reboot the database. Wouldn’t that be great?”
They were like, “That would be great.” I was like, “Why don’t we build that instead? Instead of building tools that either accidentally or purposely need customer data, why don’t we treat customer data as this thing that we never include in logs and never have access to? It definitely builds all of the other things we need to do our jobs.”
That was a pretty compelling message because they were saying, “This is going to hurt my job. What if I’m on call and I can’t get access to these systems?” “Let’s build the solutions here.” It was a bit of work trying to build all of these things and we made a mistake early on where there was a system that didn’t have customer data in it. We were still asking for this onerous approval process, which slowed people down. We fixed it and tried to build up our reputation, like, “We are going to fix this when it makes mistakes.” A stat I wanted to share with you, is before, we had something like 20% or 30% of engineers who were satisfied with how we access production.
That was before when every engineer had access to it. Now we have 85% of engineers satisfied. We made it substantially easier. The security scene took on the work of let’s improve our user experience here. Let’s make it better and faster. We made log-ins that are significantly faster than the actual connection initiation. We did a bunch of work there. It was coupled with this access restriction, but we’ve also improved the whole thing. Overall, people’s satisfaction went up. We got to a great security outcome and our product velocity is the same, if not better.
Do you have one of those OSHA-type signs, like, “It’s been 277 days since the last security alert?” Get the tag board up there for sure. Different question now. Is there any cool technology out there that you’ve got your eye on? Something that maybe not quite ready for prime time or brand new. You’re seeing it out of the corner of your eye, but you think that may become more front and center, in the focus. It doesn’t have to be a company specifically, but a tech that is growing into something you think needs to be taken seriously.
eBPF is the thing that I keep coming back to. This is the Berkeley Packet Filter in Linux kernel. What it allows you to do is run some user mode code in the kernel and it lets you have access to things. There’s an explosion of companies doing stuff with eBPF. I was talking to a company that was doing the coolest vulnerability analysis using eBPF. It sounds like Berkeley Packet Filter is such a networking thing and it’s the initial purpose here, but they’re using it for like, “Is your code affected by vulnerabilities?” Something that’s been challenging is SSH session logging. It’s good at SSH session logging.
I saw a great tool using eBPF doing that at making sure you can’t log bash history. What if somebody changes their shell or what if somebody launches the session to shell? eBPF can catch those things as well. That is the technology that I am absolutely most excited about. It’s because it is taking this very simple idea and allowing us to get all of this visibility and inspection without needing to risk the stability of your system by installing the kernel module. I’m looking forward to some EDR tools using eBPF. If you have an eBPF EDR tool, please hit me up. I would love to find out more about it. I think that would be great to hear about.
That is, without question, the best answer I have gotten to that maybe in any show where I’ve asked. Usually, there’s a bit of hemming and hawing. There are a few of the old chestnuts about, “The blockchain is interesting if we can get machine learning.” That was very specific and very in the weeds. Here’s the reason why. In fact, if you’re doing it, please contact me because I want more.
You can tell I’m a huge eBPF fan. I’ve been tweeting about this as well and I’ve had a few vendors now add that to their email pitches to me and they’re like, “We are also using eBPF, which we’re a fan of.” Honestly, that pitch works because I’m like, “You must have the right thing here.” Whereas when somebody gives me a blockchain or whatever thing, I’m instantly like, “I don’t even think about it.” I don’t even look at it for two seconds. If you use the right keywords, of course, I’m like, “I want to see what you’re doing.”
Moving on to Leadership Corner. You, Sean Cassidy, when you are not doing these things to keep the company doing those things in order to make the world a better place, what are you doing? What’s on your playlist? Are you reading or cooking? Do you go kite skiing?
I am the father, first and foremost. I spend a lot of time with him. He’s great, so we do a lot of things. He’s been biking at the pump track, so I’ve been bringing my bike there and going mountain biking. I’m like, “I didn’t think three-year-olds could be on a pump track,” but they can. It’s great. It’s a ton of fun.
What else do I do? I love to bake bread and pizza. I have a pizza oven. Bread is the exact opposite of security in some ways because there’s no adversary and no problem here. You can bake the perfect loaf bread and in this thing, you can slowly get better at it. Every time you do it, it gets a little better. Also, you follow a formula which you can’t do in security.
As an engineer, you can’t make things happen. You can’t do the leftPad thing once it’s all together. You got to sit and it’s chemicals.
Sometimes, it’s like, “I don’t know if this is going to work.” Sometimes it doesn’t and that’s okay. I love movies. It’s a big hobby of mine, especially classic movies like detective-type stuff. I love Sherlock Holmes.
It makes sense for what you do and the teams that you lead. I can see that happening.
Specifically the Jeremy Brett Sherlock Holmes from the ‘80s. That is absolutely far away from the best Sherlock Holmes. It’s incredible.
Benedict Cumberbatch might want a word.
I don’t hear this as much anymore, but some people say I look like Benedict Cumberbatch. I’m like, “No. Jeremy is the best Sherlock. Don’t accuse me of looking like Benedict Cumberbatch.”
There are worse lots in life. Shameless plugs if you’ve got anything going on. You’ve already mentioned that you’ve tweeted about stuff and websites. What’s happening with events or speaking engagements? Are you writing, publishing, doing cool charities you’re involved with, or anything like that?
You can follow me on Twitter, @Sean_A_Cassidy. That’s probably where I’m most active. The website is www.SeanCassidy.me. I blog occasionally, but it’s not something I’ve been doing a whole lot. I’d love to do it a little bit more. That’s where you can find me. I love talking about security. If you want to hit me up and have a chat, I love doing this. As I said, eBPF or otherwise, I love hearing from folks running security programs, what’s working, and what’s not. Feel free to reach out.
If they want more info on Asana, where can they go for that?
www.Asana.com. I’d also love to talk about it if you want to use it for your security team. We obviously use it a ton for our security team and we have a bunch of different workflows, so feel free to hit me up for that too.
This is the man who gives very detailed and definitive answers. If you reach out, he’s already given you both his work email and Twitter. You can find him. He will talk back.
I will respond.
Sean, you have a lot to do. I appreciate you taking the time to join us. That is it for this episode. For more information on all that’s good in the world of cybersecurity, make sure you check us out. You can find us on LinkedIn and Facebook. The site is Elevatesecurity.com. My name is Matt Stevenson. You can find me @PackMatt73 across all the socials.
We’ve been doing these shows for a while now and we’ve had a lot of fun. We’ve had an incredible stack of guests. Sean, consider this your invitation to the inevitable round table where we will assemble the Avengers. As I said, we got a Hulk, a few Iron Men out there, some Black Widows, and all kinds of good stuff, so please come back.
I can’t wait.
Everybody, wherever you go for your podcast, that’s where you can find us. All we ask is you subscribe, rate and review. You’re never going to miss the great folks like Sean and everybody else coming on the show. Until then, we will see you next time.
Important Links
- ElevateSecurity.com
- LinkedIn – Elevate Security
- Facebook – Elevate Security
- Asana
- @Sean_A_Cassidy – Twitter
- www.SeanCassidy.me
- The Tangled Web
- The Web Application Hacker’s Handbook
- eBPF
- @PackMatt73 – Matt Stephenson
About Sean Cassidy
Sean is the Head of Security at Asana. Sean has been in information security for over 15 years, and focuses building security into an organization’s culture. He was previously Head of Security at Patreon, and the CTO of DefenseStorm.
