Recently, we sat down with Dan Robinson, CTO, and Jerry van Leeuwen, DPO, from Heap, to discuss their approaches to security and what they’ve learned as far as security goes throughout their careers. Being able to speak with both of them was great, since Dan brings the executive perspective, and Jerry brings the developer and privacy perspective. We wanted to share the insights that came out of our conversation.
Tell me a little bit about yourselves and your time at Heap
Dan: I joined Heap as their first engineer six years ago, and have since become the CTO. As such, I’ve been with the company and the technology for essentially the entire journey. Heap is a product that captures data about our customers’ customers, and it would be a major incident if it were breached, so we take security incredibly seriously. As a result, I’ve spent a fair amount of time really focused on learning security best practices and how to approach it at a SaaS company.
Jerry: I joined Heap two years ago as a developer. When GDPR started coming about, we needed a DPO (Data Protection Officer), and I stepped into the role. I’ve worked closely with security in past roles, but this was my first time being responsible for aspects of it. I’ve focused a lot on improving our privacy efforts and compliance in the past couple years.
How have you approached security at Heap so far? What’s your philosophy on it?
Dan: To me, security is not about building a high wall, it’s about gardening. It’s about reducing risk. You will never have zero risks — people make mistakes, you develop new things, and so on, so it’s about reducing the risk of catastrophe. The way you do that is by following proven approaches and best practices. It’s more about creating systems to organize your risks and doing the work to resolve them. You have to invest in tooling, engage with external resources to find holes, and execute on what you find.
If I had to summarize our approach into a guiding security philosophy, I would say that it’s about maximizing leverage. There are so many potential actions to take on security, so it comes down to following the highest leverage pursuits, and taking an honest accounting of what’s working or not. At our stage, security is achieved via execution, not white papers.
Jerry: One thing to keep in mind is to not take security too far. It’s tempting to never stop adding security, but you need to keep an eye out for what will be effective for you rather than just adding more complexity. Keep it simple but robust. Prioritization matters a lot for security, more so than it does for features, even. Prioritizing the wrong feature is an opportunity cost, but prioritizing the wrong security measure can have legal consequences.
How have you scaled and evolved security at Heap?
Dan: In the early days, our first step was getting our cloud fundamentals in order. Making sure our machines were in VPCs, that our databases were secure, etc. We focused on building the most basic possible security infrastructure to ensure that nothing was wide open to the world. For any new startup, this is a great place to start.
From there, we started doing pentests with 3rd parties to help get our house in order. We put basic infrastructure in place to ensure that our perimeter wasn’t Swiss cheese. Once we started working with pentesters, we made it a policy to fix anything that they found that was medium risk or above.
To stay safe, you need to have processes in place to vet your security. After a while, our company had a secure perimeter, but no depth beyond that. If an attacker got inside, they were home free. By systematically reviewing our risks, we were able to understand these kinds of gaps and move to fix them.
Jerry: We moved beyond our early focus on the perimeter because we were aiming to get SOC2 compliance. On top of that, GDPR was coming into place. SOC2 asks a lot of questions about processes, which we didn’t have a lot of at the time. We created them to get into compliance and fix these gaps. Now we have things like a robust vendor security vetting process, and our security is much better off for it.
As you grow, you security needs change in terms of where you should focus. Incident management processes, for example, have grown in importance for us over time. Like with everything in startup life, you need to refine your security processes as you scale. No process will work from now until forever. This circles back to my point on being careful not to do too much all at once. You don’t want to lock yourself into something that won’t scale. Security needs to be able to change for your company. You want to find the right level of security and investment now that will protect you and still leave room to change and improve in the future as your organization and target customer base grows.
For us at Heap, scaling has included expanding our security posture, not just deepening it. One good example is with physical security. When we were smaller, everyone knew everyone by sight, so there was less need for training around challenging unfamiliar faces. As we’ve gotten bigger, we need additional measures in place, and have set them up.
Dealing with unlocked laptops is another good example. We have a specific gif that people post in Slack when they catch someone’s laptop unlocked. Not everyone will be perfect. That’s not the goal. It’s more about the constant process of reminding each other and taking it seriously (without shaming them too much). We don’t want it to be too abstract. Otherwise people will be doing security stuff as a ritual without understanding why they’re doing it. It’s important to find the right balance without adding blame. The tone matters. The goal with these efforts is to raise security awareness across the board, and get at least a few people to be true security champions.
What have you learned about security at Heap?
Dan: Our processes have needed to evolve a half dozen times as we’ve grown. Take data access for example. In the earliest days of the company, the guideline was “don’t be dumb with the customer’s data.” Now we’ve grown as an organization and have more people, so we need written guidelines on the specific cases when accessing customer data is ok. We need trainings, audit logging, etc. It’s important to iterate on this every six months as we grow and scale.
Additionally, two repeated themes keep coming up:
- There are practical security investments you can make that have leverage. Several times we’ve found architectural investments that can lift many areas at once, e.g. setting up our DevOps tooling such that by default almost nothing had a public IP, or building code abstractions such that developers aren’t writing raw SQL.
- Process matters. We’ve set up a risk register process that’s been very valuable. Our risk register is essentially a JIRA board with every single non-urgent risk anyone thinks about or finds. So it’s a combination backlog and idea board. Having a risk register has led us to make a lot of proactive investments on security, such as restricting access to tools from people who don’t need it, setting up solid audit logging, etc. Those are the types of initiatives that won’t come from a pentest. The risk register process forces us to prioritize all the risks to the business and what we can do to mitigate the most important ones.
Jerry: It’s really important to keep track of failures — that’s where you learn the most. The near misses are super insightful. Something we’ve learned is to not just say “we were lucky” and move on. To be secure, you have to pay attention to the near misses. Often times, they’re the only indicator you have that something needs your attention before a real incident happens.
Something else we’ve learned is the importance of balance. We’re very intentional around not creating roadblocks in front of people innovating. We don’t want to do things that will slow our team down until it’s necessary. Sometimes you need to find adequate, workable security rather than ideal security, and at the lowest cost you can find it. And what’s “adequate” shifts all the time.
What are the challenges you see blocking your ideal state of security today?
Jerry: I would say that there’s nothing particular that’s blocking us from achieving our ideal state of security at Heap. It’s more about triaging what we can do and determining and prioritizing what we should do. For us, the challenges are more around choosing a course of action and the timeframe rather than around being stuck on any particular point.
Dan: Something we focus on is making security as easy as possible for our developers by taking it off of their plate as much as we can. I believe that trying to keep security top of mind for developers is generally the wrong strategy. If the security of your product relies on engineers making secure choices every day, you’re going to have a bad time. Instead, our strategy is figuring out how to make the secure way, happen by default. We want to make it harder to do insecure stuff than it is to do secure stuff. We try and give our developers the right primitives. For example, use default sets of libraries with protection against injections, which reduces a whole surface area.
Essentially, we want to make a really clear paved path that is extra work to deviate away from. Beyond that, it’s important to have external validation. We have neutral third parties come in and punch holes in what we’ve built. The external attention is super important.
Jerry: Deep account separation is a good example of how we make security the default option. Customer accounts at Heap are separate at a very deep level in the code, so it’s much harder to touch the wrong data by accident. It goes with the way the product is pitched. Heap isn’t meant to cut across customer accounts, so people won’t try to cut around that blocker. Customer isolation is baked in so deeply into the code and business model, that it becomes an accepted default into any feature or work our team does.
What excites you about the future of security?
Dan: The tools are getting better every year. We’re seeing more and more cross-sectional coverage of all threats. Every six months there’s a new tool that gets me excited and that I want to prioritize and explore.
Jerry: For all its imperfections, GDPR and the like is making concerns about data privacy front of mind, and that’s a good thing. It’s driving innovation for making personal data more secure. Tools are coming out that can help with this and are moving the game forward. That’s good for everyone involved, and I’m excited to see where that all goes.
Do you have any advice for companies or people beginning their own security journey?
Dan: What counts as “good security” depends on the stage of your company and your current posture. When you’re first getting going, start with the obvious stuff. Make sure that the front door is safely secured and that you have walls on your house. Beyond that, it becomes a question of gardening. Like gardening, you need to give your security constant attention and have a process you take seriously around determining your risks and investing where necessary.
One additional piece of advice: getting hacked is one of few existential external events that will cause your company to no longer exist. Get serious about it, and get your first pentest by the time you’re making your first meaningful revenue. Take some care in choosing your pentester and don’t cheap out here, since the quality of the investigation can vary massively. Don’t delay security, because the potential consequences are very real.
Jerry: Yes, the goal of pentests is about finding as much as possible, not about getting a “passing grade.” You want to set your pentester up for success and make it as easy as possible for them to really dig into your system. Also, make sure you do your pentests under NDA!
Overall, security isn’t about removing all of your vulnerabilities, but rather about removing the biggest ones and taking them below the threshold of being worth the time and effort that it would take to exploit them for an attacker. Security is not one size fits all. Some parts of your system need more care and attention, and you need to tailor your efforts for the specific realities of your system.
—-
Thanks so much to Dan and Jerry for their time. If you’d like to read more interviews with people in the industry, check out our conversation with Adam Surak on scaling security at Algolia.
