Getting security to scale: learnings from modern app sec teams

modern app sec teams

Why are we writing this?

In April, I visited the Bay Area from Paris for a week with some members of my team. While we were there, we sat down with a half dozen application security teams, all in companies that are less than 15 years old with 100 to multiple thousands of employees. These are some of the best app sec teams operating today, and we figured there was a lot we could learn from them. Our goal was to understand the challenges and the way these teams are working.

We summed up what we learned in this blog post, in order to share their successes and their approaches to security with the ultimate goal of passing their hard-won lessons onto other security teams.

What’s an app sec team?

Before we dive into the takeaways, let’s start with what the teams we talked to do. An application security (app sec) team’s mission is to increase the security of the services developed by their company without impacting the velocity of other software teams.

They belong in companies in which software development is a key asset and requires rapid iteration to support the business.

Typically, these teams are mixed in with other security teams, and they all report to the CISO, if present, or CTO, if not. There are differences from company to company, but generally, these teams are responsible for the following areas:

  • Application security: The app sec team is responsible for handling everything from project inception to release from a security standpoint. They often work as “consultants” to provide guidance and advice to developers. This aspect is also sometimes called product security.
  • Incident response and monitoring: They are in charge of monitoring the production systems and kicking off the response process for any incidents that arise.
  • Corporate and IT security: They often manage internal security (employees directory, asset management, devices monitoring, etc.)
  • Governance Risk and Compliance (GRC): This area is about managing certifications like SOC2, Fedramp, and ISO 27001.

The modern app sec team’s philosophy

The app sec teams we met during our trip had different priorities and projects, but they shared an overarching similarity on their beliefs and goals. We found that these teams use similar strategies to try and achieve these goals, and all of them are quickly improving the way they work towards strengthening their security.

Be enablers rather than gatekeepers

The security team must be aligned on the company objectives. Their goals should be as close as possible to those of the developers: ship features with security awareness. Hence, they need to be working hand-in-hand with the dev team in order to find pragmatic ways to deal with security concerns.

Some modern app sec teams prefer to hire software developers rather than security engineers, just because it gives them much more legitimacy and ease to talk to developers. The cost of leveling them up in security is superseded by the benefit of having much better trust, empathy and communication channels. Their willingness to take this route shows how important that trust is to app sec teams.

Do things that scale

The typical app sec team headcount is ~1% of the developer team’s headcount. That’s the average of all the app sec teams we interviewed, no matter the company size. That’s a clear asymmetry, and as it’s the norm, it requires a strategic approach in order to improve the security level of what the developers ship. Modern app sec teams don’t have the luxury to do many things manually if they want to keep up with developers.

Hence, they often combine the following approaches:

  • Delegate day-to-day responsibility to the developers by
    • Increasing their security consciousness
    • Giving them bigger responsibilities around security
  • Focus on catching the low hanging fruit
  • Involve security as early as possible in the development process (aka shift security left)

Understand your developers

Working closely with developers is at the heart of what app sec teams do. The modern app sec team wants each developer team to see them as an ally, and to keep them in mind – so any security subject or concern can be promptly and openly discussed.

There are different ways of building this relationship:

  1. the security champion: an individual developer who will be the security enthusiast in their own team. They will become the primary point of contact between the security and dev teams
  2. the developer champion: this is the complement of the previous one. Modern app sec teams sometimes hire developers (rather than security specialists), so that they can bring a greater understanding of the developer mindset and connect more closely with dev teams.
  3. process-based: The third manner is to rely on formalized process. By leaning on well laid out structures for new projects, app sec teams can ensure that they are present at the inception of new features and projects to spot security concerns early and help the team architect with these concerns in mind.

Given the disparity in headcount, it would be super difficult to train all developers to ship absolutely secure code. Modern app sec teams take a more strategic and scalable approach, by leveraging training for spreading good practices. It helps developers to take incremental steps towards better security rather than act as a be-all, end-all set of documents and processes.

Trust your developers

Developers are not incentivized to build high security. Traditionally, they are incentivized to ship good features with high velocity. These days however, modern software companies are enlarging the software team’s responsibilities:

  • Performance: developers are responsible for the performance of their software. Thinking that a team of ops will throw hardware at it is a thing of the past (the rise of APMs 10 years ago is a good example of this).
  • Quality: plenty of modern companies don’t have QA, they rely on the team and on internal dogfooding in order to ship quality software.
  • Usage metrics: the focus of developers is shifting towards usability and creating features that users actually interact with. Indeed, modern companies often say they work with product engineers rather than software engineers.

One thing that’s missing from this list today is security. However, that is something that needs to change. Developers cannot be experts at everything, but they should be aware of the impact that their creations have, and also of the risk the bring.

The modern app sec team needs to trust developers to be responsible for the security of the things they ship. If they realize that some important risks exist for a given feature, it is the feature team’s responsibility to acknowledge the risk (or refer it to other stakeholders). They can choose either to accept the risk, or delay shipping the feature. Working this way requires developers to take on some level of security responsibility and awareness. App sec teams should help build that level of awareness.

Staying in the loop with new software projects

One challenge for app sec teams is to ensure that they’re involved when new projects are started. The earlier the security team gets involved in the project, the better the chances that they’re able to spot security challenges early, and to work with the software team to overcome them.

All companies that managed to solve this have a formal SDLC (software development lifecycle), where each new software project or feature goes through formal steps. In order to make the security step as light as possible (and thus scalable), it can be automated, just requiring developers to answer such questions:

  • are you using a database?
  • is this on the Internet, or internal only?
  • are you handling customer data?

How modern app sec teams keep track of applications

Having an understanding of what applications your company has deployed is crucial for app sec teams. You can’t ensure that something is secure if you don’t know it exists! Unfortunately, in reality for most teams, knowing what applications or services actually exist in the company is hard. Developers push new code daily and don’t necessarily have a way to systematically let app sec teams know about a newly deployed service or application. Plenty of modern solutions such as Heroku or Kubernetes make developers able to automatically start completely new applications without involving other teams in the organization.

So how do modern app sec teams keep tabs on the applications that their company has deployed? Some are doing a manual inventory of the in-house applications, which provides a point-in-time vision of the company services, but gets quickly outdated. Other, more mature teams tend to take a more formalized approach. An example of this is how the Shopify app sec folks team up with SRE to keep tabs on new applications.

A formal SDLC can help teams build a list of all deployed services.

Common tools modern app sec teams use today

Given how important being able to automate and scale is for app sec teams, their tooling matters a lot for them. For the app sec teams we spoke to, their toolkits vary widely, and the context in which each approach is used depends a lot on the specific company. The following though were consistently used across app sec teams:

  • Pentests: Most teams leveraged pentests in some manner, whether targeted at a feature, a part of the application, or across an entire application. Some are done for compliance reasons, while others are done as a way to challenge the current understanding of the security of the application.
  • Bug bounties: a common practice that app sec teams implemented were bug bounties. Having these helps the team handle more bugs and issues than they would be able to find and deal with on their own.
  • A CI code testing tool: many teams are leveraging linters to check basic assumptions about the code being pushed. Few teams actually use SAST tools, but a notable exception is Brakeman, which is widely used for Ruby on Rails codebases. These tools are usually heavily generating false positives and require the app sec team to review everything prior submitting reports to the developers.

Regarding CI, the fewer languages the software teams use, the easier it is for the app sec team to build relevant tools in the CI. When an organization uses multiple languages, the app sec team often falls back to using simpler tools, such as linters.

Using tooling to help developers work securely

App sec teams often build tools to put security guardrails on some practices. A common example is building a helper to securely create AWS S3 buckets. Anyone who has ever created a S3 bucket knows that doing it securely is super hard (AWS rights management is powerful but complex). App sec teams will then create tools for their developers that makes it easy to create an AWS bucket with a step-by-step approach. Some decision factors include:

  • does the bucket need to be publicly accessible?
  • will the bucket store sensitive data?
  • does it have write-only, read-only specifics?
  • did we enable encryption at rest?

Having a tool like this not only helps to create more secure AWS buckets, but also gives the app sec team visibility into what S3 buckets are being created, and lets the team enforce defaults tailored for the company’s specific needs.

Ensuring that tools integrate well with developers

A key consideration for app sec teams is how a new tool will fit into the developers’ existing ecosystem. Changing the developer’s workflow by adding a new tool means:

  • one new place they get notifications from
  • one new thing they need to learn
  • one new thing you have to train them for

If a new tool requires any kind of substantial change, the result is often that the tool won’t be used (and usage cannot be forced – this would introduce friction in the app sec / developer relationship). On the other hand, when it’s done well, new tools can have a great impact. Here are some great examples of a tool seamlessly integrating into developers’ workflows:

Commercial vs open source tools

Plenty of other tools are used by modern app sec teams, and there is some consideration between open source and commercial options. Open source tools are more common, but open source projects are usually time-consuming to configure and use properly, and are harder to put in place than some commercial solutions.

Two famous examples follow:

In general, very few commercial tools are considered relevant at the enterprise level by the app sec teams we spoke to today.

The social network of app sec teams

The security space is a friendly one among the people who work in it. Some app sec teams are especially influential, and communicate about their work, sometimes even release part of their custom toolkit as open source projects. The industry values the work of great teams and is especially looking at the app sec teams from the following companies:

  • Netflix
  • Slack
  • Cruise Automation
  • Coinbase

Also, blogs are popular and widely read in the space, making them a great place to learn what’s going on and to share best practices. Some recent blog posts that were often referenced by the app sec teams we spoke to include:

Many teams and practitioners share their practices at conferences as well. Some conferences that were frequently brought up include:

  • OWASP AppSec
  • BSides SF

Conclusion

Spending a week meeting various modern app sec teams was extremely insightful for understanding their challenges and philosophies. In a nutshell, modern app sec teams feel responsible for building friendly and productive relationships with their developer teams. One of the ways to get this done is to recognize the developer’s team maturity and to work with them around taking responsibility for the security of what they ship. This is the best way to acknowledge that security is only one dimension to consider when shipping software.

I really felt that the companies that manage to achieve the best level of security while maintaining the ability to iterate on software quickly are the ones where the developer teams and the app sec teams are all aligned around their shared end goal: delivering value to the company.

Share
Tweet
Share
Share

Jean-Baptiste Aviat spent half a decade hunting vulnerabilities at Apple, helping developers solve them, and developing security software. He is now CTO at Sqreen.

Related Tags
Subscribe
Notify of
guest
3 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments