If you’re looking to make your applications more secure, you’ve probably heard of the OWASP Top 10. The OWASP top 10 covers the most common vulnerabilities, but plenty of other vulnerabilities can still be found and have significant business impacts.
In the first episode of the AppSec Builders Podcast, “Solving Race Condition Vulnerabilities”, Sqreen CTO, Jb Aviat, spoke with the founder of WeHackPurple, Tanya Janca, to discuss her expertise in solving for this lesser-known vulnerability. The episode describes race condition vulnerabilities, reviews a few real-world hacks, and presents actionable tips security and technology teams can take to solve this class of vulnerability.
What is a Race Condition Vulnerability?
First, let’s understand what a race condition vulnerability is and why you should care about solving it.
A race condition is when a system uses a resource that can be accessed concurrently (by the system itself or others). An attacker is able to take advantage of the time gap between:
- when the system performs a security control on the shared resource;
- when the system performs an operation on this shared resource.
If the attacker manages to change the shared resource between 1. and 2., then a race condition vulnerability is present. This type of vulnerability is also called ToCToU: Time of Check, Time of Use.
Let’s describe an example of a race condition using a bank ATM scenario:
ATMs are designed to handle one withdrawal at a time of your remaining balance. Let’s say your remaining balance is $50 and you arrange with a friend to withdraw your $50 remaining balance from two different ATMs at the same time. If you and your friend are successful in withdrawing $50 each simultaneously from two different ATMs then you have successfully exploited a race condition attack on your bank.
A similar, real life one, was found at Starbucks.
We’ve outlined some of the key takeaways on race conditions from the AppSec Builders Podcast Episode 1 here:
1. Race condition is a vulnerability in business code
It’s important to note that Race Condition vulnerabilities do not come from using a specific library, they come from using shared resources and forgetting that those resources are shared. And shared resources are Legion (in particular in Web programming) with databases or caches. A source code can be 100 percent bug-free from a SAST perspective, but still present some race conditions.
Race condition bugs, therefore, require the software engineers to think of the code with one more dimension in mind… time. You can think of it as an adversarial context, i.e. what happens if a part of the code is executed in parallel and if any shared resource changes state at any point in this code execution. This is what makes this class of bugs extremely hard to detect in most setups.
There are different kinds of race conditions. Most are found as bugs in custom business code like the Starbucks custom code hack by security researcher Egor Homakov. Egor found a race condition on the gift card page and was able to generate unlimited credit on transferred gift cards.
A popular one is called Dirty Cow, which is a race condition inside the Linux kernel released in 2016. It allowed a reliable and local privilege escalation on most Linux machines. The kernel didn’t coherently check for the memory permissions prior to allowing writing, and so the exploit allowed an attacker to turn a read-only mapping of a file into a writable mapping.
Another popular race condition is the Spurious DB (or POP SS) vulnerability that originated from the CPUs. In 2018, two researchers realized that all of the major operating system vendors misunderstood the Intel manual for 64-bit architectures resulting in an exception when the instruction immediately following is an interrupt.
2. Ensure developers know about database transactions for HTTP apps (commit / rollback)
Most ORMs provide a way to perform database transactions, as described herein, Ruby on Rails:
The Rails example is straightforward:
and also in Node.js with the Sequelize package:
For reference, the Go SQL package transaction reference can be found here.
3. Develop programs to find race condition vulnerabilities
- Start a security champions program.
“I’m a big fan of security champions programs. Before I even knew that they existed, I accidentally started one because I had started an exact program and I was kind of learning on the fly. And basically, I started giving software developers a copy of OWASP Zap because it’s easy to use and it’s free. And my budget was zero. And before I knew it, I had on each team that one that was super into scanning” Tanya Janca mentioned in our AppSec Builders podcast. Find that person that is security-minded, that’s obsessed with security and you can invest in that person to learn more about modern AppSec, attend conferences and develop a focus area in security. Even if you can’t start a formal security champions program, start one softly.
- Testing and code review:
There is an open-source testing option for race conditions and there is a C++ program that is called tsan on the clang compiler, called thread sanitizer, that will also work to find a lot of race conditions. Safe Rust is also an option, Rust is a memory-safe language that’s low level like C and C++ and has a lot of guarantees out-of-the-box to prevent data races and to enforce at compile time. So if you don’t leave it, you will have these guarantees applied to your code. Manual peer review on merge requests are also a great opportunity to check for race conditions, provided the reviewers have been trained to catch similar bugs!
- Using modern frameworks and programming languages, like Rust or Go.
Go includes a utility for finding race conditions in Go code. The Go Race Detector is based on the C/C++ ThreadSanitizer runtime library, which has been used to detect many errors in Google’s internal code base and in Chromium. Rust guarantees an absence of data races – but does not prevent general race conditions.
- Add Race Conditions to your bug bounty program
Add race conditions to your scope of work and be sure to nudge hunters with
experience in finding race conditions. Recommend them tools such as RaceTheWeb that will help them find such vulnerabilities!
- Training
Developers can only find what they know and most classes don’t teach race condition vulnerabilities. Follow WeHackPurple to learn more about training opportunities.
4. You don’t need to completely get rid of race conditions
As with most things in application security, thorough prioritization is needed. It is unlikely nor maybe even desirable to get rid of 100% of the race condition bugs in your organization. Also, not all race conditions have security impacts! Rather, assessing the riskiest business functions of one’s system should help you prioritize what needs to be audited. Also, add race condition to the list of vulnerabilities you’re evaluating when performing threat model (e.g. Microsoft STRIDE), so you can be certain that upcoming projects will check for this category of vulnerability.
5. Any system handling shared resources can be vulnerable to race conditions
There is a misconception that Node.js cannot have race conditions because Node.js has just one thread. In reality, Node.js had a lot of threads with one for the main loop. There are four asynchronous operations out-of-the-box and if you generate a simple Node script in macOS it will have 7 threads out-of-the-box which creates a lot of opportunities for race conditions.
If you have only one thread, like Node Express or Koa, your other web frameworks can still serve several HTTP requests concurrently, making them susceptible to a race condition vulnerability.
