Behind the scenes: building a dynamic instrumentation agent for Go

TL;DR

We’ve been working hard at Sqreen to make our protection transparent and frictionless. We recently released Sqreen for Go, which detects and blocks security issues inside Go applications without requiring any code modification. To make this possible, we leverage dynamic instrumentation to insert additional security logic into the program’s behavior at run time. Continuing with our dynamic instrumentation blog series, in this post, we’re going to discuss dynamic instrumentation, Sqreen’s Go agent, and our approach to bringing them together.

This detection and protection is based on a microagent, which is a software component that retrieves data from inside the running program and communicates with an external component to report statistics or specific data. The agent described here is a Go package communicating with Sqreen’s backend. It is specifically designed to safely instrument production systems, and thus involves sophisticated methods to maximize stability with a minimal performance impact.

Sqreen’s dynamic instrumentation has many advantages:

• Easy and fast installation since it does not require source code modification.
• It fully covers the program’s stack, including all third party libraries, the standard library, and the language runtime.
• The security logic can be updated via our central configuration dashboard, without requiring you to redeploy the running application

The agent is divided into three main parts:

1. The instrumentation engine that handles the low-level machinery of enriching a function with extra business logic;
2. The security rules engine that adds security protection logic to functions by using the instrumentation engine. It follows high-level logic descriptions sent by Sqreen’s backend, such as protecting SQL queries against SQL injections;
3. The data recording mechanism that asynchronously gathers data from the security protections and periodically sends it to Sqreen’s backend.

The following sections detail how we solved this challenge for Go. The example of the standard Go package database/sql will be used as an example of a component we need to protect against SQL injections.

Instrumenting Go code

The concept of runtime instrumentation is widely used in dynamic languages, often called monkey patching. But Go is a strongly and statically typed language, whose source code is compiled by the Go compiler into a program file containing binary machine code:

At run time, this binary program file is loaded and executed by the target operating system and hardware: 

Therefore, a running Go program is a running binary program, where run-time instrumentation modifying the binary code is considered unsafe, insecure, and sometimes even impossible in production environments.

Choosing the right instrumentation solution

The previous diagrams also give us every place where instrumentation could be performed: from source-level instrumentation performed by a tool developers could manually use, to hardware-level instrumentation of the live running program. What this means is that a large number of solutions are possible here. Choosing the best solution for Sqreen is a matter of selecting the best fit for the main requirements of Sqreen agents:

• Developer-friendly: easy development and deployment experience;
• Production-ready: able to apply security protections set up on live running programs, suitable for production environments by being efficient, safe and secure.

This results in the following matrix:

In short, we consider that the compile-time instrumentation technique:

• Has no impact on the production environment as it happens in the development environment only.
• Is automatically and transparently performed by the compiler.
• Is efficient by not requiring Operating-System round-trips (ie. it is entirely user-space).
• Is portable by not relying on any Operating-System nor hardware support support.

We consider otherwise that: 

• Binary-level instrumentation is impossible to make safe and secure for production environments.
• Current trap-based techniques such as user-space probes are inefficient due to hardware interrupts. They are also not portable and require insecure execution privileges.
• Source-code modification would be too complex to manage for developers, but also limited to the software parts they own.

So based on this analysis, we settled on the idea of leveraging compile-time instrumentation to add run-time instrumentation capabilities to the Go language.

Hooking strategy

We need the compiler to add instrumentation hook points to Go functions to allow Sqreen’s microagent to monitor and protect the function execution. For example, we want to be able to hook into the SQL execution function to detect SQL injections in the function parameter containing the SQL query string, ultimately aborting the function call in case of a detected attack.

To do so, our hooking strategy specifically allows us to be able to monitor and protect the function execution by:

• Reading the function parameters in order to monitor and possibly detect attacks.
• Being able to immediately return from the function to safely abort the function call and prevent an attack from happening.

The following code snippet shows an example of an instrumented Go function with the previously described hooking strategy:

 You can run the full example on the Go Playground: https://play.golang.org/p/zAQaf_rGaRs

Compile-Time Instrumentation: adding hook points to the Go program

The Go compiler doesn’t provide such hook point instrumentation by default. But code generation is very common and widely used in Go. For example, it is used for code coverage or race detection by the Go compiler. The Go standard library has everything required to parse Go source code, modify it, and regenerate it. Therefore, we decided to do safe source-level instrumentation in a standalone instrumentation tool to integrate into the Go compiler in order to produce the instrumented Go program file. 

This tool gets automatically called by the Go compiler every time it compiles a Go package, allowing it to instrument the function definitions it has.

The original source files are not modified and the instrumented source files are generated into the compiler build directory. Note that it is, therefore, possible to take a look at the instrumented source code using the Go build option -work which prints and keeps the Go build directory.

Being at the compiler level also allows for a full instrumentation coverage of every component involved in the Go program. We instrument the least amount of Go packages and the list is documented at  https://docs.sqreen.com/go/instrumentation/#list-of-instrumented-packages

Run-Time Instrumentation: attaching extra security protection logic to hook points

Thanks to the compile-time instrumentation efforts above, the Sqreen Go agent can now find the hook points using the hook table, a Go array also generated by the tool, and is able to instrument them when necessary. In the SQL injection protection example, once enabled on our dashboard, the agent receives the SQL-injection security rule. It contains the information required to instrument the SQL execution functions (i.e. there are 3 today: prepare, exec, and query), along with the SQL-injection protection logic.

The Go type information of the hook point is also used to safely validate and attach the security logic to the hook point using the standard Go reflection package. The operation of attaching it is atomically performed and safe for concurrent usage.

Once attached, the function execution can be observed and, most importantly, aborted when an attack is detected. The security protection can perform its detection from the function call execution context and leverage the Go function signature to be able to immediately return a non-nil error to abort the function call safely.

Our security protections also automatically abort the HTTP request handler by canceling the Go request context and responding to the HTTP request according to the selected settings in our dashboard.

Data transmission

Everything attached to the request handling path is made asynchronous to avoid impacting the response time. When a new HTTP request starts, our middleware function attaches a data structure to the request, which can be retrieved and used by our attached protections in order to gather security-related information for that request, such as attack details, metrics, and events. When the request is terminated, this request data structure gets asynchronously sent to the agent through a non-blocking Go channel to avoid blocking the HTTP request handler sending it.

The agent goroutine is thus sleeping most of the time and awoken by the Go scheduler when the Go channel has data available. The agent gathers the received data into a batch that finally gets sent to our backend server periodically (every 20 seconds by default). Our security protections don’t collect sensitive data, and the agent enforces it by scrubbing everything transmitted from the running program to our backend. Overall, this implementation is straightforward asynchronous Go programming.

Performance and robustness

The Go agent exclusively uses standard Go techniques, but its major role requires us to carefully consider a few points:

• We look at the Go scheduler’s impact and the concurrency pressure. We address this by always using a small, fixed amount of goroutines (currently 3). A good example is our metrics management, which is lockless thanks to atomic operations. These are in place in order to avoid having a scheduling impact due to any blocking operations that would have occurred with a Go channel alternative. This choice makes its execution overhead negligible and doesn’t involve any data queue management. This is especially important in the context of Go servers which are highly concurrent and can manage thousands of concurrent goroutines.
• We ensure that the agent always has time and memory usage boundaries thanks to execution deadlines (especially for our attached protections), and maximum data structure lengths.
• We pay attention to the garbage collector pressure by using memory pools where possible and avoiding frequent small memory allocations.

The agent is also specifically designed to be robust against backend communication failures by not relying on the backend to function; handling a high amount of request traffic; and even handling unexpected internal behaviors by ultimately restarting or stopping itself for maximum safety.

Going further

We described in this blog post the high-level concepts of a production-grade instrumentation agent. Dynamic instrumentation agents can be used to handle many different tasks, such as performance monitoring, error monitoring, or security. 

At Sqreen, our Go agent leverages dynamic instrumentation techniques in order to protect applications and increase visibility for Go applications. Interested in learning more about Sqreen or trying the Go agent on your own? Take Sqreen for a spin yourself.

Stay tuned for the upcoming second part about the actual Go internals and implementation details!