Developer-friendly Security Reporting

Reporting application security vulnerabilities is an awful thing to do to someone. If you want a primer on how to communicate security effectively, check out this article I wrote based on 20 years (of mistakes) in the field.

Communicating Security with Developers via Test Failures

This article is about a specific idea. Rather than generate a massive PDF report, or a zillion JUnit tickets, or a Slack alert volcano, or yet another "single" pane of glass... what if we could simply fail any test cases that cause code with a security problem to run. This is a very natural and non-threatening way to let developers know there's a problem and exactly what to do about it. The goal is to help developers fix vulnerabilities as part of their natural workflow -- without having to run scans and triage reports -- so they can check in clean code.

So, let's try a very simple example. Let's say you want to ban starting native processes in Java. Starting native processes isn't the most dangerous thing in the world, but whatever -- it's *your* coding standard -- this is about how you can effectively automate a broad range of security policies. You could simply grep the code, but the call might be buried deep in a library or other component. It might get loaded at runtime via dependency injection. And anyway, there are quite a few method signatures to search for, so you may not find them all.

Using the Java Observability Toolkit (JOT)

For this article, we're going to use the Java Observability Toolkit (JOT). It's a free and open source library that allows you to create security sensors without writing any code. Sort of a voltmeter that lets you put probes deep into your application so that you can see what's going on way down the stack -- all the way down into libraries, frameworks, appserver, and runtime. It's a very useful tool you can use for a lot of different purposes.

First, we need to create a JOT file with a single sensor (junit.jot). It's just a bit of YAML that defines where to instrument and what the sensors should do. In this case, the only "method" we need to list is the java.lang.ProcessBuilder.<init> constructor -- this is where calls like Runtime.exec() end up. You can add as many banned methods as you want. Then we "scope" this sensor, so it will only fire when the methods are invoked somewhere inside a JUnit test. And finally, we define an "exception" which will cause the JUnit test to error out and tell the developer why.

sensors: 
- name: "banned-methods"
  description: "Fails JUnit tests that cause banned methods to be invoked"
  methods:
  - "java.lang.ProcessBuilder.<init>"
  scopes:
  - "org.junit.platform.commons.util.ReflectionUtils.invokeMethod"
  exception: "Banned methods are prohibited by Security Directive 27B/6"

Then you'll need an application to test. I have a simple test case here that directly calls Runtime.exec() which in turn calls ProcessBuilder deep in the Java runtime. But the call to Runtime.exec() can happen anywhere inside any code that runs as part of this test and this sensor will find it just fine. For example, if this was an end-to-end test that submits a mock HTTP request to a servlet, JOT will ensure that any calls to create native processes are caught no matter where in that transaction they occur.

@Test
void executeRuntime() throws IOException {
     String cmd = "open -a Calculator";
     Runtime.getRuntime().exec( cmd );
}

To run your application with JOT you can download the latest version of the JOT library from github. Probably the easiest way to add JOT to your application is to just use the JAVA_TOOL_OPTIONS environment variable. Then just run your test suite as normal.

$ export JAVA_TOOL_OPTIONS="-javaagent:jot-0.9.1.jar=junit.jot"

$ mvn test

That's all it takes. In this case, you can see the test case resulted in an error and lets the developer know exactly what line of code was responsible. Also, the message helpfully references the regulation from the movie Brazil. Probably would be more useful to let them know what they should do instead. Just update your JOT with whatever message you like.

More Sophisticated Sensors

There's no much that you can't make observable with JOT. You could flag transactions that don't make an access control check. Or that use a weak encryption algorithm. You can see any backend connections that are made. Or which APIs consume credit-card information.

You can use "captures" to be very specific about exactly when a sensor will fire. You can filter on the value of the Object, Parameters, and Return of the "methods" listed. Or you can use the full power of Spring Expression Language to invoke methods on those objects (or any static methods) -- and filter on whatever that returns. All this so that your sensors will only fire exactly when you want them to.

You can learn more about JOT from the project homepage. There are more examples of what you can do with JOT in the resource library. You can make any of those JOTs fail test cases by adding the JUnit "scope" and an "exception" as we did above. Let me know what you use it for!