Continuous Profiling for Java applications

Profiling a Java Rideshare App with Pyroscope

Note: For documentation on Pyroscope's java integration visit our website for java. It's also worth noting that Pyroscope's java integration is powered by our JFR parser.

Background

In this example we show a simplified, basic use case of Pyroscope. We simulate a "ride share" company which has three endpoints found in RideShareController.java:

• /bike : calls the orderBike(searchRadius) function to order a bike
• /car : calls the orderCar(searchRadius) function to order a car
• /scooter : calls the orderScooter(searchRadius) function to order a scooter

We also simulate running 3 distinct servers in 3 different regions (via docker-compose.yml)

• us-east
• eu-north
• ap-south

One of the most useful capabilities of Pyroscope is the ability to tag your data in a way that is meaningful to you. In this case, we have two natural divisions, and so we "tag" our data to represent those:

• region: statically tags the region of the server running the code
• vehicle: dynamically tags the endpoint

How to create static tags

Tagging something static, like the region, can be done in the initialization code in the main() function:

public class Main {
    public static void main(String[] args) {
        Pyroscope.setStaticLabels(Map.of("REGION", System.getenv("REGION")));

        [ all code here will be attatched to the "region" label ]
    }
}

How to create dynamic tags within functions

Tagging something more dynamically, like we do for the vehicle tag can be done inside our utility OrderService.findNearestVehicle() function using pyroscope.LabelsWrapper

Pyroscope.LabelsWrapper.run(new LabelsSet("vehicle", vehicle), () -> {
    [ all code here will be attatched to the "vehicle" label ]
});

What this block does, is:

1. Add the label new LabelsSet("vehicle", vehicle)
2. execute the code to find the nearest vehicle
3. Before the block ends it will (behind the scenes) remove the LabelsSet("vehicle", vehicle) from the application since that block is complete

Resulting flame graph / performance results from the example

Running the example

To run the example run the following commands:

# Pull latest pyroscope and grafana images:
docker pull grafana/pyroscope:latest
docker pull grafana/grafana:latest

# Run the example project:
docker-compose up --build

# Reset the database (if needed):
# docker-compose down

What this example will do is run all the code mentioned above and also send some mock-load to the 3 servers as well as their respective 3 endpoints. If you select our application: rideshare.java.push.app.itimer from the dropdown, you should see a flame graph that looks like this (below). After we give the flame graph some time to update and then click the refresh button we see our 3 functions at the bottom of the flame graph taking CPU resources proportional to the size of their respective searchRadius parameters.

Where's the performance bottleneck?

The first step when analyzing a profile outputted from your application, is to take note of the largest node which is where your application is spending the most resources. In this case, it happens to be the orderCar function.

The benefit of using the Pyroscope package, is that now that we can investigate further as to why the orderCar() function is problematic. Tagging both region and vehicle allows us to test two good hypotheses:

• Something is wrong with the /car endpoint code
• Something is wrong with one of our regions

To analyze this we can select one or more tags from the "Select Tag" dropdown:

Narrowing in on the Issue Using Tags

Knowing there is an issue with the orderCar() function we automatically select that tag. Then, after inspecting multiple region tags, it becomes clear by looking at the timeline that there is an issue with the eu-north region, where it alternates between high-cpu times and low-cpu times.

We can also see that the mutexLock() function is consuming 76% of CPU resources during this time period.

Comparing two tag sets with FlameQL

Using Pyroscope's "comparison view" we can actually select two different sets of tags using Pyroscope's prometheus-inspired query language FlameQL to compare the resulting flame graphs. The pink section on the left timeline contains all data where to region is not equal to eu-north

REGION != "eu-north"

and the blue section on the right contains only data where region is equal to eu-north

REGION = "eu-north"

Not only can we see a differing pattern in CPU utilization on the timeline, but we can also see that the checkDriverAvailability() and mutexLock() functions are responsible for the majority of this difference. In the graph where REGION = "eu-north", checkDriverAvailability() takes ~92% of CPU while it only takes approximately half that when REGION != "eu-north".

Visualizing diff between two flame graphs

While the difference in this case is stark enough to see in the comparison view, sometimes the diff between the two flame graphs is better visualized via a diff flame graph, where red represents cpu time added and green represents cpu time removed. Without changing any parameters, we can simply select the diff view tab and see the difference represented in a color-coded diff flame graph.

Future Roadmap

While this is one popular use case, the ability to add tags opens up many possibilities for other use cases such as linking profiles to other observability signals such as logs, metrics, and traces.

We've already began to make progress on this with our otel-pyroscope package for Go... Stay tuned for a version with Java coming soon!

We'd love to continue to improve our java integration and so we would love to hear what features you would like to see.