Troubleshooting Spring Boot Applications with Sentry

Some months ago we wrote a quick guide on how to use Sentry with Spring Boot and Logback. Since then, we've continued working on improving the development experience, added several features for error monitoring and reporting, and, most importantly, implemented the performance feature in Sentry Java SDK with a dedicated integration with Spring MVC. If you haven't yet used Sentry in a Spring Boot application - nothing to worry about - you will find all the necessary steps below.

How to configure Sentry Spring Boot Starter

Sentry’s Java SDK is split into several modules so that you pull only the dependencies you need. For Spring Boot users, the two most interesting ones are a sentry-spring-boot-starter and an integration with a logging framework of your choice - either sentry-logback in case you use Logback, or sentry-log4j2 for an integration with Log4j2 framework.

It's important to always use the same versions for all Sentry dependencies - as mixing them can cause unexpected issues, potentially discoverable only in runtime. To simplify version configuration, we've added something called a Bill Of Materials module (concept you may be familiar with if you use Spring Cloud or Testcontainers) - sentry-bom - once included, ensures that all Sentry-related dependencies are in sync.

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.sentry</groupId>
            <artifactId>sentry-bom</artifactId>
            <version>5.6.1</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
		<dependency>
        <groupId>io.sentry</groupId>
        <artifactId>sentry-spring-boot-starter</artifactId>
    </dependency>
		<!-- to use Logback -->
    <dependency>
        <groupId>io.sentry</groupId>
        <artifactId>sentry-logback</artifactId>
    </dependency>
		<!-- to use Log4j2 -->
    <dependency>
        <groupId>io.sentry</groupId>
        <artifactId>sentry-log4j2</artifactId>
    </dependency>
</dependencies>

Once the dependencies are there we can move on to configuration. Sentry Spring Boot Starter uses Spring's environment abstraction to retrieve configuration properties - which means that it can be configured either through an [application.properties](http://application.properties) file, Java system properties, environment variables, or a 3rd party service providing configuration to a Spring Boot application, like Vault or AWS Secrets Manager. Either way, if you've ever run a Spring Boot application in production - nothing will surprise you there.

While there are plenty of configuration options, the only required one is to provide a sentry.dsn that you can copy from your project settings in the Sentry Dashboard.

# application.properties
# replace dsn with a dsn specific for your Sentry project
sentry.dsn=https://b687180f91ef43e0908351beac84123a@o420886.ingest.sentry.io/6090909

The Sentry SDK is meant to be invisible for your project - to not affect the performance or disturb your application logs. But, especially at the beginning when you just want to make sure that integration works as expected, it is advised to turn on Sentry SDK debug logs to find out if the SDK is up and running and events are actually sent to Sentry's backend. To do so, just set sentry.debug property to true.

# application.properties
sentry.debug=true

Now, every unhandled exception in a Spring MVC flow is converted into a Sentry event and sent to Sentry backend. Same thing happens to every error log statement.

If you've paid attention - you noticed that we did not touch logging configuration at all, so how does it work that an error log is captured by Sentry? Sentry Spring Boot Starter provides an auto-configuration for Logback - once it finds out that both logback and sentry-logback are in the classpath, it will configure SentryAppender with some sane defaults - all error logs are converted to Sentry events, all log statements with the level lower than error are converted into breadcrumbs - which get added as a metadata to an actual error - if one happens.

While this configuration is convenient for many projects, if it does not suit your needs, you can configure what gets converted into an event and what into breadcrumb by changing properties:

# application.properties
sentry.logging.minimum-event-level=info
sentry.logging.minimum-breadcrumb-level=debug

When you need more sophisticated appender configuration, you can also configure appender in a logback.xml file manually, and auto-configuration will just skip this step. You can also completely disable SentryAppender configuration:

# application.properties
sentry.logging.enabled=false

How to Configure SentryOptions

SentryOptions is a central object to configure Sentry behaviour. In fact, when you set the sentry.dsn or sentry.debug property, it actually gets set on SentryOptions. There are plenty of configuration options. We will focus today only on those that are most important for a successful project integration.

• sentry.environment - in which environment does your application run. By default, we set it to "production" to be able to easily distinguish between events coming from your other environments like QA or staging. Make sure to give this property a meaningful value

# application.properties
sentry.environment=staging

• sentry.release - the release version of your application. Once it’s there, you can see which release introduced an error or degraded performance. It can take any string as a value - for applications released with real version numbers - it's a no-brainer - but many applications don't use any kind of versioning, and the actual version is a Git commit id. Fortunately, you don't need to set it manually.

To get commit id assigned automatically as a release, include this plugin:

<build>
    <plugins>
        <plugin>
            <groupId>pl.project13.maven</groupId>
            <artifactId>git-commit-id-plugin</artifactId>
        </plugin>
    </plugins>
</build>

Gradle users can achieve the same result by using the gradle-git-properties plugin:

plugins {
    id "com.gorylenko.gradle-git-properties" version "2.3.2"
}

If configuring SentryOptions via simple values does not fit your use case, you can also configure SentryOptions programmatically by registering custom Sentry.OptionsConfiguration<SentryOptions> bean:

@Bean
Sentry.OptionsConfiguration<SentryOptions> customOptionsConfiguration() {
    return options -> {
        if (...) {
            options.setSampleRate(..);
        }
    };
}

How to Filter Exceptions in Sentry

Not every error is equal so now you can easily filter out exceptions that you don’t want to be sent to Sentry:

sentry.ignored-exceptions-for-type=com.myapp.ServiceException,com.myapp.AppException

Again, if your filtering logic is more complex, you can register a bean of type BeforeSendCallback. If this function returns null, an event will be excluded.

@Bean
SentryOptions.BeforeSendCallback beforeSendCallback() {
    return (event, hint) -> {
        if (...){
            return null;
        }
        return event;
    };
}

Spring MVC integration

Sentry's SDK shines especially with Spring MVC, where each event reported to Sentry is automatically decorated with all relevant contextual data about HTTP request that triggered this event: URL, request parameters, headers – everything except the request body.

Request body can be attached to events only if request has a content type application/json and requires configuring property sentry.max-request-body-size to one of values:

• SMALL - where body size is smaller than 1000 bytes

• MEDIUM - where body size is smaller than 10000 bytes

• ALWAYS - where no matter what, body content is attached to event

It is important to note that attaching request body requires loading it to memory - use this option wisely.

Since body can contain sensitive personal information, the Sentry SDK has to be configured with sentry.send-default-pii=true

Performance Monitoring

In addition to seamless error reporting experience, Sentry SDK for Java also offers application performance monitoring capabilities with a goal to find out what and how much contributes to time needed to execute whatever your application does. If you are new to performance monitoring topic, I suggest taking a look into a Sentry Performance Monitoring docs to get familiar with general concepts.

To turn performance monitoring on, Sentry SDK has to know what percentage of potential traces should be sent to Sentry - which likely depends on how much traffic your application takes. This can be configured either using a simple property sentry.traces-sample-rate which accepts values between 0.0 and 1.0 - where 1.0 means 100% of traces will be sent to Sentry.

# application.properties
sentry.traces-sample-rate=1.0

If sample rate is more dynamic, for example depends on the URL, you can also register a bean of type TracesSamplerCallback that executes sample logic for each incoming request:

@Bean 
SentryOptions.TracesSamplerCallback tracesSamplerCallback() {
    return context -> {
        CustomSamplingContext customSamplingContext = context.getCustomSamplingContext();
        if (customSamplingContext != null) {
            HttpServletRequest request = (HttpServletRequest) customSamplingContext.get("request");
            // return a number between 0 and 1 or null (to fallback to configured value)
        } else {
            // return a number between 0 and 1 or null (to fallback to configured value)
        }
    };
}

Out of the box we ship seamless integration with Spring MVC, where each incoming HTTP request is turned into a transaction. Transactions can be further broken down into spans - where each span identifies time needed to execute a certain part of the request execution - requests to external systems, executing database queries or just calling beans methods.

RestTemplate instrumentation

Each RestTemplate bean, created through a RestTemplateBuilder is automatically instrumented to create span for each outgoing HTTP request.

@Bean
RestTemplate restTemplate(RestTemplateBuilder builder) {
  return builder.build();
}

Spans contain additional metadata like request and response size. Intercepter requests get extra header, sentry-trace enabling downstream services integrated with Sentry SDK to continue a trace.

JDBC instrumentation

Very often, a majority of the time needed to execute a request is spent on waiting for a database server to execute a query. With sentry-jdbc module in place, each query is turned into a span so you can easily identify which queries need tuning.

Under the hood, sentry-jdbc uses P6Spy, a mature framework for intercepting JDBC activity. There are few steps you need to take to enable JDBC tracing:

1. Add a dependency to sentry-jdbc module

<dependency>
    <groupId>io.sentry</groupId>
    <artifactId>sentry-jdbc</artifactId>
</dependency>

1. Include p6spy prefix in JDBC url:

spring.datasource.url: jdbc:p6spy:postgresql://localhost:5432/db

1. Change database driver:

spring.datasource.driver-class-name: com.p6spy.engine.spy.P6SpyDriver

1. Disable P6Spy log file generation - create a file in src/main/resources/spy.properties with content:

modulelist=com.p6spy.engine.spy.P6SpyFactory

Bean Instrumentation

Every Spring bean method execution can be turned into a transaction or a span. To enable this feature, you must include spring-boot-starter-aop in your application:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

Creating span from a method execution is as simple as annotating a bean method with @SentrySpan:

@Component
class PersonService {

  @SentrySpan
  Person findById(Long id) {
    ...
  }
}

Keep in mind that spans can be created only if they run within a transaction - which is provided for Spring MVC requests by default in sentry-spring-boot-starter. What about non-spring-mvc use cases, like scheduled jobs or handling messages from message brokers? It depends.

Similar to @SentrySpan, each method can be annotated with @SentryTransaction that would mark the execution of the method as a transaction which can be broken further down into spans. This works flawlessly for scheduled jobs or messaging, where message processing is not considered as a continuation of an ongoing transaction started in another service. If a transaction is meant to continue a trace, we need to write some code.

As an example, I will use Spring AMQP integration with RabbitMQ, but the principles stay the same for every messaging system capable for passing message headers.

To continue a trace, the message has to have a Sentry Trace header. When sending a message, we can obtain the current trace header by calling Sentry.traceHeaders()

Message message = MessageBuilder
        .withBody(rating.getBytes(StandardCharsets.UTF_8))
        .setHeader(SentryTraceHeader.SENTRY_TRACE_HEADER, Sentry.traceHeaders().getValue())
        .build();
rabbitTemplate.send("score-updates", message);

On the listener side we must retrieve the value of this header and start the transaction manually. We must also take into account that the header may have an invalid value - in such case, InvalidSentryTraceHeaderException is thrown and we must decide to start a completely new transaction or not to start a transaction at all. In the following example, if the header has an invalid value, we will start a new transaction.

@RabbitListener(queues = "queueName")
void handleMessage(String payload, @Header(SentryTraceHeader.SENTRY_TRACE_HEADER) String sentryTrace) {
    ITransaction transaction;
    try {
        // continue existing transaction from given sentry trace header
        transaction = Sentry.startTransaction(
                TransactionContext.fromSentryTrace("transactionName", "operation", new SentryTraceHeader(sentryTrace)), true);
    } catch (InvalidSentryTraceHeaderException e) {
        // start new transaction if sentry trace header is invalid
        transaction = Sentry.startTransaction("transactionName", "operation", true);
    }
		...
}

Once the transaction has started, we need to make sure that it always finishes, no matter if the execution ended with an exception or finished successfully:

ITransaction transaction = // ...
try {
    // ...
		transaction.setStatus(SpanStatus.OK);
} catch (Throwable e) {
    transaction.setThrowable(e);
    transaction.setStatus(SpanStatus.INTERNAL_ERROR);
    throw e;
} finally {
    transaction.finish();
}

The above code is a boilerplate that you likely don’t want to have in your listener methods. Let's refactor it into a reusable helper class:

class SentryTransactionHelper {
	static void runInTransaction(String sentryTrace, String transactionName, String operation, Consumer<ITransaction> runnable) {
	    ITransaction transaction = startTransaction(sentryTrace, transactionName, operation);
	    try {
	        runnable.accept(transaction);
					transaction.setStatus(SpanStatus.OK);
	    } catch (Throwable e) {
	        transaction.setThrowable(e);
	        transaction.setStatus(SpanStatus.INTERNAL_ERROR);
	        throw e;
	    } finally {
	        transaction.finish();
	    }
	}

	private ITransaction startTransaction(String sentryTrace, String transactionName, String operation) {
	    try {
	        // continue existing transaction from given sentry trace header
	        return Sentry.startTransaction(
	                TransactionContext.fromSentryTrace(transactionName, operation, new SentryTraceHeader(sentryTrace)), true);
	    } catch (InvalidSentryTraceHeaderException e) {
	        // start new transaction if sentry trace header is invalid
	        return Sentry.startTransaction(transactionName, operation, true);
	    }
	}
}

Now the actual listener code that should be wrapped in the transaction looks very clean and readable:

@RabbitListener(queues = "queueName")
void handleMessage(String payload, @Header(SentryTraceHeader.SENTRY_TRACE_HEADER) String sentryTrace) {
    SentryTransactionHelper.runInTransaction(sentryTrace, "transactionName", "amqp", transaction -> {
				// metadata can be added to transaction
        transaction.setData("payload", payload);
				// code that handles the message
    });
}

Other integrations for Java SDK

Along with Sentry Spring Boot Starter, the Java SDK contains integrations with Log4j2, JUL, traditional non-Spring servlet applications, OpenFeign, Apollo GraphQL client, GraphQL Java library (the foundation for Spring GraphQL and Netflix DGS). Head over to Java SDK docs to learn all the details.