Naming is hard, semantics are hard(er): Inside Sentry Conventions
Sentry is fully committed to attributes. Today, logs, metrics, and spans have attributes. Maybe even more telemetry in the future. Almost everything you search, filter, group, or alert on in Sentry is just stored as an attribute.
You might wonder, how on earth do we keep them aligned? How does a Sentry SDK know which attribute to send when, and what information to send in it? How does the product know how it should work with, or show a specific attribute? How do we avoid ambiguity or divergence, on a semantic or also syntactic level? How do we deal with old SDKs sending different attributes than newer SDKs?
A couple of years ago, attributes (formerly known as span data) were used more sparsely and we were far from aligned. Our SDKs sent what they thought was right, mostly based on our SDK development guidelines, but often also on judgment about what fit the respective language, framework, or library best. This led to quite a bit of divergence across our ~100 SDKs, and sometimes even across instrumentations within the same SDK.
For you as a Sentry user, this matters more than it might seem at first. Say you query on http.request.method, but one of your services still runs an older SDK that sends an outdated http.method attribute instead. Without some help from our side, you’d simply miss that data. With more features being built around attributes and a much richer product surface to explore your data, we learned that there was a need for more alignment and for handling these cases better. We’re not yet fully there, but over the last year, we got a lot better, thanks to Sentry Conventions.
What are Sentry Conventions?
A while ago, OpenTelemetry started standardizing attributes set on spans to define the proper meaning and semantics behind them. The idea was to specify a set of attributes with specific names and what information these should carry. This produced a large database of syntactic but more importantly semantic rules called OpenTelemetry Semantic Conventions. Today, they not only specify rules for attributes, but also for naming of and data on metrics, logs, and spans. They are the source of truth for OTel instrumentation.
About two years ago, we introduced Sentry Conventions. The purpose was clear from the start: Sentry Conventions are a superset of OTel. They extend OTel’s database, introducing Sentry- or instrumentation-specific attributes and concepts on top. The goal is to provide a standardized set of semantic specifications across our SDKs, our ingestion pipeline, and the product.
Or in other words, if an SDK sends attribute X, our ingestion pipeline knows how to work with it and the product can rely on the value specification, all defined in conventions.
Why not just use OTel’s conventions? Some data is just very specific to Sentry (span ops for example). They wouldn’t make sense to be brought into OTel. So we roll our own but base them on OTel. We’re in good company with this, as some other observability vendors do the same.
What’s in Sentry Conventions?
Today, Sentry Conventions specify:
- Attributes set on our telemetry. Not just the name and type but also a bunch of useful metadata we use across the stack:
- Data scrubbing policy, defining whether an attribute should be scrubbed automatically, when users configure scrubbing rules for it, or never (if critical functionality depends on it).
- Deprecation rules, marking an attribute as deprecated, listing a replacement and strategies for migrating from the old to the new attribute.
- Descriptions, types, a changelog holding changes to the specification, and an indication if the attribute is OTel-conforming or invented by us.
- Well-known values for attributes that must be set to specific values. For example,
gen_ai.operation.name.
- Span
ops telling SDKs which span operations to set on specific spans. - Span name and description rules, specifying how names and descriptions of certain spans should be constructed.
- Metric specifications, how a metric is called, what it measures, and which attributes it should carry.
We went from roughly 300 attributes a year ago, to more than 900 attributes today. A lot of this came from backfilling attributes into conventions that our SDKs already sent. While doing this, we were able to align on one attribute if SDKs sent different versions of the same information. We’re not at 100% coverage yet, but much closer to it than a year ago.
Besides keeping the specs, we also surface them in different ways:
- We publish NPM and PyPI packages, as well as Rust code, with all (well, most) of the data.
- We publish a Sentry Conventions website which can be used to search for and look up conventions. Some of us use this daily. It has a nice search, LLM-friendly versions of every page, and, of course, dark and light mode 😎 (or in other words, all the essential features a developer needs).
How conventions flow through Sentry
Every attribute passes through a few stages before you see it in Sentry. Conventions are involved in all of them.
SDKs
Starting with JavaScript, our SDKs use the conventions packages to set attribute names and span ops. This has some key advantages over hard-coded strings or locally-defined constants: We’re aware if an attribute gets deprecated and can switch to the new attribute in the next SDK major version. We also avoid accidentally shipping typos or incorrectly spelled attributes (this NEVER happened 👀).
For the JS SDKs, it’s highly important to minimize bundle size impact. So we made sure the conventions package doesn’t blow up your bundle before adding it as a direct dependency of all our JS SDK packages.
Relay
Relay, our ingestion service, uses conventions for:
- Applying data scrubbing rules on attributes
- Inferring span descriptions for streamed and OTLP-ingested spans
- Inferring span names for transaction-based spans
The last two need a bit of explanation. Streamed spans, as well as OTLP-ingested spans, have a name but no description. Relay uses available attributes on a span to infer a span description for them, so that we can still show a high entropy, descriptive … description in the product. Transaction-based spans have a description but no name. Relay does the opposite here, and infers a span name for them to have a low-cardinality, group-able name for every span.
Querying
Snuba, our storage and query layer, and the Sentry backend use conventions to coalesce deprecated and current attributes. When you query an attribute, you get results whether the SDK sent the deprecated name or the current one. This works in both directions: querying the current or the deprecated name returns the same results. So in the example from above, querying on http.request.method also finds data that older SDKs sent as http.method.
The backend also uses conventions to return user-facing attribute names when they differ from the ones the SDK sent, and to hide attributes we marked as internal.
The product
In Sentry, conventions power search features in Explore, like descriptions of attributes and attribute types (so that you can apply numeric filter conditions for numbers). They also allow us to build deprecation-safe features. For example, if a feature depends on an attribute that was newly deprecated, conventions expose a deprecation chain, so that we can look up any of the deprecated or current keys for this attribute in an attribute bag. That feature can even depend on a deprecated version itself, and it would still work.
What we learned
Looking back at the last year, a few things stood out to us:
- Specify first, ship second. Today, we first add new attributes to conventions, wait for agreement on the naming and semantics, and only then send them in SDKs. This took some getting used to, but it saves us a lot of back and forth later.
- Deprecations are part of the spec. A deprecated attribute in conventions lists its replacement and a strategy for migrating. This way, ingestion, querying, and the product can handle old and new attributes for you.
- Conventions can be tested. This is a huge advantage over conventional docs or just written rules. A metric spec, for example, must reference only attributes that already exist. Also, we can flag when a spec references a deprecated attribute, which ideally it should not. We also test attribute deprecation statuses and strategies and type adherence, and we check that conventions updates don’t conflict with each other.
- Migrating is painful, but worth it. More often than not, we were quite “liberal” in how we named or described spans or how we set attributes. The process to move towards standardized names, descriptions, and attributes is quite painful (I speak from experience migrating span names in the JavaScript SDK) but we think it’s worth it in the end and it leads to a more consistent product experience.
Closing out
To briefly sum up, conventions help us to deal with semantics across the stack:
- We know what data must (not) be scrubbed
- We can build querying and search experiences that handle deprecated and current attributes seamlessly
- We can reason about semantics and when something needs to change
- We move towards a future where SDKs get more aligned, and diverge less in sent data and semantics around them
Conventions are very much a team effort, with many folks across Sentry contributing, be it with the repo and the model definitions, with adopting conventions across the stack, or with contributing data.
Naming is hard. Semantics are hard(er). Conventions solve neither the second hardest problem in computer science, nor do they make semantics standardization any easier themselves. But they give us a source of truth, a room to talk about them, and a platform to rely on them. Eventually, we’ll only have to think hard about names and semantics once, and then regret them later.