Sentry Blog

New Sentry Pricing: Outcomes and Observability for Everyone

2017-05-25T00:00:00-07:00

Like your team, Sentry moves fast. We eat our own dog food, drink our own champagne, take our vitamins, and monitor for errors. We’re obsessive about building the most important features and enabling the best possible experience to serve every developer everywhere.

One of the great benefits of having tens of thousands of Sentry users sending billions of data points every month is that we’re able to see a massive variety of events and patterns, including those that are a reflection of usage and workflows, not just code errors and crash reports.

So, in the spirit of making Sentry as easy to adopt and scale as possible for all types of users, we’re rolling out new outcomes-oriented pricing plans. Wait, what the heck?

Counting Events is the Biggest Barrier to Success

Yeah, that’s right: outcomes. The more attention we’ve paid to customer use cases, the more we’ve come to understand that counting events is the biggest barrier to success. Ultimately, the only number of events that aligns to every objective for error monitoring is MORE. Flexibility is among the most important features Sentry offers, and packaging to enable greater flexibility at lower total cost is the only way to ensure every Sentry user can drive the outcomes they care about, from faster issue resolution to proactive triage to root cause targeting to reducing technical debt.

In order to help drive the goal of outcomes, the new plans are designed to give you greater observability and let you track more errors without worrying as much about the cost of each event. The decisions you make about monitoring, as well as how you align product development priorities, relies on seeing things that you didn’t previously know were important. So, if better resolution workflows are a priority, there’s a plan with the features you need to support them. If easier event allocation and management is important, the new pricing plans make it easy to prioritize that, too.

What that actually means: Sentry users have told us (and we’ve seen) that they run out of events before diving into key use cases or adopting the workflows that help prevent, not just detect, errors. You love that it’s open-source and that it’s simple to get started, but guessing how many events you’ll need for the next month or year can actually compound the problem you set out to solve in the first place. Setting the bar for events too low sucks, especially when you really start to achieve developer efficiencies and want to scale.

Discovering More Reasons to Monitor & Setting More Interesting Goals

Better error monitoring makes your entire team more effective and able to focus on improving your product so that more users use it more. That means not only needing more events as you scale, but also discovering more reasons to monitor and setting more interesting goals. If you’re worried about hitting your events cap, and if each additional event carries a marginal cost, that uncertainty forces you to take your eye off the ball.

With outcomes-oriented pricing, you’re allotted more events than your first, second, third, fourth, and fifth use cases will likely require without any of the extra expense. As your monitoring starts to depend on more breadth and structure, Sentry levels-up on event bandwidth along with the features you need, like our export API, data forwarding, and per-project rate limiting. Plus, with annual discounts under contracts (versus having to pay in advance), it’s way easier to adopt a plan that suits your development roadmap, not blocked by typical up-front billing hurdles.

So, please welcome bigger, better error monitoring into your dev stack with Sentry. Still open source and easy to use, but now with plans assured to give you more events and features so your team can focus on building awesome products and gaining more insight, not counting bugs.

Welcome Elizabeth Campbell

2017-05-19T00:00:00-07:00

We’re excited to welcome Elizabeth Campbell (a.k.a. EC$$) to team Sentry as head of BD.

EC will work with partners to grow and extend Sentry’s reach. Before Sentry, EC was at Dropbox and Facebook, where she worked on strategic partnerships, operations, and everything in between.

Outside of work, you can find EC reading way too much Politico and sprinting through an airport near you. Right now, she’s listening to Ludacris’s freestyle of “Llama Llama Red Pajama” and dares you to tell her this is not art.

PyCon After-Party: Eat! Drink! Mingle and Be Merry!

2017-05-16T00:00:00-07:00

We are hosting an after party at Pycon together with Linode. Come hang out and join us for some food, drinks, games and fun with the community. This event will be one for the books, we look forward to seeing you all there!

When

Friday, May 19th at 6:40 PM—8:40 PM

Where

Spirit of 77, 500 NE Martin Luther King Jr Blvd, Portland, OR 97232

Please bring your badge and expect to comply with the PyCon Code of Conduct.

Automatically Get Commits With Heroku Releases

2017-05-15T00:00:00-07:00

Tracking Heroku releases with Sentry just got a whole lot better.

With the deploy hook configured, you can track releases of your Heroku app in Sentry and see new errors as they arise. Now, with our expanded Releases feature, Sentry can automatically retrieve the relevant commit data associated with your release each time you deploy a new version. When you enable this feature, you also gain access to deploy emails so that relevant team members can be alerted immediately when their code has gone out.

Having access to this commit data for a release can be extremely helpful in debugging new errors, as it adds the ability to see which files have changed most recently, and which members of an organization have been involved in pushing code since the last release.

If you’re using Heroku to deploy your app and want to take advantage of release tracking with Sentry, just follow these steps:

Connect your repository to your Sentry organization so that we can automatically retrieve your commit data.
Enable the Heroku integration in your Sentry Project Settings.
In the Heroku Plugin Configuration, specify which repository and deploy environment to be associated with your Sentry project.
Navigate to your Project’s Release Tracking settings and copy the deploy hook command to your Heroku config.

And voilà! You’ll immediately start getting rich commit information and deploy emails with each new release.

Sentry Turns 9!

2017-05-12T06:00:00-07:00

Today marks nine years since the very first commit to Sentry — or as it used to be called, django-db-log.

Sentry was born to solve a very simple problem: within a Django project, which normally sends you an email for every error, how do we log those to the database and expose aggregate views? This was essential for any high traffic application due to various failure scenarios overwhelming your inbox. While you could achieve the similar results with logs, this also helped us prioritize which issues were the most important, often by how frequently they were occurring. Additionally it gave us immediate access to that data via a web UI.

Over time we expanded the scope of Sentry. It started as simple aggregation in Django, then expanded into collecting things like stack locals and richer context. Due to community demand we eventually brought it into the larger Python ecosystem, removing the Django requirement. As the months and years went by, many of the core contributors grew into other roles at new companies, and often those companies weren’t using Python at all. This was the trigger that really lead us to our initial cross platform functionality, and started with support for languages like Ruby (via Chef) and JavaScript (via Mozilla).

The biggest change of course was when we spun out the cloud service – sentry.io – and began to put real investment into the project. For the first few years of it’s life we used the revenue from the service to fund community events (ever been to karaoke with us at PyCon?), but it was still very much a straight forward mixed open source plus cloud service. If you didn’t want to host it yourself, we did it for you for small fee.

Today we have massive technology companies like Stripe and Uber running Sentry behind their firewall, but equally large organizations like Dropbox and Hubspot trusting our cloud. This part of the story has been interesting, as we’ve hit a point where very capable engineering organizations use both our cloud as well as open source offerings, simply because they wanted to outsource part of the burden. All in all, Sentry has continued to grow and saturate the market, to the point where you’d be hard pressed to find a technology company that doesn’t rely on it, let alone a reason to choose a different product.

Looking at where things have ended up, it’s crazy to think about the shape of things just a couple of years ago. Sentry the company is now 30 full-time employees spread across San Francisco, Austin, and Vienna. While we’re still heads down building the core of Sentry, our ambitions continue to grow. A lot of investment lately has been into many things that are under the hood, such as cross-language call stacks and first class deploy tracking. While Sentry may look and feel similar to what it was almost a decade ago, under the hood we’ve become so much more.

Our story is far from over, and if you’re interested in taking part in that journey, we’d love to talk to you!

@ Your Teammates With Mentions

2017-05-12T05:00:00-07:00

Collaboration is a major 🗝️ for success. You can now mention other members in issues, making teamwork even easier. Use @ to tag team members - they’ll receive an email and be subscribed to that issue.

Any mentioned user will be able to see why they are getting notifications in the body of the emails they receive:

In your issue dashboard you can do a quick search for subscribed:me to see which issues are you subscribed to, any with a little green comment icon will be issues you are mentioned in.

You can manually unsubscribe from any issue by selecting Unsubscribe Me. (a.k.a Don’t @ me)

Welcome Ryan Goldman

2017-05-12T00:00:00-07:00

We’re excited to welcome Ryan Goldman to the Sentry team as VP Marketing.

Ryan most recently led marketing at SignalFx and has built or contributed to teams at Mixpanel, Cloudera, and Cisco. He’ll be helping us find new ways to reach and serve the developer community in order to put a new Sentry in every home.

Prior to marketing for tech, Ryan managed international development projects for NGOs, moonlighted as a music journalist, and created a social network for trading mixtapes. He also likes to read, run, and volunteer with animals.

Keep an Eye on New Code With Deploys

2017-05-09T05:00:00-07:00

No matter what review process you have, going live is the ultimate test for new code. With Sentry’s new Release features, you’ll have the tools you need to confidently guide deploys across the finish line.

When Sentry is aware of the timing and scope of your deploys, it will send a rich deploy emails to the involved parties detailing how many files have changed and what commits are being released.

Deploy tracking isn’t just for production - the extra context will allow you to track what commits go to what environment at what time, giving you the complete picture of a changeset’s lifecycle.

The email also links you to the release summary for each project affected, which will give you a breakdown of new issues and resolved issues for your release – everything you need to be confident your code is running smoothly, or to be ready to triage unforeseen problems.

Sending Deploy Information

Deploys in Sentry work together with the Release features you unlock by integrating commit data. If you already tag events with a release id in your SDK and send commit data (Release Documentation), then there’s only one step left to unlock deploy tracking and emails. Post to the Deploy API as part of your deployment process, and we’ll do the rest!

Shell script example:

# Create a new deploy
curl https://sentry.io/api/0/organizations/$ORGANIZATION_SLUG/releases/$RELEASE_VERSION/deploys/ \
  -X POST \
  -H 'Authorization: Bearer {TOKEN}' \
  -H 'Content-Type: application/json' \
  -d '
  {
    "environment": "production",
    "name": "my-deploy"
}
'

Stay tuned!

This is part two of a series of posts about the new features we’ve added to our Releases workflow. In the meantime, you can check out our docs for more details on how to get started. As always, let us know what you think on Twitter and the Sentry forum.

Welcome Dena Metili Mwangi

2017-05-09T00:00:00-07:00

Please help us welcome Dena Metili Mwangi to the team at Sentry.

Dena joins us fresh from Hackbright Academy and has a background in Economics. Most recently, she spent her time playing with data and surveys while working on impact evaluations at the World Bank. When not coding, you’ll find her at a coffee shop or playing with other people’s cats. At Sentry, she’ll be focused on growth engineering.

Releases Are Better With Commits

2017-05-01T00:00:00-07:00

Knowing what code changed recently is extremely helpful in determining the cause of an error, which is why we’ve expanded our Releases feature to support commit data. By integrating commit data, you’ll unlock a number of helpful features:

Enhanced releases overview page that allows you to see new and resolved issues, files changed and authors all in the same place (see photo below)
Resolving Sentry issues via commit messages
Suggested assignees for issues
Detailed deploy emails (look out for a more in-depth look at deploys in Sentry soon)

Connecting a repository

To get started, go to your organization’s dashboard, click “Repositories” and then click “Add Repository.” This will create a webhook and we’ll start collecting commit data, which you can then reference in releases.

Adding commits to releases

To indicate which commits are part of a release, you simply send along a refs attribute when creating a release. In the below example, we’re telling Sentry that this release contains the my-repo repository, in which the current version (HEAD) is 2da95dfb052f477380608d59d32b4ab9. When you send this information, we’ll figure out which commits are part of the release based on your git history and previous releases.

# Create a new release
curl https://sentry.io/api/0/organizations/:organization_slug/releases/ \
  -X POST \
  -H 'Authorization: Bearer {TOKEN}' \
  -H 'Content-Type: application/json' \
  -d '
  {
    "version": "2da95dfb052f477380608d59d32b4ab9",
    "refs": [{
        "repository":"my-repo",
        "commit":"2da95dfb052f477380608d59d32b4ab9"
    }],
    "projects":["my-project","my-other-project"]
}
'

Resolving issues via commits

Once you are sending commit data, you can start resolving issues by including fixes <SHORT-ID> in your commit messages. For example, a commit message might look like:

Prevent empty queries on users

Fixes MYAPP-317

When Sentry sees this, we’ll automatically annotate the matching issue with a reference to the commit, and, later, when that commit is part of a release, we’ll mark the issue as resolved.

Stay tuned

We’ll have a series of posts about the new features we’ve added to our releases workflow. In the meantime, you can check out our docs for more details on how to get started. As always, let us know what you think on the Sentry forum.

Welcome Erik Lee

2017-04-25T00:00:00-07:00

Help us welcome Erik Lee to the Sentry team!

Erik joins the Sentry team from Mixpanel and RainforestQA, where he helped customers make sense out of data and build better products, respectively. At Sentry, he’ll be working with the business team to enable customers to succeed using Sentry.

While not sipping his LaCroix, you can find him chasing his dog with his girlfriend and coming up with excuses to justify his sneaker collection.

Welcome Jan Crisostomo

2017-04-13T00:00:00-07:00

Join us in welcoming Jan Crisostomo to the Sentry team!

If you’re an engineer, you may recognize Jan’s work from such emails as, “Engineering opportunities with [Company]!”, “Just following up on my previous email”, and “PLZ DON’T BLOCK ME”. Jan has recruited and managed talent programs for tech companies like Google and Dropbox, most recently at Bessemer Venture Partners where she advised and recruited for a variety of early-stage startups.

Jan will help build the Sentry team as Head of Talent, which she finds ironic since she has no talents. When she’s not self-consciously crafting third-person biographies, Jan can be found sipping tea, traveling, and writing chapters for her inevitable cult classic The 4-Hour Nap.

The Troubles With iOS Symbolication

2017-04-11T00:00:00-07:00

Who does not love iOS? It’s a great operating system. However I can tell you about a type of person that has a love/hate relationship with iOS: engineers who have to debug crashes on iOS devices. iOS makes debugging crashes trickier than most environments which in turn makes the job of tools like Sentry and Crashlytics that much harder. In this blog post we want to give you a bit of insight into how Sentry deals with iOS crashes and what is necessary for you to have an enjoyable crash reporting experience.

Crashing in the First Place

The first part that goes into a crash is to generate a report we can actually send to Sentry. For this to work you need something that can generate you a basic backtrace at the moment a crash happens. There are two popular libraries for iOS that can do that. One is KSCrash, the other is PLCrashReporter. Those two libraries hook into different parts of the OS to respond to errors and to extract a backtrace. This in itself is already a very complex undertaking and we’re glad that others have done this task for us.

There are many different situations that can cause crashes and each of them has different characteristics. I don’t want to get too much into detail here but it’s important to understand that not all crashes will result in the same quality of reporting. An extreme case for iOS are C++ exceptions which will not create proper backtraces on iOS because of how the exception system works.

When we manage to report a stacktrace and some important data on crashing we persist that temporarily on the device. Next time you start the application we send that information to the server. The stacktrace is the most interesting part and this also is the first complexity that spills over to the server side.

To give you a bit of an idea what this looks like, here is an example stacktrace after you extract it:

CrashLibiOS                     0x100077c4c
CrashProbeiOS                   0x200050220
UIKit                           0x31d104d30
UIKit                           0x31d104cb0
UIKit                           0x31d0ef128
UIKit                           0x31d10459c
UIKit                           0x31d68f628
UIKit                           0x31d68b6c0
UIKit                           0x31d68b1e0
UIKit                           0x31d68a49c
UIKit                           0x31d0ff30c
UIKit                           0x31d0cfda0
UIKit                           0x31d8b975c
UIKit                           0x31d8b3130
CoreFoundation                  0x3111ffb5c
CoreFoundation                  0x3111ff4a4
CoreFoundation                  0x3111fd0a4
CoreFoundation                  0x31112b2b8
GraphicsServices                0x314690198
UIKit                           0x31d13a7fc
UIKit                           0x31d135534
CrashProbeiOS                   0x20004f2a4
libdyld.dylib                   0x30f0f65b8

So the first step would be to find some names for those addresses. This process is often called “symbolizing” or “symbolicating”. We can already see where the addresses are located because the device sends us a list of loaded images (object files) and where they are loaded into memory. To find the names we need to look at symbol tables.

Stacktraces on iOS

So as you can see stacktraces are fairly incomplete. While we can easily find out what frameworks the addresses are contained in, it’s unlikely that you will be able to find the function names for them on the device. There are two cases you have to keep apart here. One case is where the symbols are in fact missing, the other one is where symbols are marked as redacted.

Missing symbols are typically what you have in release builds for your own applications. In release builds most of the symbols you encounter are not actually on the device so if we were to try to locate the function names on the device we will not succeed. Instead they are stored in what is commonly referred to as a “dsym file”. Technically a dsym file is a macho file just like an executable but it only contains the symbol table and debug information. So while they could be in the same file, they usually are not. When I said that “most” symbols are not on the device, this refers to the fact that some symbols need to be in the file. This is because most applications on iOS are written in Objective-C. This is relevant because Objective-C implements methods through a mechanism that is based on the idea of sending messages from object to object. These messages are referred to as “selectors” and they are essentially the name of the method.

PLCrashReporter and some other tools are often attempting to find such symbols even if the normal symbols are not on the device, however for the bulk of the symbols you need to do this on the server.

The second case of missing function names we need to concern ourselves with is a weirder one: redacted symbols.

Redacted Symbols

Redacted symbols are symbols that are indeed available on the device but tools like KSCrash or PLCrashReporter cannot access. When iOS loads system libraries it removes symbols so that when one attempts to read the symbol by parsing the framework one will only come across a symbol with the name <redacted> .This is most likely done to save some memory or for security reasons. Because all system frames will have the same constant string as a symbol there is a lot that does not have to be loaded into memory.

The downside is that we are not able to tell you which function in UIKit caused your crash. When you hook your phone up to Xcode you can see such symbols though. So how does that work? The answer is a bit bizarre and requires some understanding of what happens when iOS loads the system libraries.

When iOS redacts symbols it stores a copy of the original symbol on the file system in a cache file that is not accessible for non rooted devices. The file is named dyld_shared_cache_arm64 for arm64 etc. From the file name you can see that this is considered a cache file. This means the file is updated as redacted symbols are added to it. Apple built this system to primarily support the flow where you debug your own device. If you run your own app and you hook it up to the debugger all the frames that you are interested in will have their redacted symbols added to the cache file. When you connect the phone to Xcode, Xcode will go in and “prepare the device for development” and that will essentially download the cache file and run it through a process where dummy debug symbols are built for it. It will in fact create a folder structure below ~/Library/Developer/Xcode/iOS DeviceSupport for your version of iOS and put new macho files in there with symbols recovered from the cache file.

Now you can guess what the problem with this is: if you have never seen a symbol it won’t be in the cache file. This is particularly noticeable if you are working with “legacy” architectures. For instance if you hook up an arm64 device with Xcode it will be able to extract some armv7 symbols but it will most likely not find all. Your chances are most likely higher if you are running a lot of 32bit apps to populate the cache, but you might as well just hook it up with an older device instead. Whenever you add a device to Xcode it will merge together the symbols it extracts.

This shows one of the core issues that come up with symbolizing on iOS: you need to collect as many of these debug symbols as possible.

Symbolicating App and System

Sentry is using two separate systems for resolving functions. For customer debug symbols we are using our own LLVM based symbolication library for Python. We fetch debug symbols from our S3 backed asset storage and then symbolicate based on the symbols we have on our device. This scales quite well to the workload caused by apps. These are typically large symbol files but there are not that many per app.

On the other hand dealing with symbols from the system libraries is a different story. There are thousands of symbol files and because the cache might be incomplete we actually want to be quite fuzzy over them. As example for this fuzziness is that we might be dealing with incomplete debug symbols for system libraries from one SDK. In that case we want to try a few older versions as well in case we find matches there.

To achieve this goal we wrote a separate system we call the sentry symbol server and it is a simple HTTP service written in Rust that takes a batch request of addresses to symbolicate and then responds with the function names if it finds them. It uses a custom file format that can be memory mapped in. We then use a separate build process to create these mmap’ed files and put them to S3. In regular intervals the server checks back with S3 and fetches new memory maps if necessary.

The Final Result

After symbolication our boring crash report from before looks more like this:

CrashLibiOS        -[CRLCrashNULL crash] (CRLCrashNULL.m:37)
CrashProbeiOS      -[CRLDetailViewController doCrash] (CRLDetailViewController.m:53)
UIKit              -[UIApplication sendAction:to:from:forEvent:]
UIKit              -[UIControl sendAction:to:forEvent:]
UIKit              -[UIControl _sendActionsForEvents:withEvent:]
UIKit              -[UIControl touchesEnded:withEvent:]
UIKit              __UIGestureEnvironmentSortAndSendDelayedTouches
UIKit              __UIGestureEnvironmentUpdate
UIKit              -[UIGestureEnvironment _deliverEvent:toGestureRecognizers:usingBlock:]
UIKit              -[UIGestureEnvironment _updateGesturesForEvent:window:]
UIKit              -[UIWindow sendEvent:]
UIKit              -[UIApplication sendEvent:]
UIKit              ___dispatchPreprocessedEventFromEventQueue
UIKit              ___handleEventQueue
CoreFoundation     ___CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__
CoreFoundation     ___CFRunLoopDoSources0
CoreFoundation     ___CFRunLoopRun
CoreFoundation     _CFRunLoopRunSpecific
GraphicsServices   _GSEventRunModal
UIKit              -[UIApplication _run]
UIKit              _UIApplicationMain
CrashProbeiOS      main (main.m:16)
libdyld.dylib      _start

And this allows us then to render the crash report in a more presentable way. Because we know which symbols are from your app and which ones are from the system we can by default hide frames you likely don’t care about:

In an Ideal World

In an ideal world Apple would provide a web service that does what our symbol server does. You give it the UUID of the image you want to symbolicate and the address in it, and you get back a response of the symbol that is at that address. At present the process of collecting all the symbols from different SDK versions is slow, requires a lot of manual labour and is not even guaranteed to always succeed.

Future Plans

Sadly we are limited to providing system symbol resolving on our cloud hosted version. There are some concerns about the redistribution of system symbol files which is why we currently cannot offer this service for on-prem customers.

If you are interested for support for system symbol symbolication for on-prem installations leave your feedback in the forums. We are playing with the idea of making our symbol server a public API in case there is demand for it.

If this article was of interest of you let us know. We might do a followup where we explain our heuristics and the technical challenges on doing server side symbolication.

Open Source Sprint at Sentry

2017-04-03T00:00:00-07:00

In March, we hosted a sprint for the SF Python community to help newcomers interested contributing to open source. For two days, over 50 people came together in the Sentry offices to work on widely used projects like Mypy, Zulip, Certbot, and of course, Sentry. In particular, five attendees merged their first commits to a major open source project.

If you’d like to see what was built, check out the PRs below! And if you’d like to start contributing to Sentry, check out our docs for setting up your development environment and come chat with us on our community forum.

Contributors to Sentry

travisoneill helped us improve our project search parser
scotchka improved our public issue pages
jamesmura worked on filtering MySQL passwords, project search jumping to the right event when you search for a given event id, and adding event_id to our webhook payload
ganeshkbalaji simplified our member invite flow
jgriffith23 added read-only member

Welcome Evan Ralston

2017-03-31T00:00:00-07:00

Evan Ralston joins Sentry this week as an operations engineer.

Evan previously worked at a number of Bay area startups where he helped improve uptime of their web services. He will be helping Sentry’s operations team get ahead of projects, instead of dealing with systems problems reactively.

When not automating himself out of a job, Evan likes convincing people to watch Steven Universe.

Tips for Reducing JavaScript Error Noise

2017-03-27T00:00:00-07:00

If you’re using Sentry to monitor and debug browser JavaScript issues, you might be suffering from a common affliction: noisy, low-value errors that make it harder for you and your team to identify high-priority issues.

This happens because browser JavaScript is perhaps the single most complex environment from which to capture errors – because it’s not just one environment! There are multiple major browsers, JavaScript engines, operating systems, and browser extension ecosystems, all of which come together to make capturing good errors difficult.

Sentry does a decent job out of the box cutting through all this noise, but for the best results, it needs your help. Below are a few additional steps you can take to configure Sentry to greatly reduce the amount of noisy errors you receive.

Whitelist your URLs

Sentry’s browser JavaScript SDK(Raven.js), by default, will pick up any uncaught error triggered from your web app. That includes code running on your page that isn’t necessarily authored or controlled by you. For example, errors triggered from browser extensions, malware, or 3rd-party applications like chat widgets, analytics, and ad code.

To ignore such problematic errors, you can configure Raven.js to whitelist errors originating solely from your own code:

Raven.config('your-dsn', {
    whitelistUrls: [
        'www.example.com/static/js', // your code
        'ajax.googleapis.com'        // code served from Google CDN
    ]
}).install();

This example configuration ensures that only errors that originate from scripts served from www.example.com/static/js and ajax.googleapis.com are reported to the Sentry server. This small configuration change is the easiest, most impactful change you can make to reduce errors.

Use inbound data filters

Inbound data filters are a Sentry feature designed to discard known low-value errors from your projects. They are easily toggled inside your Sentry project settings, and any errors they discard as a result do not count towards your account quota.

There are 3 filters that are particularly valuable for JavaScript developers:

Legacy browsers – old browsers like IE9 produce low-fidelity error reports that aren’t always actionable
3rd-party extensions – automatically drop errors from known browser extensions, malware, and ad scripts
Web crawlers – drop errors triggered from known web crawlers like Google Bot

Inbound filters are not as powerful as configuring Raven.js to whitelist error URLs, but they’re nice because they can be enabled with a single click from inside your project settings.

Note that inbound data filters are open source and contributed to by the community. If you are encountering an error that you feel should be discarded globally by these filters, consider opening an issue/pull request.

Use the latest version of Raven.js

Sentry’s browser JavaScript SDK is under active development, and changes are frequently made to both improve the quality of error reports and reduce the quantity of low-value errors.

For example, Raven.js 3.12.0 now suppresses back-to-back duplicate errors by default. This is a life-saver if you are suffering from errors that trigger from asynchronous loops (e.g. from setTimeout or XMLHttpRequest callbacks). In long lived applications, errors like these can result in thousands of events for a single user!

To get the best experience, keep your copy of Raven.js up to date. Sentry will tell you when there’s a new version available, but it’s also worth checking the changelog periodically to see what’s new.

Use source maps

Source maps don’t just make debugging your production stack traces easier, they make it easier for Sentry to group errors into individual issues. This means that events bucket into a smaller, more manageable set of issues, which means less noise in your issue stream and less 2AM emails about broken code.

Making source maps a part of your build and deploy process isn’t as easy as toggling a button, but our in-depth source map documentation has everything you need to get started. Besides helping to reduce noise, source maps may be the single most profound improvement you can make to your monitoring and debugging workflow.

Ignore troublesome errors

Some errors you’re just never going to fix. When that happens, it might be time to declare bankruptcy and ignore them entirely.

You can either ignore the error via the Sentry UI, or configure Raven.js to prevent them client-side using the ignoreErrors option. Doing so from Raven.js is ideal because errors discarded at the client-level do not reach Sentry’s servers and do not count against your event quota.

Here’s what it looks like:

Raven.config('your-dsn', {
    ignoreErrors: [
        'Can\'t execute code from freed script',
        /SecurityError\: DOM Exception 18$/
    ]
}).install();

Be careful though: once you make this change, you’ll never see these errors again. And error strings that were previously just nuisances could become bigger problems down the line that you’ll never be informed of. Choose your ignoreErrors array wisely!

Note that browsers can produce different error messages for the same fundamental error. For a single ReferenceError, you may need to input multiple strings/regexes to cover all possible browsers.

Upload your source files (and source maps)

When Sentry encounters an error triggered from a JavaScript file, it attempts to download that source file from your app servers in order to correlate line and column information with actual source content. This source content is the basis of Sentry’s error grouping algorithm.

If your source files are only accessible over the web, there’s a lot of bad things that can happen. For example, you might have a build process that removes old JavaScript files from servers as you deploy new ones. If your users trigger errors from older cached scripts, when Sentry goes to download them, they will no longer be available. Not having access to that content can mess up the grouping algorithm, which means separate issues will be created for errors that would normally be bucketed under an existing issue.

To avoid these and other interruption scenarios (e.g. network availability), we strongly recommend you upload your production JavaScript files and source maps as release artifacts. This means that Sentry will always have direct access to these files, ensuring maximum grouping effectiveness.

Uploading source files is done using the Sentry API, and is pretty simple:

$ curl https://sentry.io/api/0/projects/:organization_slug/:project_slug/releases/:release/files/ \
  -X POST \
  -H 'Authorization: Bearer YOUR_TOKEN_HERE' \
  -F file=@app.js.map \
  -F name="http://example.com/app.js.map"

To learn more about artifacts and releases, please see our documentation.

Welcome Kelly Carino

2017-03-20T00:00:00-07:00

We’re excited to announce that Kelly Carino is joining the Sentry team.

Kelly joins us from Massdrop where she worked on improving the customer experience as well as scaling the support team. During her last few months at Massdrop, she completed an online course at Coding Dojo. At Sentry, she will be focusing on technical support and building out projects with the team.

React Native

2017-03-14T00:00:00-07:00

We have released a dedicated React Native SDK with some awesome features. If you are using raven-js with React Native, we recommend that you switch over to our brand new react-native-sentry SDK.

With this new SDK, Sentry is now able to provide mixed stacktraces. This means that if a JavaScript call causes a crash in native code, you will see the last call from JavaScript before the crash. This also means that with the new SDK, native crashes are properly handled on iOS.

Of course, if an error occurs in JavaScript, we also provide you with a useful stacktrace.

Since we use our powerful, native Swift SDK in the background, you will also get much more information about the device and operating system.

When you also have an Android app, it will gracefully fall back to use raven-js since we currently only fully support iOS.

To start using the new SDK, see the react-native-sentry documentation.

Introducing Reprocessing

2017-03-02T00:00:00-08:00

Introducing Reprocessing for iOS

In order to provide useful and human-readable stacktraces for iOS crashes, developers have to share their app debug symbols with Sentry. If a crash comes in, Sentry uses these debug symbols to map memory address to the according function name and line number.

For example: the unreadable 0x205d2d000 becomes ViewController.onClickFatalError(AnyObject) -> () (ViewController.swift:113)

Now the user knows exactly where the app crashed and can fix the bug. But for this to work inside Sentry, there was a catch: you had to provide these symbols before the crash occurs. And not all iOS developers have access to symbols before they go up on the App store.

Today we are proud to announce our new reprocessing feature. With this update, iOS events that cannot be processed due to missing debug symbols will be held from the event stream and reprocessed once the symbols have been uploaded.

Why reprocessing matters

This has two major advantages for iOS developers. The first is that this keeps noise down due to bad grouping caused by lack of information. In the past, if events were submitted before debug symbols were uploaded, you could easily end up with lots of incorrectly grouped errors that were just duplicates from older issues.

The second benefit is that since we put those issues on hold temporarily, we will not send out any email notifications from those until the debug symbols are up.

While in a perfect world you would never see an event before the debug symbols are ingested, we know that this is hard to do in practice. In particular, if you have bitcode-enabled builds, it can take a long time for processing to finish on the iTunes Connect side. This means that there is a good chance that someone may start (ab)using your app before you’ve had time to upload symbols.

Enabling reprocessing

Reprocessing is disabled by default and can be turned on in the project settings. Additionally, if you have an issue that lacks debug symbols, we’ll point you to the reprocessing settings page.

In the project settings you can turn it on with a flip of a switch:

There you can also see a log of all the reasons events were not processed (for instance, because of missing debug symbols, because broken debug symbols were uploaded etc.).

When debug symbols are missing you can see this on the event stream. A red warning bar will appear and inform you to upload missing symbols.

Our command line tools have been updated to trigger reprocessing automatically. If you are using an older version of sentry-cli that does not have this automatically enabled, you can also manually trigger reprocessing from the project settings.

How it works

What happens if it’s turned on?

When the feature is enabled, all events that are missing mandatory debug symbols go into a queue and will not show up until their debug symbols are uploaded. Currently we only require a small set of debug symbols and when all those are up the event is free to be reprocessed.

What is considered a mandatory debug symbol?

This is currently implementation defined and we might tweak it in the future. For now, you can see the list of debug symbols we deem required in your project settings once we encounter such debug symbols. We recommend uploading all debug symbols you have.

What triggers reprocessing?

There is an API you can hit which starts the reprocessing. By default this is done automatically by the sentry-cli tool but this can be disabled. You can also trigger reprocessing again from the project settings.

What about optional debug symbols?

We recommend uploading optional debug symbols first or to not trigger reprocessing until you have them all up. Once an event is no longer on hold we no longer permit reprocessing.

Can I see events on hold?

There is currently no way for you to see such events but we will investigate this option in the future.

What happens if I disable reprocessing?

If you disable reprocessing, events that are already on hold will be processed and shown to you in the stream view. Future events that come in will also show up as they arrive. This, however, might cause bad grouping because information relevant for grouping could be unavailable. We recommend against turning off reprocessing.

Dodging S3 Downtime With Nginx and HAProxy

2017-03-01T00:00:00-08:00

Like many websites and service providers, we use and depend on Amazon S3. Among other things, we primarily use S3 as a data store for uploaded artifacts like JavaScript source maps and iOS debug symbols; which are a critical part in our event processing pipeline. Yesterday, S3 experienced an outage that lasted 3 hours, but the impact on our processing pipeline was very minimal.

Last week, we set off on solving one potential problem that we were experiencing: fetching data out of S3 was neither as performant nor reliable as we would hope. Our servers are in Dallas, while our S3 buckets are in Virginia (us-east-1). This means that we see an average of 32ms ping times, and 100ms for a full Layer 6 TLS handshake. Additionally, S3 throttles bandwidth to servers outside of their network, which limits the ability for us to fetch our largest assets in a timely manner. While increasing performance was our primary goal, this project turned out to be extremely beneficial during the S3 outage and kept our processing pipeline chugging along for the duration of it.

We started off by putting together a quick S3 proxy cache that lived in our datacenter that would cache full assets from S3, allowing us to serve them on our local network without going to Virginia and back every single time. The requirements of this experiment were:

minimize risk to production-facing traffic
avoid introducing single points of failure
prove the concept without increasing hardware bill
avoid committing changes to application code

We completed our task with two popular services, no application code required. We used nginx as an S3 cache, while using HAProxy to route requests back to S3 if nginx were to fail.

The goal of the nginx server was to leverage the proxy_cache and store all of our S3 assets on disk when requested. We wanted to leverage a large 750GB disk cache and keep a very large set of actively cached data.

Our new proposed infrastructure would look like this:

The setup should look relatively familiar for anyone who has worked with service discovery. Each application server that’s running our Sentry code has an HAProxy process running on localhost. HAProxy is tasked with directing traffic to our cache server, which will proxy upstream to Amazon. This configuration also allows for a failover to occur, allowing HAProxy to talk directly to Amazon and bypass our cache without any interruption.

Configuring HAProxy is quick, only taking eight lines:

# Define a DNS resolver for S3
resolvers dns
  # Which nameserver do we want to use?
  nameserver google 8.8.8.8
  # Cache name resolutions for 300s
  hold valid 300s

listen s3
  bind 127.0.0.1:10000
  mode http
  # Define our s3cache upstream server
  server s3cache 10.0.0.1:80 check inter 500ms rise 2 fall 3
  # With actual Amazon S3 as a backup host using our DNS resolver
  server amazon s3.amazonaws.com:443 resolvers dns ssl verify required check inter 1000ms backup

On each application server tasked with communicating to S3, the HAProxy Admin ends up displaying this:

This gives us a live view of the cache’s health from the perspective of a single application server. Ironically, this also came in handy when S3 went down, clearly depicting that there was exactly two hours and 57 minutes of downtime during which we could not communicate with Amazon.

Configuring nginx was a little bit more involving since it’s doing the heavy lifting:

http {
  gzip off;

  # need to setup external DNS resolver
  resolver 8.8.8.8;
  resolver_timeout 5s;

  # configure cache directory with 750G and holding old objects for max 30 days
  proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=default:500m max_size=750g inactive=30d;

  server {
    listen 10.0.0.1:80 default_server;

    keepalive_timeout 3600;

    location / {
      proxy_http_version 1.1;

      # Make sure we're proxying along the correct headers
      proxy_set_header Host s3.amazonaws.com;
      # Pass along Authorization credentials to upstream S3
      proxy_set_header Authorization $http_authorization;
      # Make sure we're using Keep-Alives with S3
      proxy_set_header Connection '';

      # Configure out caches
      proxy_cache default;
      # Cache all 200 OK's for 30 days
      proxy_cache_valid 200 30d;
      # Use stale cache file in all errors from upstream if we can
      proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;
      # Lock the cache so that only one request can populate it at a time
      proxy_cache_lock on;

      # Verify and reuse our SSL session for our upstream connection
      proxy_ssl_verify on;
      proxy_ssl_session_reuse on;

      # Set back a nice HTTP Header to indicate what the cache status was
      add_header X-Cache-Status $upstream_cache_status always;

      # Set this to a variable instead of using an `upstream`
      # to coerce nginx into resolving as DNS instead of caching
      # it once on process boot and never updating.
      set $s3_host 's3.amazonaws.com';
      proxy_pass https://$s3_host;
    }
  }
}

This configuration is allowing us to use a 750GB disk cache for our S3 objects as configured by proxy_cache_path.

This proxy service had been running for a week while we watched our bandwidth and S3 bill drop, but we had an unexpected exchange yesterday morning:

HAProxy had immediately notified us when S3 started showing signs of trouble. In a random turn of events, the proxy that we had implemented to serve as a cache was now serving all of Sentry’s S3 assets while the Amazon service was offline. During the full three hours, we exhibited some problems when users attempted to upload artifacts, but the event processing pipeline happily kept flowing.

From start to finish, our proxy cache took less than a day to implement. In this past week, we have reduced our S3 bandwidth cost by 70%, gained even more performance and reliability when processing events, and took back the majority of the eggs we had the S3 basket.