Sentry at Disqus

Since Sentry’s conception at Disqus in early 2010, its use inside has grown considerably.  The entire Disqus engineering team (16 people and counting) use Sentry in one way or another, deploying 20 Sentry instances across a wide range of services.

So as not to be deadbeat parents, we recently caught up with Disqus engineers Mike Clarke and Ted Kaemming for an update.

A little history:  As Disqus started taking off in 2009 (by 2010 it would be handling 200M unique visitors), traditional logging became problematic.  A need existed for real aggregation, but also the power of introspection.  In early 2010, Sentry was conceived to solve exception logging (primarily for Django, as it's what powered Disqus).  Today, Disqus is the #1 online discussion and commenting service, serving 1.1 billion unique monthly visitors at 2.5 million sites.  While it's average is only 1-2 events per second in Sentry, at times there are spikes of up to 500 errors a second.

As the open source version of Sentry matured, its use inside Disqus grew, encompassing the entire engineering team.  While they were still primarily using Django, Flask and JavaScript were becoming more important, thus Sentry needed to expand. It soon gained functionality and support for most popular platforms and languages (Python error tracking, PHP error tracking, Java error tracking, Node.js error tracking, JavaScript error tracking, etc.).

On the operations side, Sentry is used primarily as an early warning system, allowing for rapid response to issues as they arise. Disqus engineer Mike Clarke explains, “As we ship code, which we do at least daily, I’m constantly staring at Sentry.  Watching the trends to see if something new is broken versus something that’s been broken for a while.”

As we ship code, which we do at least daily, I’m constantly staring at Sentry.

On the development side, Sentry helps prioritize resources.  “When we’re building stuff, there’s always going to be errors. It helps to find out how frequently those errors are occurring to prioritize which things get tackled first.  Being able to look at Sentry over a time period, or just overall, tells us what’s really most impactful to the user”, explains developer Ted Kaemming.

Sentry is also used by support. When a user gets an error, they’re given a hash they can include in a support email.  This hash gives the support person a clue as to what might be broken, and who in engineering to talk to.  If the email becomes a support ticket for engineering, engineering can then use Sentry to debug, using the search and lookup features (among others) to examine the specifics of the error.

Being able to look at Sentry over a time period, or just overall, tells us what’s really most impactful to the user.

Disqus also uses Sentry for diagnostic logging, using it to see how errors in one part of the system affect other systems.  It logs warnings in addition to errors, and even uses Sentry for auditing and security tasks.  “If you’re masquerading as another user doing super admin type things, we’ll log that to Sentry so you can follow if someone is unexpectedly doing bad things to somebody else.  The grouping is so effective that when something out of the ordinary does happen, regardless of it’s an error or not, it’s a convenient way of visualizing it”, explains Mike Clarke.

In all, Disqus uses Sentry in a variety of places, more than 20 projects at this point.  They report that setup is surprisingly easy for new projects, and adoption shows no signs of slowing. “Sentry is great in terms of visualizing stack traces for debugging and that sort of thing, but there’s many ways we can know when our system is broken. How Sentry consolidates that into a single location is what makes it truly useful for us”, explains Mike Clarke.

Header photo credit: Disqus Team at PyCon 2013 by Ed Schipul