Bringing Codecov into the Sentry Family: Where Code Coverage Meets Application Monitoring

Today Codecov is joining the Sentry family. Codecov began as a code coverage reporting tool in 2014 and has since emerged as a market leader in the test analytics space. Codecov makes coverage actionable for over two dozen test frameworks, and has helped over a million software developers improve their approach to testing, coverage, and code reliability.

You might be asking, what do test analytics have to do with application monitoring? To understand that, we must first step back, and focus on what happens when everything goes wrong, when we don’t test our code correctly (or test it at all), and when we fail to monitor it. When we think about solutions, we focus on the problem first, and the problem we care about is software outages.

Software outages begin with problem analysis - quickly connecting symptoms (or outcomes) to possible underlying implementations. That same skill has been valuable when thinking about Sentry’s place in the world, how we evolve our product, and ultimately why anyone even uses our software. That analysis tells us that Sentry exists to service developer productivity. We focus on identifying a problem - as quickly in its lifecycle as possible - and ensuring you have the right information and tools at your fingerprints to bring it to resolution. Many might think of that as MTTR (mean time to recovery), but we’ve only ever focused on how quickly we become aware, and can do whatever we choose to do, even if that means deciding that an issue isn’t important. It’s about helping software teams maximize the time in the R&D phase, rather than within incident management.

On the pre-release side of the software development lifecycle, software testing is paramount to ensuring that high-quality code is being shipped to end users. It has long been Codecov’s belief that shipping healthier, higher-quality code leads to less risk, better user experience, and – most importantly – increased developer productivity. Codecov’s approach to improving developer productivity concerns what happens before the Pull Request, centering on code coverage and automated testing. However, like Sentry, Codecov’s focus has always been to make the developer aware and then leave the decision making in the hands of the developer. To that end, even though we have approached how to improve developer productivity from two different perspectives, Codecov and Sentry’s mission is the same: build the best developer tools in the world by staying deeply committed to a developer-first mindset.

When we started speaking to the Codecov team, we heard about their journey through working with tens of thousands of organizations. In short, they discovered that code coverage as we knew it previously was a useful metric, but there was little universal agreement about what that metric meant: Is 100% coverage the goal? Why can a fully covered code base still break? If not 100%, what amount of testing is enough? Don’t believe us? Just Google “Hacker News Code Coverage” and read the debates.

Those debates, happening in teams around the world, reveal the deeper questions to be answered: How are you testing your code? How resilient are the tests? Why are you testing that part of the codebase and what is the risk to the customer/user? Do you need to run all of your tests on every commit? These questions are the ones that Codecov used to inspire its rich set of features and its roadmap that will be integrated with Sentry.

As far as Sentry + Codecov goes, you can expect business as usual. We have a few ideas for touch points that we think can help improve the value if you’re a customer of both, and you’ll likely see some of that post-holidays. Within Sentry we’ll help you understand if the issues we’ve identified were covered by your test suite - helping you determine if they could have possibly been avoided. Codecov will also continue to pursue some of its ongoing initiatives that we are particularly excited about, such as Impact Analysis and Smart Automated Test Selection.

Longer term we will bring both products together in a more cohesive story, but also continue to drive independent value. Our investments around Codecov will continue to focus on maximizing the utility of your tests and CI infrastructure, whereas Sentry will focus on telling the story of code once it reaches production.

For now, we will sign off with a warm welcome to the Codecov team, and big ambitions for 2023.