Monitoring and Prioritizing Edge Cases, Without ‘Gold-plating’ Non-critical Issues

Rated as the top order tracking and revenue generation app on Shopify, Rush lets businesses build and personalize their own dashboards to manage the post-sale process with real-time data, custom product recommendations, and user feedback. Their business model focuses on low touch and user-centered design (UCD), which leaves little room for issues impacting how people interact with the platform.

Most of their users come through the Shopify marketplace, which makes them quick to install and quick to uninstall. Developers pay close attention to edge cases with the biggest potential impact on customers and closely monitor release health to resolve any issues in production.

As Co-founder and Product Lead, I know how important is for our business model to ensure a great customer experience. We are a data-driven solution and we operate in the big data space, where you can’t see all the issues or handle all cases or variations. Stanislav Stankov, Co-founder & Product Lead, Rush.

Making APM “part of the business’ DNA”

Rush supports more than 1,500 brands – and counting – on Shopify and brands using the app have seen an average of 30x ROI with more than $10m in post-sales opportunities influenced through the shipping process alone. Part of that success has been how users can customize their order tracking pages and delivery notifications with analytics, CTAs that drive additional sales, and real-time shipment data.

Store owners are offered preloaded templates with the option to modify the app at the code level using CSS. Each template includes Sentry by default, allowing Stanislav’s team to see how a brand’s customers interact with the order tracking page, and whether any scripts are impacting performance. After that, the team monitors users’ services through their frontend projects dashboard.

All of this is done during onboarding and saves a lot of time configuring, and expands visibility on the frontend, with the added benefit that everyone has a basic working knowledge of Sentry.

A lot of companies use sentry for error monitoring, but we use it to determine the impact an issue has on the entire business.

Integrating Sentry with their CI/CD pipeline

Rush’s system serves roughly 2.3m calls a day and with a customer base used to an app marketplace experience where you uninstall a non-performant tool almost as quickly as you installed it, poor UX has a direct impact on the bottom line. The difference between a customer and ‘what could have been’ is context, so they crafted a workflow that, in most cases, caps time to resolution at 10 minutes.

They do this by integrating Sentry directly into their CI/CD pipeline to view source code context obtained from stack traces and use custom tags to configure the level of detail attached to errors, depending on the log level of the application and customer tenant.

Over time the team built up a ‘hit list’ of issues to keep an eye on. When there’s an error, issue grouping and their consistent monitoring of edge cases help prioritize what to solve first based on volume or impact.

Prioritizing issues is priceless for us. With that, we’re not only able to increase system resiliency and quality, but we avoid gold-plating, giving us time to work on impactful tasks.

Knowing that certain errors only surface at deployment, developers pay close attention to release health and if an alert is triggered after a deployment, their CD can automatically roll back if the error rate increases above a custom threshold, giving them time to investigate; with custom error context already attached to tackle it in production before a user even notices.

After we started using Sentry we got a much better understanding of the quality of the app we were building, it’s not just about monitoring and fixing issues, we have data-driven insight into how our system operates.

If you’d like to learn more about how Rush has made Sentry work for them, check out our full conversation with Stanislav here.