Growing Adoption by Building a Modern Clippy

Remember Microsoft’s trusty assistant, the annoying/endearing Clippy? We built something similar for the modern web, minus the annoying parts, and discovered that, if done right, it can increase user engagement and success with your product.

At Clippy’s inception, the idea of a virtual assistant was very appealing. But Clippy itself had two significant flaws:

1. It showed up at unexpected times with irrelevant suggestions. Part of the problem was that it was optimized for new users, but kept appearing for users already proficient in what they were trying to do.
2. It was based on a misunderstanding of research that suggested that people respond to computers as if they are human. Microsoft took this quite literally and gave Clippy the “human” interface that became a defining but annoying characteristic of Clippy.

At Sentry, we faced a similar onboarding problem.

Assistant: Sentry’s Clippy

Sentry is a real-time error-tracking service. When it comes to debugging errors, more context and customizability is usually better. However, the flip side is that this can make the new-user experience overwhelming. We have good documentation, but there is no substitute for a great in-app experience. We also have an onboarding widget, but engagement with it is low because it is like a list of TODOs rather than a guided tour of the product.

This led to the creation of Assistant, a widget that pops up unobtrusively when we think the user may need help with something new. For example, the Issue details page is optimized for a repeat user familiar with the interface, but new users could use a more gentle introduction. Assistant guides the user through the most important features on this page.

Extensibility

Since this is a useful pattern that can be used in different contexts, we wanted to build it in a way that would allow us to highlight anything in the product. For example, we have a guide that introduces Alert Rules to you if we detect that you’re sending a large number of errors but haven’t set up alerting.

To do this, we created a JSON-like specification for guides that is parsed and rendered by our frontend React app. Creating a new guide is a matter of specifying the steps in a declarative format, tagging the React components we want to be highlighted at each step, and writing the business logic to determine when the guide should be triggered. All in all, it should take less than a couple of hours to do this for a new guide.

Take a look at this example guide specification:

'releases': {
    'cue': 'What are releases?',
    'steps': [
        {
            'title': 'Releases',
            'message': 'A release is a specific version of your code deployed to '
                       'an environment. When you tell Sentry about your releases, '
                       'it can predict which commits caused an error and who '
                       'might be a likely owner.'),
            # `target` specifies the React component that should be highlighted
            'target': 'releases-title',
        },
        {
            'title': 'Releases',
            'message': 'Sentry does this by tying together commits in the release, '
                       'files touched by those commits, files observed in the '
                       'stacktrace, and authors of those files. Learn more about '
                       'releases here.'
            'target': 'releases-body',
        },
    ]
}

Measuring Success

When we work on new features, it’s not just to create new, shiny things. It’s about improving customer experience. To know if we actually made an impact on customer experience, we measure a pre-determined success metric.

An important success metric we use for many onboarding changes is the first-event rate. This is the percentage of new users who successfully install Sentry and send an event. We A/B tested Assistant on users and found that it increased our first-event rate by almost 6%.

We also track the “health” of different guides. If too many users dismiss a guide or don’t find it useful (as determined by a feedback button at the end of the guide), we improve it or cut it out. Here you can see us tracking open/dismiss/finish rates for a few different guides.

By providing additional, well-timed context, Assistant positively impacted Sentry user engagement. We also came out of the build process with a scalable system for continued guide iteration. Interested in helping us with our next project? We’re hiring on the Growth team!