How Nextcloud Uses Sentry to Build Private, Federated Clouds

Christoph Wurst, software engineer at Nextcloud GmbH, recently wrote a blog post about using Nextcloud logs as Sentry Breadcrumbs. We thanked him profusely, and then we asked him to write another blog post — this time, for us.

Nextcloud is an open source, self-hosted file share and communication system. You might even consider it a platform, in the sense that apps run on top of Nexcloud and it provides a variety of features beyond file handling, including calendar/contacts, project management, communication, and more.

Nextcloud also offers over 200 integrations to extend its functionality, from social networking to kanban boards to error tracking with Sentry.

Nextcloud Sentry integration

The Sentry app for Nextcloud keeps track of unhandled errors and exceptions from the Nextcloud core and any apps running on it. For that, both the PHP and JavaScript SDKs are used to hand over all crash information to Sentry.

With the integration, errors are collected and aggregated from Nextcloud in the Sentry web interface.

How it works

Nextcloud registers a global error handler in PHP that captures all unhandled errors. Errors are then passed to the central logging service. Apps can also acquire an instance of the logger to log any errors, warnings, and debug information.

In order to register crash reports, we needed to adapt the logger implementation. With this adaptation, when an error occurs, the logger adds the information to the system log file, and also invokes the crash reporter callbacks and passes some context information. In this case, Sentry is the crash reporter, and data is passed on to the Sentry SDK.

To enhance Issue reports on Sentry, the Nextcloud logger passes logging information (debug, info, warnings, etc.) to the crash reporters. Sentry then uses this information to build Breadcrumbs.

On the client-side, the Sentry browser client does its magic and hooks into the error handlers of popular frameworks. In addition to the captured console logging statements that Sentry shows, the Breadcrumbs generated from the Nextcloud logger display automatically.

How the integration helps developers at Nextcloud

Unlike other companies using Sentry for error tracking, Nextcloud is not directly operated by the team that develops the software. Instead, the software is hosted on-premise by Nextcloud users.

At Nextcloud GmbH, engineers use Sentry for their personal Nextcloud instances as well as the company instance. In the past, this arrangement has helped find issues in betas, RCs, and stable releases that might not have been noticed otherwise.

Sentry treats app IDs automatically assigned by the system logger as tags on its user interface, making it possible to filter tickets for specific apps. We find these tags especially useful when maintainers of subsystems want to check if errors were triggered by any of their maintained apps, like this example from Nextcloud's Mail app.

Because Nextcloud passes Sentry release data, Sentry knows the Nextcloud version and can detect regressions. This means an issue will be automatically re-opened if it had been closed as resolved in a previous version.

We often rely on this feature when testing pre-releases and going from Alphas to Betas to stable releases: should a known bug be triggered after it was marked as resolved in the previous release, Sentry will automagically re-open the ticket and warn about the regression.

Hunting bugs

When Sentry signals a new issue, one of the developers will triage the report and assign the developer who might know how to address the issue (most likely because the bug was triggered by an app they maintain). This is where the app ID, the stack trace, and Breadcrumbs help a lot.

The developer can then use the information from the report to try to reproduce the issue. Once it's confirmed and fixed, the issue is marked as resolved. This will silence Sentry alerts until the next release. Should Sentry see the issue again, it will re-open the ticket and send another alert about the regression.

At Nextcloud, we see most new/unknown issues from the front-end, as errors in the back-end of the application are usually caught and logged already. Errors on the front-end are often unnoticed, by both the admin and the users, and might cause issues for a long time without anybody ever noticing. Sentry has consistently revealed errors nobody had noticed before, ultimately contributing to Nextcloud's increased stability.

Emphasizing security

Sentry and Nextcloud both place a strong focus on protecting user data and security. Sentry automatically removes sensitive parameters in our data, like passwords and API keys, from reports with its data scrubber.

Nextcloud also uses a strict content security policy (CSP). Unfortunately, as an admin, issues with CSP are usually unnoticed unless the users report them. Sentry, however, already has support for CSP violation reports, which will be integrated soon.