An Engineering Manager’s Tips for Scaling High-Velocity Teams
Forethought is a leading AI company providing solutions that transform the customer experience for brands like Marriott, Asana and Instacart.
As a high-growth startup with 2x annual growth in their engineering team, the team faced increasingly complex processes and found that what had worked in the past wasn’t going to cut it anymore.
Reduced time to resolution by up to 94% with improved issue ownership
Deploy new releases an average of 2.5 times a day without sacrificing quality
Improved search latency by 20-30%
Integrated internal apps like Zendesk for easy investigation and,
Adopted a more strategic approach to team growth and management
Forethought's core engineering team handles upwards of 52 million requests a day and maintains common services between other internal teams, infrastructure, data, and tools. But as they grew in size from 6 to 40 engineers – with the goal of doubling that number within a year – they faced a growing backlog of issues and challenges measuring the impact and health of their services.
As our team of engineers grew, we had to split up. While that made sense from a product management and tracking perspective, we didn’t anticipate how difficult it would be to determine ownership of issues that were happening in production and the knock-on effects. Jad Chamoun, Engineering Manager, Forethought.
For them, a customizable solution that could strip away some of the complexities that come with rapid growth would help the team:
Better define ownership and increase developer autonomy
Prioritize issues more efficiently and,
Get better visibility into engineering velocity, to set realistic goals and determine if management/growth strategies are working
As an existing Sentry customer already getting value from our Python SDK, Forethought went all in, taking full advantage of default capabilities like application performance monitoring.
Lacking ownership of issues and concerded about technical debt, Jad’s team integrated their GitHub CODEOWNERS file with Sentry, allowing them to divide up and assign responsibilities to specific teams. Today, errors are filtered by team, automatically triaged and assigned instead of everyone getting all the errors at once.
With alerts, the core engineering team added an additional layer to how they deal with issues, giving them detailed insight into the volume of errors they receive over time and highlighting those with the highest impact, helping them direct – and redirect – engineering resources more strategically.
Finally, Sentry’s Slack integration notifies the responsible team directly – instead of everyone getting all the alerts in a single channel and then manually figuring out who’s going to tackle it – so engineers can communicate within the channel and resolve the issue faster.
I had to be able to balance my engineers’ time between fixing bugs and building new features, for that to happen we needed a solution that helped us stay on top of our backlog.
Jad’s team uses Dashboards to keep an eye on the highest-priority issues as they’re happening and the number of total issues happening at any given moment. They’re able to narrow these down to only core engineering’s list of issues, and with Sentry’s Jira integration, see whether an issue is already being dealt with or, if not, assign it to the right team.
A timeline of unresolved issues shows where teams are improving or where errors continue to happen. For Jad, the amount of new errors introduced and the number of regressions they’re getting in production is an important indicator and helps determine when and where to double down on code.
How long it takes them to resolve an issue is another key metric they keep an eye on. Adding this layer of visibility, defining team roles and having the relevant engineers assigned to an issue reduced their time to resolution from months to just days and, in some cases, hours.
We’re a high-velocity team, we move fast, but we shouldn’t compromise quality.
A bird's-eye view of team performance means Jad can better define internal OKRs and service-level objectives. Sentry Stats lets him keep an eye on time to resolution, volume of new issues, and latency when when processing requests from other internal teams; ultimately enriching metrics linked to deliverables and improving team management. These metrics are used to report to upper management, measure how teams are progressing towards goals, whether management-level baselines are realistic, and if their strategies are working.
With Sentry, Forethought has given their engineers more autonomy, improved the health of their teams from a management perspective, and empowered them with the tools to succeed at scale. To date, this has helped them:
Reduce time to resolution by up to 94% with improved issue ownership
Deploy new releases an average of 2.5 times a day without sacrificing quality
Improve search latency by 20-30%
Integrate internal apps like Zendesk for easy investigation and,
Become more strategic about team growth and management
Forethought is currently hiring and Jad is on the lookout for backend, DevOps and Data engineers. Check them out if you’re looking to join an awesome team.
And if you’d like to learn more about how Forethought uses Sentry, check out our full conversation with Jad here.
