Performance Vitals: a unified scoring system to guide performance health and prioritization

Tl;dr: The following post details how we measure client performance across products and cross-functional teams at Coinbase.

By Leonardo Zizzamia, Senior Staff Software Engineer

A lot has changed since 2018 when the Coinbase web team consisted of only a few engineers. Back then, while working on making our product faster with a small group across a single platform, we could rely on pre-existing ). The NTBT measures the amount of time the application may be blocked from processing code during the 2 second window after a user navigates from page A to page B. The NTBT metric is the summation of the blocking time for all long tasks within the 2s window after this method is invoked.

The image below is an example of an NTBT performance mark in our discovery back to the community in the coming quarters.

Creating Backend Vitals

As we did with Web and React Native Vitals, we extended the Vitals standard to backend services including GraphQL and Backend Services.

The two metrics we first created are:

• GraphQL Response Time (GRT): Round trip time for the GraphQL service to serve a request.
• Upstream Response Time (URT): Round trip time for the API Gateway to serve a backend service.

To determine a Good Score to represent backend latency, we considered several points:

1. From a user’s perspective, the system response time feels immediate when it is less than 1s.
2. We also have to take into account that the network cost could vary between 50ms-500ms, depending on which region a user is reaching our product from.
3. Based on points 1 and 2, GraphQL latency should not exceed 500ms, meaning the upstream services must respond in under 300ms because GraphQL queries have to await the slowest endpoint.
4. Therefore, we concluded that the threshold for a GRT Good score is 500ms, and URT Good score is 300ms.

For Backend Vitals we aim for at least 99 percent of measurements for each logged request to meet the “Good” threshold.

As we continue to improve our performance, we will revisit our Good scores annually, potentially even lowering them over time so we can further lower latency for our users.

The instrumentation for Backend Vitals is made up of three essential pieces. First, we use our in-house analytics library to define metadata like the product, platform, and pages. Then, we propagate this information into our APIs, and ultimately we co-locate the performance metrics with the Web or React Native metadata.

Performance Vitals discoverability and prioritization

Using the same metric scoring and attribution system across different specialties at Coinbase makes it easy to identify areas of opportunity and aligns both frontend and backend engineers in performance efforts.

All Performance Vitals are based on real-time data from our production applications and can be discovered by standardized filters, such as: product name, platform, page, is logged in, geo region, GraphQL operation, and backend service.

This level of accuracy becomes especially useful for Real Time Anomaly Detection. All teams are able to own the performance metrics for their product surface, giving them the ability to have automated monitors for performance changes and be alerted when regressions occur.

In case of a performance regression, we use the percentage of the regression to determine if it’s critical to open an incident and mitigate the issue as soon as possible, or create a bug that can be solved in the coming sprint.

Quarterly and annual planning

Performance Vitals are perfect for KR planning, as they measure a score from 0 to 100 and they can be easily stored for over a year. Common language for all performance KRs also makes it easier to create shared goals for teams across the organization.

A few examples of how you can frame your KRs are:

• [Year KR] Reach NTBT Good Score of 90%, up from 70% in the Coinbase Mobile App.
• [Quarter KR] Improve LCP Good Score from 70% to 85% in the Coinbase Web.

Up Next

Performance Vitals come back to finding a common language, whether it’s standardizing filters, setting quarterly KR’s, or unifying a scoring system. From a small team working on an API regression to large initiatives led by multiple organizations, speaking the same language helps all types of product prioritization.

In the future, we plan to open source some of our learnings and share more about measuring and driving impact for Critical User Journeys and how we use automation and internal processes to enable everyone at Coinbase to build performant products.

Performance Vitals: a unified scoring system to guide performance health and prioritization was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.