Does Averaging p99 Give You the Real p99?
I tested two ways of calculating an hourly p99. First, I split each hour into five-minute buckets and calculated p99 for every populated bucket, then averaged those bucket p99 values. I calculated p99 once again using all requests from that hour. The results were different.
I checked this across 167 qualifying hours from the Azure Functions trace. Compared with the hourly p99 calculated directly from raw requests using PostgreSQL's percentile_cont(0.99), the averaged five-minute values had a median absolute relative error of 16.75%.
A percentile is not additive. Once a bucket is reduced to one p99 value, you lose the individual request durations that were used to calculate it.
Consider two buckets: one has 100 requests, and the other has 100,000. Averaging their p99 values gives both buckets equal weight, even though one represents 1,000 times more traffic. To calculate the hourly p99, you need to use all 100,100 requests together.
This matters when p99 is used to decide whether a service meets its service-level objective (SLO). If the averaged value and the raw hourly p99 fall on opposite sides of the threshold, they can produce different PASS or FAIL decisions. I test this later with a controlled latency experiment.
Store Sketches, Not Numbers
A percentile sketch is a compact data structure that summarizes a set of values so you can estimate percentiles later. In this experiment, those values are function invocation durations. Because sketches can be merged, I can combine the five-minute sketches into an hourly sketch and then calculate p99 from that merged result.
On Tiger Cloud, I used Timescale Toolkit to build the percentile sketches. Its percentile_agg() function uses UddSketch to process all request durations in each bucket and store a compact summary of them. The p99 returned from that sketch is approximate, and Toolkit reports a relative error bound for the estimated percentile value.
The pattern is simple:
-- five-minute bucket
percentile_agg(duration_ms) AS latency_sketch
-- merge sketches at a higher level
rollup(latency_sketch)
-- calculate p99 from the sketch
approx_percentile(0.99, latency_sketch)
I stored the sketches in TimescaleDB continuous aggregates, which save calculated summaries for reuse. The five-minute sketches were then merged into hourly sketches, and the hourly sketches into daily ones.
Get Tiger Cloud Running and Load the Trace
I ran the experiment on a free Shared service in Tiger Cloud. After creating the service, I connected to the database and confirmed that TimescaleDB and Timescale Toolkit were available.
Figure 1. Selecting the free Shared compute option in Tiger Cloud for the experiment.
The trace contains 1,980,951 invocations across 424 functions in 119 applications over 14 days. It records how long each function invocation took to complete, rather than the full time an HTTP request takes to reach the server and return a response.
I converted the durations to milliseconds and used January 1, 2021, 00:00 UTC as the starting point for the trace timestamps, then loaded the data into a TimescaleDB hypertable using PostgreSQL's COPY command.
For the hourly p99 comparison, I selected the function with ID 283 based on its request volume and activity. I made this selection before calculating any percentile errors.
Function 283 had 535,667 invocations across 238 populated hours. For the comparison, I kept only hours with at least 1,000 invocations and activity in six or more five-minute buckets. Each qualifying hour also needed a median of at least 100 invocations per populated bucket. That left 167 hours for the comparison, containing 487,152 invocations across 1,654 populated five-minute buckets.
Build the Rollup That Keeps the Distribution
With the data loaded, I built two ways to calculate hourly p99. The first used the five-minute p99 values to calculate an hourly result. The second used the percentile sketches from the previous section.
Averaging the p99 values
I started by calculating p99 separately for each populated five-minute bucket using PostgreSQL's percentile_cont():
percentile_cont(0.99) WITHIN GROUP (ORDER BY duration_ms)
For each hour, I calculated the average (AVG) and maximum (MAX) of the five-minute bucket p99 values. Both methods used the already calculated bucket p99 values, rather than the individual request durations. MAX tells you how bad the worst five-minute bucket was. That can help spot a short spike, even when only a small fraction of the hour's requests experienced the slowdown.
Merging the sketches
For the second approach, I used TimescaleDB continuous aggregates to store the results at five-minute, hourly, and daily levels. I started with percentile_agg(), which creates a sketch from the individual request durations in each five-minute bucket.
Then I used rollup() to combine the five-minute sketches into hourly sketches. I repeated the process with the hourly sketches to create the daily results.
-- Used in the hourly and daily aggregates
rollup(latency_sketch) AS latency_sketch
I can then use approx_percentile(0.99, latency_sketch) to estimate p99 from the stored sketches.
Checking the stored results
I built the sketch aggregates using all the invocations from function 283. I used TimescaleDB's refresh_continuous_aggregate() function to calculate and store the results.
After refreshing, I checked the number of stored rows and invocations represented at each level.
| Aggregate | Rows | Invocations |
|---|---|---|
| Five-minute | 1,894 | 535,667 |
| Hourly | 238 | 535,667 |
| Daily | 14 | 535,667 |
Since the dataset covered a fixed historical period, I didn't need an automatic refresh policy.
Does Rollup Error Change an SLO Decision?
Across the 167 qualifying hours, I compared AVG, MAX, and the rolled sketch against the hourly p99 calculated directly from raw request durations using PostgreSQL's percentile_cont(0.99). AVG had a median absolute relative error of 16.75%, producing a lower value in 31.14% of hours and a higher value in 68.86% of hours. MAX had a median absolute relative error of 135%, showing that taking the highest five-minute p99 did not recover the raw hourly percentile.
The rolled sketch had the lowest median absolute relative error of the three methods, at 0.0759%. Its largest difference from PostgreSQL's hourly p99 was 17.10%.
PostgreSQL's percentile_cont() can return a value between two request durations, while Toolkit's sketch estimates a discrete percentile. This difference affects the comparison, since Toolkit's 0.1% error bound does not apply to PostgreSQL's interpolated p99.
Setting the SLO threshold
To check whether these differences affected SLO decisions, I used a fixed threshold of 10 ms, chosen after exploring different thresholds. This was an experimental SLO, not one used in Azure production.
An hour counted as PASS when p99 was below 10 ms. Values of 10 ms or higher counted as FAIL. Using PostgreSQL's percentile_cont(0.99) as the reference, 153 of the 167 qualifying hours passed, and 14 failed.
I compared the other methods against these decisions. A false green means a method reports PASS when the reference says FAIL. A false red means it reports FAIL when the reference says PASS.
| Method | False greens | False reds |
|---|---|---|
| AVG of five-minute p99 | 4 | 51 |
| MAX of five-minute p99 | 0 | 126 |
| Rolled sketch | 0 | 7 |
AVG made errors in both directions. MAX reported many failures in hours that passed according to PostgreSQL. The rolled sketch came closest to the reference with seven disagreements across the 167 hours, all of them false reds.
Introducing controlled latency incidents
The control hours showed how the methods behaved on the original data. I wanted to see what happened when some invocations became slower.
Of the 167 qualifying hours, 61 passed the 10 ms SLO and had activity in all twelve five-minute buckets. Each had at least 1,000 invocations. I selected the five busiest hours for the controlled incidents.
I simulated each incident by making some requests slower. Every incident started 17 minutes into the hour, and I tested durations of 5, 10, and 20 minutes. For each duration, I selected 5%, 10%, or 25% of the requests that occurred during the incident, then added 5, 10, or 20 ms to their original durations to see how different levels of added latency affected the hourly p99.
Testing all three durations, affected-request percentages, and added-latency levels across the five hours gave me 135 scenarios. A fixed seed kept request selection reproducible, and the requests were selected without considering their original durations.
For each hour, incident duration, and affected percentage, I kept the same selected requests across the three added-latency levels. Increasing the delay from 5 to 10 to 20 ms then changed only how much those requests slowed down.
For every scenario, I recalculated the raw hourly p99 using PostgreSQL's percentile_cont(0.99), along with AVG and MAX, then rebuilt the Toolkit sketches using the modified invocation durations.
Comparing the incident results
Across the 135 scenarios, PostgreSQL's raw hourly p99 marked 54 as PASS and 81 as FAIL.
| Method | False greens | False reds | Correct decisions |
|---|---|---|---|
| AVG | 58 | 0 | 77 |
| MAX | 0 | 47 | 88 |
| Rolled sketch | 0 | 0 | 135 |
AVG missed 71.6% of the failing scenarios in this predefined test.
To show what one of those missed failures looked like, I examined a five-minute incident where I added 10 ms to 58 of 581 eligible invocations.
Figure 2. Hourly p99 by method during a five-minute incident. Dashed line: 10 ms SLO.
The reference p99 reached 11 ms, crossing the SLO threshold. AVG returned a value below 10 ms and marked the hour as PASS. The rolled sketch estimated 10.9902 ms and correctly marked it as FAIL.
To see where AVG missed failures, I compared its decisions across different incident durations, affected-request percentages, and added latency levels. Figure 3 shows how the number of missed failures varied across those settings.
Figure 3. AVG false greens against PostgreSQL's p99. Each cell shows the count out of five hours.
I also ran two checks to verify the stored sketches:
- Request counts: In all 1,894 populated five-minute buckets, the number of values recorded in each sketch matched the actual number of invocations.
- Daily rollups: Across all 14 days, sketches built directly from raw durations produced the same daily p99 estimates as sketches merged through the five-minute, hourly, and daily aggregates. Compared with PostgreSQL's raw daily percentile_cont(0.99), the estimates had absolute relative errors ranging from 0.0441% to 0.0910%.
Gotchas, Limitations, and Conclusion
These results have a few limitations. The Toolkit sketch produces approximate p99 estimates, which can differ from PostgreSQL's raw p99. In the original control hours, the sketch reported seven false reds compared with PostgreSQL, while it agreed with PostgreSQL in all 135 incident scenarios. The same level of agreement is not guaranteed for all datasets.
The incident experiment covered five hours, all from the same UTC date. AVG missed 58 of the 81 failing scenarios, but these results come from a limited set of simulated incidents and don't tell us how often AVG would miss failures in production.
The dataset records function invocation duration rather than end-to-end HTTP latency. Since the data was historical, the aggregates were refreshed manually. A live system would need automatic refresh policies that update earlier time buckets when data arrives late.
Overall, the experiment shows how averaging five-minute p99 values can hide hourly SLO failures. The rolled Toolkit sketch had the lowest median p99 error and produced the fewest incorrect SLO decisions among the three methods tested.