Share SaaS Speed Tests While Building in Public

The short answer

Share a SaaS speed test with the task, timing boundary, versions, workload, and comparable conditions. Report correctness checks, failures, typical duration, and slow-end results together. Distinguish controlled testing from real-user evidence, retain the method, and keep the public conclusion within the measured scope.

Share a SaaS speed test by naming the task, timing boundary, compared versions, workload, and test conditions alongside the result. Include failures and slower responses as well as the typical time. Keep controlled-test results separate from evidence about real customers, and make your public claim no broader than the work you measured.

Building in public gives performance work an audience. A useful update can help another founder inspect your approach or help a prospective user understand a specific improvement. A large multiplier with no method leaves both readers guessing about what changed.

The most valuable detail may be a limit: the new version improved the typical response, but the slow end still needs work. Publishing that distinction gives your next update a clear question to answer.

Name the beginning and end of the timer

Start with the user’s task, then identify exactly which part you measured. Opening a report may involve a request, server work, data transfer, browser rendering, and a final interaction. An improvement in one part does not tell you the duration of the whole task.

Check the metric definition rather than inferring it from a label. For example, Grafana k6 defines HTTP request duration as sending, waiting, and receiving time, excluding initial connection setup. That number is useful, but it does not establish when a person can read and use the resulting screen.

Write the boundary in the public post: request duration, report generation, or time until the requested chart is usable. Readers can then interpret the result without supplying a more ambitious meaning themselves.

Separate a controlled test from a customer experience

Google’s web.dev explanation of lab and field data distinguishes controlled measurements from observations of actual users. Devices, networks, locations, and other conditions can differ. Use the same distinction when sharing a SaaS improvement, even if your test concerns an API rather than a web page.

A local test can reveal a promising change. State that it ran locally. A staging test can help compare deployed versions under chosen conditions. State those conditions. Evidence collected from real use needs its own description of the population, dates, and measurement coverage.

A concrete public example is Robbie Coomber’s account of rewriting PostHog’s SQL parser. The author distinguishes the laptop benchmark behind the headline from observations on production queries. These are the author’s reported results, which this article has not reproduced. The useful publishing lesson is to keep different measurement settings visible.

Check that both versions do the same useful work

Define what successful completion means before comparing speed. A report that returns quickly because it omitted a required calculation has not delivered the same task. Check output correctness, required data, and expected behavior along with timing.

Choose an input that resembles a job your intended user performs. Document its relevant size and shape using safe sample data. For a report, row count alone may omit important differences such as date range, filters, joins, or the amount of data returned.

State any deliberate change in output or scope. If you made a report faster by returning a shorter summary, that may be a sensible product decision. Explain the tradeoff so readers understand what they gain and what the comparison no longer holds constant.

Keep error behavior visible. Brendan Gregg’s benchmarking checklist warns that errors can skew a benchmark and asks whether results reproduce and matter to real use. A fast error response should not quietly improve a number presented as successful task performance.

Keep the comparison conditions in the record

Record the two versions and the setting in which each ran. Retain enough detail to repeat the comparison: tool version, relevant configuration, input, concurrency, run duration or operation count, and how you handled warm-up and timeouts.

Caching deserves an explicit note. A first attempt may do work that a later attempt can reuse. If one version sees prepared data and another starts from an empty cache, your comparison includes that difference. Decide whether you want to examine first-use behavior, repeated use, or both, then label the results accordingly.

Run comparable attempts more than once and keep their variation. One unusually good result is a weak basis for a public headline. If later runs disagree, investigate whether the workload, environment, or background activity changed before selecting a number to announce.

You can describe the environment without exposing private infrastructure. Keep secrets and customer records out of shared artifacts. Use the SAFE test for public SaaS metrics to decide what can be disclosed, then preserve the fuller technical record privately when necessary.

Show the typical time and the slow end together

The median describes the middle of the measured durations. A high percentile such as p95 helps describe slower observations. Use the definition and calculation provided by your measurement tool, and keep the operation count beside those statistics.

A lower median can coexist with a worse slow end. Your public explanation should preserve both. If a particular task is important to customers, a repeated delay there may matter more than a small improvement in typical requests elsewhere.

Keep failures, timeouts, and correctness checks alongside latency. Explain whether durations include failed operations or only successful ones. A statistic calculated after excluding failures describes that selected set; the reader still needs to know what happened to the excluded attempts.

Avoid combining different endpoints into a single improvement when their workloads or traffic proportions changed. A quicker overall figure might reflect more easy operations in the second run. Compare the same task first, then explain any broader aggregate separately.

A fictional result needs a qualified headline

Imagine a founder testing a revised query path for a SaaS report. Both versions use the same sample input and a single concurrent requester after the same warm-up procedure. The following product details, counts, and summary values are fictional illustrations. No benchmark was run for this article.

The median fell by 200 milliseconds. Relative to the original 800 milliseconds, that is a 25% reduction: 200 divided by 800. The p95 increased by 100 milliseconds. These two observations support a mixed account of the result.

“The report is 25% faster for everyone” would go beyond the example. It changes a median request-duration measurement into a universal task-experience claim. The table also says nothing about how many customers the system can support or how performance behaves at higher concurrency.

A precise public statement could read:

In this controlled comparison of the report request, the median duration fell from 800 ms to 600 ms with the same sample input and one concurrent requester. The p95 rose from 1,400 ms to 1,500 ms. The illustrative run recorded no failed requests, and all measured requests passed the defined output checks. The typical request improved, while the slow end needs investigation. Browser task time and customer performance remain unmeasured.

That passage is a model caption for the fictional example. For real work, attach the actual run record and describe its real checks. Passing a particular set of checks does not prove the software has no correctness defects.

Keep speed, capacity, and cost as separate claims

A shorter request duration under a fixed workload tells you how long those requests took. Capacity asks how much work the system can handle while meeting acceptable latency, error, and correctness conditions. Name which of those questions your experiment addresses before turning the result into a product promise.

In the fictional example, the operation count is the same in both columns. It does not establish throughput: the table does not provide the full run duration or arrival pattern. It also does not establish a supported customer count. One customer task might produce several requests, and different tasks may create different work.

Resource changes need their own explanation. If a new version uses more computing resources, publish that condition alongside the timing result. You may have made a worthwhile tradeoff, but duration alone does not establish that the work now costs less. Keep resource usage and cost measurements separate when you evaluate that decision.

Choose wording that gives readers the useful benefit without enlarging its scope. “The measured report request has a lower median in our test” states an observed boundary. A claim about handling more simultaneous users requires evidence about that workload. A claim about saving customers time needs a task boundary that includes the waits those customers experience.

If you have only the timing comparison, publish it as progress on that measurement. Leave capacity, operating cost, and customer time as open questions until you have examined them directly.

Give readers a way to inspect the comparison

A short update can link to a longer method note containing the workload, sanitized configuration, run outputs, and definitions. Include the relevant versions so that a future reader can distinguish an old result from the current product.

If you publish a chart, label the units and measurement boundary. Keep compared conditions apparent, and avoid presenting selected runs as the entire experiment. When you show a recording, disclose editing that changes the apparent duration or explain where the timer starts and stops.

You do not need to expose a repository or live environment to make progress understandable. A clear method and bounded conclusion can support useful discussion. If a detail cannot be shared, identify the missing information and narrow the reproducibility claim accordingly.

Buildside’s description of its founder community emphasizes sharing product progress and useful observations. A performance update can contribute to that conversation without implying that Buildside collected or verified the measurements for you.

Ask for criticism that can improve the next test

Invite readers to challenge a specific uncertainty. Ask whether the input resembles their workload, whether the completion condition misses an important wait, or whether another operating condition belongs in the next comparison. Those questions create a useful exchange around the work.

Keep a suggestion separate from a reproduced result. A commenter who suspects connection setup is responsible has offered a hypothesis. Test it before reporting it as the explanation. If their observation exposes a mismatch, correct the public account rather than defending the original headline.

When the change ships, link the release to the method note and say what was deployed. A controlled test before release and a measurement after deployment are separate records. Compare real-user observations only when their definitions and populations support the conclusion.

Your next update can report whether the slow-end behavior changed, which condition explained it, or why the test remains inconclusive. Building in public becomes more useful when readers can follow that investigation across updates instead of encountering a new unsupported speed multiplier each week.

Common questions

Do I need a performance tool to share a speed improvement?

You need a clear measurement method, not a particular vendor. State the timing boundary, tested versions, workload, conditions, and limits. Use a method that lets you check correctness and inspect repeated measurements.

Can a local benchmark show that customers have a faster product?

A local benchmark describes the tested conditions. Customer experience requires evidence from the relevant real-user setting or a carefully bounded task test. Keep those conclusions separate.

Should I wait until every timing measure improves?

You can share mixed or inconclusive progress. Report the improvement and regression together, explain what remains uncertain, and state the next investigation. A public progress update does not need a winning result.

Sources

• Grafana k6 documentation: Built-in metrics

• Google web.dev: Why lab and field data can differ

• Brendan Gregg: Evaluating the Evaluation, a Benchmarking Checklist

• Robbie Coomber, PostHog: SQL parser engineering account

• Buildside: About the founder community

Frequently asked questions

A note from Uriel Bitton

This guide applies published benchmarking methods to founder updates. Its numerical example is fictional; it does not report a benchmark executed by Uriel, Buildside, or a customer.

Keep building with us

Get practical founder lessons and product updates in your inbox.

Subscribe or Join Buildside.

Keep reading