Share AI Evaluation Results While Building in Public

The short answer

Share AI evaluation results with the task, versions, input set, scoring rule, attempt counts, and remaining failures. Separate successful attempts from inputs handled consistently. A higher total can hide a regression, and offline evaluation results do not establish customer accuracy or time savings.

Share AI evaluation results while building in public by publishing the task, tested versions, input set, scoring rule, attempt counts, and remaining failures beside the headline result. Separate successful attempts from inputs handled consistently. A stronger total can hide a regression that matters to the person deciding whether to try your feature.

A public update should let readers understand what changed and challenge the next test. It should also preserve the boundary between an offline evaluation and a working product in somebody else's hands. A successful generated answer shows one result under stated conditions. It doesn't establish customer adoption, time saved, or reliable performance across every workflow.

The useful story is often a decision: one change helped with missing deadlines but introduced an invented owner. Sharing that tension gives other builders something specific to inspect and prospective users something concrete to assess.

Define the task before announcing the score

Write the user task in ordinary language before choosing a number. For a meeting assistant, “extract the actions explicitly agreed in these notes” is more useful than “understand meetings.” Name the required output and the mistakes that would make it unsuitable.

Claude's evaluation documentation recommends specific, measurable criteria relevant to the application, with test cases that reflect its tasks and edge cases. The public-writing implication is to explain what counted as success before presenting how often it happened.

A founder might require each extracted action to include the agreed task, the stated owner, and the stated due date. Where the notes don't name an owner or date, the output should leave that field empty. A plausible guess would fail that particular rule, even if the resulting sentence sounds helpful.

Choose that rule because it matches your feature's job. A brainstorming assistant may welcome suggestions; an extraction feature may need to preserve the source faithfully. Avoid carrying a good score from one job into a claim about the other.

State what the system can actually do after producing the output. Extracting a proposed action list and creating tasks in a customer's workspace are different product behaviors. If your test covered only the list, keep your public description at that boundary.

Keep inputs, attempts, and successful outputs distinct

An input is one case in your evaluation. An attempt is one run of that case. Repeating eight inputs three times produces 24 attempts, while the input variety remains eight cases. It doesn't become a study of 24 users or 24 different meetings.

Anthropic's guide to agent evaluations, published January 9, 2026, distinguishes tasks from repeated trials and explains that outputs can vary between runs. It also distinguishes finding a success across attempts from succeeding consistently. Those distinctions are useful when a founder summarizes an AI feature's progress.

Report both the total successful attempts and how those successes are distributed across inputs. If you reran a case until it worked, say how many attempts you made. A selected screenshot should not silently stand in for all the outputs you saw.

Keep the input set, scoring rule, and repetition plan the same when comparing versions. Record the model identifier, prompt version, relevant settings, tools, and supplied context. Identical settings don't guarantee identical outputs; they make the comparison easier to interpret.

When several components changed together, describe the result as a comparison of those complete configurations. Don't credit a prompt edit alone if the model or reference material changed too.

A fictional comparison reveals what the total misses

Consider Cedar Notes, an invented SaaS meeting assistant. Its founder is comparing prompt versions A and B for extracting agreed actions. The product, inputs, versions, counts, and example posts in this section are fictional. No evaluation or model calls were run for this article.

The imagined test uses eight synthetic English meeting notes, with three attempts for each input under each version. An attempt passes only if the whole output meets the predefined action, owner, and date rules. A partial success remains a failed attempt in this illustration, with the reason retained separately.

The imagined scorer checks the output against a reference prepared for each case. Names and dates may not be invented; an older quoted request may not be treated as a newly agreed action. These are proposed rules for this example, not a standard for every meeting product.

Version B has three more passing attempts. Dividing by 24 gives 75% for A and 87.5% for B on these invented attempts. Those percentages describe the table, not expected production accuracy.

Now count cases that pass on all three attempts. Version A has five: cases 1 through 5. Version B also has five: cases 1 through 4 and case 7. The total improved while the number of consistently passing inputs stayed the same.

Case 5 deserves its own sentence. In the invented failing output, B assigned an owner even though the meeting note didn't name one. A handled that case correctly on all three attempts. That regression matters to the extraction promise, regardless of the higher overall total.

The unchanged five-of-eight count doesn't mean the versions behave identically. B improves cases 7 and 8, leaves case 6 unresolved, and loses one pass on case 5. The table shows where the work moved; a single overall score cannot carry that explanation.

Turn the comparison into a public progress update

Publish the result and the limitation together, so a screenshot or quoted paragraph keeps the qualification attached. Here is example copy for the fictional Cedar Notes comparison:

In our illustrative comparison, prompt B passed 21 of 24 attempts against A's 18 of 24, using the same eight synthetic meeting notes and three attempts per note. Both versions passed every attempt on five of the eight inputs. B improved handling of unstated deadlines, but it invented an owner once on a case A handled consistently. The next change will target that regression. This is an offline example; customer accuracy and time savings remain unmeasured.

For an actual founder update, replace every illustrative detail with your retained run evidence. Identify the evaluation date and exact configurations. State whether scoring was automated, human, or model-assisted, and who reviewed ambiguous outputs if anyone did.

If another model graded the outputs, name that judge and its rubric. Don't describe an unchecked model verdict as human review. Anthropic's agent-evaluation guide advises inspecting trial records and checking model-based grading against human judgment. That is a reason to inspect disagreement before turning it into a confident public claim.

You can keep a concise feed post and link to a longer method note. The short post still needs the task, denominator, material failure, and current scope. Readers shouldn't need to follow a link to discover that the celebrated result required several retries.

Share the failure without exposing customer material

Use a shareable example that preserves the behavior you want readers to examine. For Cedar Notes, a synthetic line could read, “Let's prepare the outline by Friday,” with no owner named. Show why an assigned owner would violate the extraction rule.

Keep the label close: this is a synthetic illustration of an error pattern. It is not a customer quote, a reproduced production incident, or proof that a particular model makes that mistake. If you change a real input for publication, distinguish the edited example from the original evaluation record.

Before sharing actual records, use the public SaaS metrics disclosure guide to review what the material could reveal and whether you can publish it. Keep private customer content, access details, and unpublished business information out of screenshots and attached logs.

A public explanation can describe the task, rule, and failure category without releasing every input. Be clear about what readers can inspect and what remains private. A limited disclosure supports a limited reproducibility claim.

Ask the audience for a missing case

Give readers a question that can improve the evaluation. “Is this impressive?” invites a reaction to the post. “What wording would make it unclear whether an action was agreed or merely discussed?” invites a concrete addition to the next test.

For Cedar Notes, a useful public request could ask how meeting participants distinguish a suggestion from a commitment. Invite invented examples rather than confidential meeting transcripts. Ask contributors to explain the expected output, because a tricky input without a clear expectation may expose a product-definition question rather than a model failure.

Keep audience suggestions separate from evaluated results. A comment proposes a case. You still need to decide whether it belongs in the intended workflow, define its expected behavior, run it, and inspect the outcome. A peer's agreement doesn't add another passing attempt to the table.

This is where building in public contributes to product work: the visible result gives people enough context to identify a missing assumption. The public demo feedback guide explains the related distinction between what you show, what viewers say, and what users actually try.

Buildside publishes this Journal. Its feed guide describes sharing product progress with founder context; publishing an evaluation there does not mean Buildside conducted or independently verified the test.

Make the next update comparable

Keep the original cases and their results when you add new ones. Give the expanded set a new version, and report whether a new score uses the original inputs, the additions, or both. Otherwise readers may mistake an easier mix of cases for an improved feature.

If you change a scoring rule after discovering a valid output was rejected, explain the correction. Regrade comparable saved outputs when that is appropriate and possible. A changed rule is information about the evaluation, not automatically a change in the product.

In the fictional example, the next update should return to the unnamed-owner case as well as checking that the gains on cases 7 and 8 remain. A successful new screenshot of case 5 would be useful evidence to retain, but it would not replace the agreed repeated test.

Separate the resulting product decision from the score. You might continue development, invite a bounded trial, or keep the feature unavailable while investigating a failure. Explain the decision and the remaining checks. The illustrative counts in this article establish no release threshold.

Before your next public post, choose one task whose result you can explain fully. Preserve its inputs and outputs privately, decide what is safe to share, and write the failure sentence before polishing the headline. That sentence often tells readers which part of the work needs their help.

Questions founders ask next

How many evaluation cases should I publish?

Publish enough detail to make the scope understandable, including the full input count and how cases were selected. Eight cases in this article are a teaching example, not a recommended minimum. The necessary coverage depends on the task, its variation, and the consequences of mistakes.

Should I remove an earlier update after improving the feature?

Keep a dated record of what the earlier version did, and link it to the current result. Add a correction if the old post contains an error. A later improvement can be explained without making the previous measurement appear to describe the current product.

Can public feedback replace testing with intended users?

Public feedback can identify missing cases and unclear expectations. It does not establish how intended users perform a real task with the feature. Record actual use, assistance, outcomes, and the observation period separately when you investigate that question.

Sources

• Anthropic: Demystifying evals for AI agents — tasks, trials, grading, consistency, and inspecting evaluation records; published January 9, 2026.

• Claude Platform Docs: Define success criteria and build evaluations — task-specific criteria, input coverage, and grading methods; checked November 11, 2026.

• Buildside Journal: Which SaaS Metrics Are Safe to Share Publicly? — reviewing the disclosure of public evidence.

• Buildside Journal: How to Turn a Public SaaS Demo Into Useful Product Feedback — separating a presentation, a response, and observed use.

• Buildside: Feed guide — the publisher's description of sharing product progress.

Frequently asked questions

A note from Uriel Bitton

I share practical lessons from building Buildside and learning alongside SaaS founders.

Keep building with us

Get practical founder lessons and product updates in your inbox.

Subscribe or Join Buildside.

Keep reading