What does one feature actually cost to build?

One feature built two ways is an anecdote with receipts. That is worth something, and it is worth less than a rate. The published limits are on the page because you would find them out anyway.

1. 01

   n = 1.

   One feature. A demonstration, never a rate. Unflattering, mixed, or a failure in either arm: it is published as it stands.

2. 02

   The control is not reproducible.

   One person driving Claude Code varies from one run to the next. We freeze the prompt and the subagent setup in writing and publish the transcript, which narrows it. It does not close it.

3. 03

   We pick the feature ourselves, and the rule was amended four times.

   Whoever picks the task picks the winner, and the picker here is us. The rule was published before any candidate was listed, and its first application returned an empty set. One clause was then dropped in public, the one excluding issues opened by our own team, and the rule was applied again to eight candidates: every one was skipped for not changing this repository own Ruby. The fallback named in advance was that an outside party files the feature, and no outside party turned out to be available. So the rule was amended once more, in public and before the feature issue existed, to say the team running the study files it. That is what happened. The strongest objection to this comparison, that the task suits our own arm, now has no answer beyond two checks a reader can make: the amendment is posted before the issue exists, and the feature is taken from a gap this repository already documented about itself. The issue body stands unedited and every version of the rule is public in posting order.

4. 04

   We run both arms.

   The people who built Arm B are the ones driving Arm A, the control we are competing against. An independent run would be worth more than ours, and we would rather say so than imply otherwise.

5. 05

   Calendar time is not effort.

   Our clock includes queue time you would really wait. The control's clock is one person's uninterrupted attention. That framing flatters the control, not us.

6. 06

   The reviewer is one of us.

   One reviewer, on our team. He is the one driving the control arm, so he recognises his own diff before he opens either one. Both diffs arrive with branch names and commit authorship stripped, and that is all the word blind means here. An outside reviewer would remove this limit instead of softening it; we do not have one.

7. 07

   We maintain the venue.

   Both arms run on a public repository we maintain: our org, our CI, our bot on the merge lane. An outside repository would close this limit. What it would take is a maintainer who is not us willing to let both arms open pull requests, run their CI and merge, and we do not have one.

8. 08

   The frozen model changes twice, then changes back.

   Three dated comments on the pre-registration move it and then restore it. The first two rest on a wrong reading: every account credential for the frozen model is revoked, which is true, and irrelevant, because the coding lane runs on the box's own backend and the venue keeps serving that model throughout. Every version is public and dated before any arm starts, with no result standing against it, so no figure here is one seen first and chosen afterwards. The model stays identical on both sides, so it changes what each one runs and never what tells them apart, and rule 6 still forbids a re-roll. Read the thread in posting order and judge how carefully this was operated for yourself.

9. 09

   The prediction knows the task, only not the answer.

   We publish what we expect the records to show before either arm runs, so a page that reveals its numbers only afterwards cannot ask you to believe those numbers surprised nobody. It is protected against knowing the outcome. It is not protected against knowing the task, because the selection rule names the feature first, and that asymmetry is the reason this is called a prediction and never a pre-registration. No rule locks the set after the fact: rule 6 forbids re-rolling the feature, not re-writing a prediction, so the protection is publication itself: the predictions and their scoring rules are dated public comments on an issue whose edit history is open, posted before any record exists, and a revision shows up there as a revision. A prediction that lands nowhere near the record stays on the page beside the record that breaks it. One thing it leaves alone on purpose is which pull request the reviewer prefers, since that is the one judgement here with no instrument behind it.