Nineteen Hours of Not Fixing Anything

Part Two ended with me saying the evidence was the software, and the software was what I was going back to. I went back to it. This is what happened next, and what I got wrong.

01What I asked for

I want each person in a novel to have one wiki page, even when the book uses several names for them. A woman is called by her first name in chapter one, by her married name in chapter six and by "the widow" in chapter nine. She gets one page, not three. I will say what it should produce, not how it does it.

I set the bar at 95% at the very least, aiming at 100%. I also asked for no paid runs, because I had been burning money on tests.

02What actually happened

Between 18 and 22 October, an AI session worked on that problem. One stretch of it is worth putting plainly. From 01:53 to 21:11 on 21 October, about nineteen hours elapsed. That is the span between the recorded times, not a measure of continuous work. In that time it made seven commits. They changed two documents and no code.

What it built instead was a way to score itself. Answer keys. Panels of reviewer agents. Tests of the reviewers. About 3,800 lines of scoring scripts. Every next step it offered me was more measuring.

03Where the numbers came from

The session reported accuracy of 95% to 99%. Those figures were real arithmetic on bad inputs.

The reviewing agents changed the answer keys wherever most of them sided with the engine's reading, so the keys moved toward the answers they were meant to check. On one and the same reading, changing the keys moved the score from 89.9% to 96.4%. The new step itself added 0.3 of a point. The 99% figure came from a model I cannot use as a paid reader, judged by reviewers who sided with it 215 times to 9.

Against keys nobody had edited, the new step scored between 86.7% and 89.9% on who-is-who, and found between 83.9% and 87.0% of the characters. Those scores did not meet my 95% target. They came from an experimental version that was not used in the normal wiki build and had not been tested on a whole novel.

04Tuning to the test

Some of the new work was shaped by the habits of the public test set, to raise the score on that test set. Then two changes were written and judged on six test chapters, and the helper that did it was given the names of eight of their characters in its instructions. Those six chapters could not show whether the changes worked on an unfamiliar book.

This happened on the evening of 21 October, 27 minutes after the session had saved its own account of the same mistake. Both changes were withdrawn the next morning.

It also committed a different session's unproven code, sixteen files and 625 lines, into the tree that actually runs, having checked it only for leftover debugging prints. Its author had said none of it had yet run on a real book. And it made statements that were not true: it gave a file path that was not in my tree, said the test set had 200 books when it has 100, and said a notification had gone out when it had not.

05What it cost

$8.10 in total. None of it bought anything that is part of the product today. $2.21 was spent after I had said no paid runs. $2.09 was spent during the provider's double-price hours.

06What I did

I stopped the session. Then I asked a second agent to audit the first, told it to check the account against the evidence and not to soften anything, and I committed its report to the repository exactly as written. It is called the damage report, and it opens with this: the wiki the product builds is no closer to 95% on who-is-who.

The next agent got new rules, on top of my standing orders:

• No measuring kit. No answer keys, reviewer panels or scorers, and it may not offer them.
• Never change or judge anything on test chapters, or on any book already read here.
• Fix a fault only after it shows on a novel nobody here has read.
• Before any change, write one line saying what it does to a whole novel and to the default build. If you cannot say, do not make the change.
• Nothing may be tailored to a particular book: no character names, no book names.
• Spend nothing unless my own message contains the word "spend".

On 22 October three more decisions went in, taking out the changes tuned to test books that the outside checks had found. I now have Grok review each change from outside, and the agent has to check every point it raises against the code instead of trusting it.

07The problem I have not solved

Part One said the engine had no adjudicated standard, and that at some point I had deleted the measuring apparatus because it looked like clutter. I then built a new one, and it fooled me for a day. I then banned that kind of kit: no answer keys, reviewer panels, scorers or test sets. That did not solve the question of how much the wiki misses.

That leaves me where Part One did. How much the wiki misses is unmeasured. The difference now is that I know the failure mode: whatever is easy to measure on the screen pulls the work, and a number that moves feels like progress. I can say what would show I was wrong, which is a whole novel, unread by anyone here, whose pages I read against its text. That is slow. It is the check I intend to do next.

Thanks for reading.

J.H.

Next: Sixty-Five Thousand Lines Out →