Show HN: I built "AI Wattpad" to eval LLMs on fiction

by jauws on 2/3/2026, 5:08:43 PM

I&#x27;ve been a webfiction reader for years (too many hours on Royal Road), and I kept running into the same question: which LLMs actually write fiction that people want to keep reading? That&#x27;s why I built Narrator (<a href="https:&#x2F;&#x2F;narrator.sh&#x2F;llm-leaderboard" rel="nofollow">https:&#x2F;&#x2F;narrator.sh&#x2F;llm-leaderboard</a>) – a platform where LLMs generate serialized fiction and get ranked by real reader engagement.<p>Turns out this is surprisingly hard to answer. Creative writing isn&#x27;t a single capability – it&#x27;s a pipeline: brainstorming → writing → memory. You need to generate interesting premises, execute them with good prose, and maintain consistency across a long narrative. Most benchmarks test these in isolation, but readers experience them as a whole.<p>The current evaluation landscape is fragmented: Memory benchmarks like FictionLive&#x27;s tests use MCQs to check if models remember plot details across long contexts. Useful, but memory is necessary for good fiction, not sufficient. A model can ace recall and still write boring stories.<p>Author-side usage data from tools like Novelcrafter shows which models writers prefer as copilots. But that measures what&#x27;s useful for human-AI collaboration, not what produces engaging standalone output. Authors and readers have different needs.<p>LLM-as-a-judge is the most common approach for prose quality, but it&#x27;s notoriously unreliable for creative work. Models have systematic biases (favoring verbose prose, certain structures), and &quot;good writing&quot; is genuinely subjective in ways that &quot;correct code&quot; isn&#x27;t.<p>What&#x27;s missing is a reader-side quantitative benchmark – something that measures whether real humans actually enjoy reading what these models produce. That&#x27;s the gap Narrator fills: views, time spent reading, ratings, bookmarks, comments, return visits. Think of it as an &quot;AI Wattpad&quot; where the models are the authors.<p>I shared an early DSPy-based version here 5 months ago (<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=44903265">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=44903265</a>). The big lesson: one-shot generation doesn&#x27;t work for long-form fiction. Models lose plot threads, forget characters, and quality degrades across chapters.<p>The rewrite: from one-shot to a persistent agent loop<p>The current version runs each model through a writing harness that maintains state across chapters. Before generating, the agent reviews structured context: character sheets, plot outlines, unresolved threads, world-building notes. After generating, it updates these artifacts for the next chapter. Essentially each model gets a &quot;writer&#x27;s notebook&quot; that persists across the whole story.<p>This made a measurable difference – models that struggled with consistency in the one-shot version improved significantly with access to their own notes.<p>Granular filtering instead of a single score:<p>We classify stories upfront by language, genre, tags, and content rating. Instead of one &quot;creative writing&quot; leaderboard, we can drill into specifics: which model writes the best Spanish Comedy? Which handles LitRPG stories with Male Leads the best? Which does well with romance versus horror?<p>The answers aren&#x27;t always what you&#x27;d expect from general benchmarks. Some models that rank mid-tier overall dominate specific niches.<p>A few features I&#x27;m proud of:<p>Story forking lets readers branch stories CYOA-style – if you don&#x27;t like where the plot went, fork it and see how the same model handles the divergence. Creates natural A&#x2F;B comparisons.<p>Visual LitRPG was a personal itch to scratch. Instead of walls of [STR: 15 → 16] text, stats and skill trees render as actual UI elements. Example: <a href="https:&#x2F;&#x2F;narrator.sh&#x2F;novel&#x2F;beware-the-starter-pet&#x2F;chapter&#x2F;1" rel="nofollow">https:&#x2F;&#x2F;narrator.sh&#x2F;novel&#x2F;beware-the-starter-pet&#x2F;chapter&#x2F;1</a><p>What I&#x27;m looking for:<p>More readers to build out the engagement data. Also curious if anyone else working on long-form LLM generation has found better patterns for maintaining consistency across chapters – the agent harness approach works but I&#x27;m sure there are improvements.

https://narrator.sh/llm-leaderboard

Comments