What 400-page PDFs taught us about chunking
Our first curriculum generator split by page count. It was terrible. Here is what we do now, and the three failure modes we hit on the way.
The first version of our curriculum generator was about forty lines long. It took a PDF, cut it into equal chunks of roughly 2,000 words, and asked a model to write five questions about each chunk.
It produced something. What it produced was unusable, in three distinct and instructive ways.
Failure one: page boundaries are not idea boundaries
Splitting by length cuts arguments in half. A chunk would end four sentences into a definition, and the model — being obliging — would write questions about the four sentences it could see. The result was a set of questions that were all individually answerable and collectively pointless, like being quizzed on the first half of every sentence in a book.
Worse, the second half of that definition opened the next chunk with no context, so the model invented the missing setup. Confidently.
What we do now: parse the document's structure first — headings, sections, figure captions, numbered definitions — and chunk on those boundaries, letting chunk size vary. A section that runs three pages stays one unit. A dense page with six defined terms becomes six.
Failure two: not everything in a book is teachable
Academic PDFs are full of material that looks like content and isn't: acknowledgements, reference lists, running headers, copyright pages, exercise answers, the index. Our first version dutifully generated questions about the bibliography. One tester was asked which journal published a paper cited in a footnote. They were, reasonably, unimpressed.
What we do now: classify each block before it gets near a question generator. Front matter, back matter, references and repeated page furniture are extracted and excluded. Exercise sections are kept but flagged, because the questions already in a textbook are usually better than anything we would write.
Failure three: two-column layouts
This one is embarrassingly mundane and cost us the most time. Naive text extraction from a two-column academic paper reads straight across the page, interleaving the two columns line by line. The output is grammatically shaped nonsense — and critically, it looks fine to a model, which will happily generate plausible questions about sentences that were never written.
What we do now: detect column layout geometrically before extracting text, and reconstruct reading order from block positions rather than trusting the extractor's default. We validate the result by checking sentence well-formedness; if a document fails that check, it goes down a slower OCR path rather than producing garbage quickly.
The general lesson
Every one of these failures had the same shape: the pipeline produced confident output that was wrong in a way the model downstream couldn't detect. A chunk of interleaved columns doesn't announce itself as broken. A bibliography doesn't announce that it isn't teachable.
So the rule we ended up with is that validation belongs before generation, not after. It is much cheaper to notice a document is two-column than to review every question it produced. Most of the engineering in DuckEars now sits in that pre-processing layer, and it is the least glamorous and most valuable code in the product.
The other lesson: 412 pages is our current record for a single upload, and it was a chemistry textbook that broke all three of the above at once. If you have something you think will break the fourth thing, we would genuinely like to see it.