groktocrawl scrape https://www.gutenberg.org/ebooks/11→ full Alice in Wonderland as chapter-structured markdown.
Public-domain books are the largest open corpus of high-quality text on the internet. But getting them into a format an LLM, RAG system, or agent can use means:
- Finding the right download URL (EPUB? plain text? illustrated?)
- Dealing with Gutenberg's boilerplate header/footer in plain-text downloads
- Extracting chapter boundaries from EPUB XHTML files or inconsistent plain-text heading patterns
- Splitting large texts into manageable chunks
The Gutenberg adapter solves all of this with a single groktocrawl scrape call.
The adapter intercepts any gutenberg.org/ebooks/<id> URL and runs a three-tier fallback chain:
| Tier | Source | When |
|---|---|---|
| 1. EPUB | Download .epub, extract XHTML, markdownify per file, detect chapter headings |
Preferred — handles illustrated books, correct encoding |
| 2. Plain text | Download pg{id}.txt, strip boilerplate via regex, split by CHAPTER/Chapter/Book headings |
Fallback — works for any text-only book |
| 3. Generic pipeline | Readability-lxml → browser render | Last resort (raises AdapterError to fall through) |
Metadata is enriched via the Gutendex API — title, author (with life dates), language, subjects, download count. The API call has a 3s timeout and failures are non-fatal (EPUB internal metadata fills the gap).
$ groktocrawl scrape https://www.gutenberg.org/ebooks/11YAML frontmatter:
---
title: Alice's Adventures in Wonderland
author: Lewis Carroll
gutenberg_id: 11
language: en
subjects:
- Fantasy fiction
- Children's stories
- Imaginary places -- Juvenile fiction
- Alice (Fictitious character from Carroll) -- Juvenile fiction
chapters:
- The Project Gutenberg eBook of Alice's Adventures in Wonderland
- CHAPTER I. Down the Rabbit-Hole
- CHAPTER II. The Pool of Tears
- CHAPTER III. A Caucus-Race and a Long Tale
- CHAPTER IV. The Rabbit Sends in a Little Bill
- CHAPTER V. Advice from a Caterpillar
- CHAPTER VI. Pig and Pepper
- CHAPTER VII. A Mad Tea-Party
- CHAPTER VIII. The Queen's Croquet-Ground
- CHAPTER IX. The Mock Turtle's Story
- CHAPTER X. The Lobster Quadrille
- CHAPTER XI. Who Stole the Tarts?
- CHAPTER XII. Alice's Evidence
word_count: 29897
source: gutenberg-epub
---Chapters are separated by --- horizontal rules with ## Chapter N: Title heading markup:
## CHAPTER I. Down the Rabbit-Hole
Alice was beginning to get very tired of sitting by her sister on the bank,
and of having nothing to do: once or twice she had peeped into the book her
sister was reading, but it had no pictures or conversations in it...
---
## CHAPTER II. The Pool of Tears
...$ groktocrawl scrape https://www.gutenberg.org/ebooks/84---
title: Frankenstein; or, the modern prometheus
author: Mary Wollstonecraft Shelley
gutenberg_id: 84
language: en
subjects:
- Science fiction
- Horror tales
- Gothic fiction
- Scientists -- Fiction
- Monsters -- Fiction
- Frankenstein, Victor (Fictitious character) -- Fiction
chapters:
- Chapter 1
- Chapter 2
- ...
- Chapter 24
word_count: 78538
source: gutenberg-epub
---24 chapters, 78K words. Each chapter appears as ## Chapter N in the markdown, separated by ---.
Chapter 1 begins:
I am by birth a Genevese, and my family is one of the most distinguished of that republic. My ancestors had been for many years counsellors and syndics, and my father had filled several public situations with honour and reputation...
$ groktocrawl scrape https://www.gutenberg.org/ebooks/1342---
title: Pride and Prejudice
author: Jane Austen
gutenberg_id: 1342
language: en
subjects:
- England -- Fiction
- Young women -- Fiction
- Love stories
- Sisters -- Fiction
word_count: 131977
source: gutenberg-epub
---131K words, full novel extracted. Note: some EPUB editions use non-standard chapter heading markup in their XHTML. The adapter's heading-detection heuristics catch most patterns but may merge chapters in rare cases. The full text is always present regardless of chapter boundary accuracy — all XHTML files are converted to markdown.
For programmatic consumption, append --json:
$ groktocrawl scrape https://www.gutenberg.org/ebooks/11 --jsonReturns structured JSON with metadata and markdown fields — ready for ingestion into vector databases, RAG pipelines, or LLM context windows.
- No API keys. Project Gutenberg is entirely free and unauthenticated.
- No new dependencies. Uses
zipfile(stdlib) +beautifulsoup4/markdownify/httpx(already in the scraper-svc). - Auto-registered. Drop
gutenberg.pyintoscraper-svc/scraper/adapters/— the@adapterdecorator andAdapterRegistry.load_all()handle discovery. - Priority 200. Same tier as GitHub files, YouTube, Substack — tried before the generic pipeline.
The adapter (scraper-svc/scraper/adapters/gutenberg.py, ~530 lines) follows the same pattern as the 20 other adapters in the codebase:
@adapter
class GutenbergAdapter(SiteAdapter):
name = "gutenberg"
patterns = [...] # regexes for /ebooks/<id>, /files/<id>/, /cache/epub/<id>/
priority = 200
async def scrape(self, url: str, ctx: AdapterContext) -> AdapterResult:
# Tier 1: EPUB via zipfile + markdownify
# Tier 2: Plain text via boilerplate strip + heading regex
# Tier 3: Raise AdapterError → generic pipeline
...Every book on gutenberg.org (70,000+ public-domain works) works out of the box. No registration, no rate limiting, no API keys. Use cases:
- RAG corpus: Scrape multiple books into a vector index for agentic retrieval
- LLM training prep: Chunked, cleaned, chapter-structured text ready for fine-tuning
- Literary analysis: Extract chapter metadata, word counts, and structure for quantitative analysis
- Reading pipelines: Feed books into a TTS engine or reading app with stable, predictable output
- Issue #181 — Project Gutenberg adapter specification
- PR #182 — Implementation
scraper-svc/scraper/adapters/gutenberg.py— adapter sourcedocs/adr/0001-0009— adapter framework architecture decisions