When you start scraping documentation using thor docs:generate <doc>
, Thor will call Docs.generate. This method finds the correct scraper and calls Doc.store_pages, which is the basis of the actual scraping.
This method sets up an EntryIndex in which entries are stored and a PageDb in which processed pages are stored. After that's done, it starts scraping by calling build_pages
on the scraper. For most scrapers this means the Scraper.build_pages method is called, but the Browser Support Tables and in the future the .NET scraper (which I am currently working on) implement the build_pages
method themselves.
The