Notes from DevDocs team meeting March 30, 2018
Present:
Thibaut (GitHub: @thibaut) Alex Chafee (@alexch) Joshua Burke (?) Quincy Larson (@quincylarson)
During our 70-minute call, we talked about:
We may want to refactor to ES6 to make it easier to contribute to the front end.
App cache - stores the assets and some of the index files for the docs - Mozilla may remove this technology from FireFox.
DevDocs also stores things in local storage as a backup.
IndexDB is used for downloading individual sets of documentation to your browser - stores the HTML files for all the documentation. Basically a key-value store: file name -> HTML file
IndexDB is a hard technology to work with and there are a lot of bugs (pre-dates Promises).
IndexDB is the only storage tool that gives you a lot more data storage than local storage (which is limited to 5 or 10 mb).
IndexDB can use up to half of the storage space on a hard drive. But there's no way to know how much space is left.
There are some new persistence APIs that are being worked on and shipped with new browsers, which we may be able to use in the future.
There are situations where a user runs out of space for IndexDB and all the files can become corrupted. Can also happen when a browser is updated.
All the scraping runs locally, usually once every few weeks to update the documentation.
Some people have used DevDocs as the user agent and done their own scraping more frequently than that, which has caused some organizations like Drupal to complain about the scraping.
Once scraping is done, it needs to be put in two places:
- maxcdn's storage services for people who want to download everything and run devdocs locally
- sync it with the S3 server for DevDocs, so clients can fetch the documentation from there using a XHR
All of the requests you make after opening DevDocs - after you have the app cache manifest - are going directly to S3.
Thibaut maintains a list of RSS feeds so I can know what has been updated, and manually rescrapes the documentation for that project. The tooling provides a progress bar and will let you know what has changed and whether there were any failures.
There is no external database - it's just the JSON manifest file and Sinatra.
Think of the back end and the front end as completely separate codebases. The front end works fine now - most of the maintenance will be on the backend. It will be easier to find contributors for the front end.
- Both Josh and Alex are going to get the current DevDocs project running locally and run some of the scrapers, and take notes during the process so they can update the onboarding documentation
- The four of us will meet again next week