Re: https://twitter.com/ppk/status/587545743625478144

browser-storage.md

@ppk asks: “OK, suggestions for my next bit of research?” — https://twitter.com/ppk/status/587545743625478144

@rem replies: “@ppk storage. What works in browser, what browsers, what phones, how it changes when *added to home screen", limits, etc. How's that?” — https://twitter.com/rem/status/587566978065367040

Here’s what I whipped up, please add more in the comments:

APIs:

• localStorage: https://developer.mozilla.org/en-US/docs/Web/API/Storage/LocalStorage / http://dev.w3.org/html5/webstorage/#dom-localstorage / http://dev.w3.org/html5/webstorage/#storage-0
  • Chrome Storage (Chrome apps): https://developer.chrome.com/apps/storage
• IndexedDB: https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API
• WebSQL (although discontinued, still widely available and occasionaly faster or less buggy than IndexedDB): http://www.w3.org/TR/webdatabase/
• Cordova SQLite Plugin (to overcome storage limits): https://github.com/litehelpers/Cordova-sqlite-storage

Topics:

• compability across a broad browser matrix (hello Mobile Safari)
• storage limits across browsers and OSs
• eviction policies per browser/OS
• storing objects vs. storing text vs. storing binary data / Blobs

See also:

• all the research that went into PouchDB, including:

• http://pouchdb.com

• http://pouchdb.com/blog/

  • http://pouchdb.com/2014/10/26/10-things-i-learned-from-reading-and-writing-the-pouchdb-source.html
  • http://pouchdb.com/2014/09/22/3.0.6.html
  • http://nolanlawson.com/2014/04/26/web-sql-database-in-memoriam/

• http://offlinefirst.org / https://github.com/offlinefirst/research

There's also the cache API from ServiceWorker https://slightlyoff.github.io/ServiceWorker/spec/service_worker/#cache-objects

http://www.sitepoint.com/html5-local-storage-revisited/
https://github.com/tantaman/LargeLocalStorage
https://github.com/mozilla/localForage / http://mozilla.github.io/localForage/

Looking for

• simplified overview of WebSQL (basically a list of methods and properties with summaries of their use)
• info on FileAPI (again, list of methods and properties); got intro on http://www.html5rocks.com/en/tutorials/file/filesystem/ , but a good list would be even better
• NO tools please; I only test browsers

Existing research into browser storage quotas: http://www.html5rocks.com/en/tutorials/offline/quota-research/

@ppk: listed PouchDB because they have tons of stuff on underlying storage mechanisms.

IndexedDB/WebSQL browser quirks:

• http://www.raymondcamden.com/2014/09/25/IndexedDB-on-iOS-8-Broken-Bad#comment-1898600588
  • Don't miss the comments! Includes stuff about homescreen apps, UIWebView, WKWebView
• https://gist.github.com/nolanlawson/08eb857c6b17a30c1b26
  • I compiled some of the more egregious bugs here
• http://pouchdb.com/2014/09/22/3.0.6.html
• http://w3c.github.io/test-results/IndexedDB/all.html
  • IndexedDB test results for all browsers:

Cool stuff you can do with IndexedDB (Dexie is awesome btw):

• https://hacks.mozilla.org/2014/06/breaking-the-borders-of-indexeddb/

IndexedDB 2.0 proposal:

• https://www.w3.org/2008/webapps/wiki/IndexedDatabaseFeatures

Obscure storage methods:

• window.name hack: https://twitter.com/nolanlawson/status/579006280381173760
• others listed by Lawnchair (blackberry, ie userdata, window.name): http://brian.io/lawnchair/adapters/

One bit of research that hasn't been explored yet AFAIK: it seems "officially" both WebSQL and IndexedDB are ephemeral (see this and this), but in practice I've only heard of iOS randomly deleting data. And even then, it's just hearsay. It would be interesting to know which browsers wipe data, what the threshold is, etc.

@janl: Yes, I get that, but others might want me to test tools or go through a script they wrote. I won't.

@ppk cool!

@pp-koch: the surprising truth about WebSQL is that it really is just SQLite. I've found the question isn't "which SQLite methods can you use?" but rather "which ones can't you use?".

E.g. we use sqlite_master in PouchDB itself, and full-text search is also supported (demo). I know pragma is unsupported, not sure about more exotic ones like vacuum.

http://html5doctor.com/introducing-web-sql-databases/
https://github.com/axemclion/IndexedDBShim

https://code.tutsplus.com/tutorials/toying-with-the-html5-file-system-api--net-24719
http://caniuse.com/#feat=fileapi
http://caniuse.com/#feat=filesystem
http://dev.w3.org/2006/webapi/FileAPI/
http://www.w3.org/TR/file-upload/
http://www.w3.org/TR/file-writer-api/

http://html5demos.com/file-api

Also the issue of binary data in the browser (blobs, arraybuffers, etc.) is probably worthy of its own blog post, but there's some info here:

• http://pouchdb.com/faq.html#data_types (Chrome has had four major Blob bugs which they finally fixed as of v43, IE and Firefox has been good pretty much since day 1, Safari doesn't support it in IndexedDB among other stuff they don't support, binary data in WebSQL is super weird)
• https://github.com/nolanlawson/blob-util (some mention of which APIs aren't supported by which browsers, with Android/Firefox/IE being the big offenders, but really you'd have to see the source code to know exactly which ones, mostly BlobBuilder/canvas/FileReader stuff)

Also sorry I'm focusing mostly on the negative, but figured it's best to take a hard look at browser storage in all its gory detail. 😃

@nolanlawson: Fine, but I know nothing about SQL, so WebSQL being SQLite doesn't really help me.

In fact, this may be a serious problem for my research. But I'll start with localStorage, I decided that much, particularly since I have some old tests lying around somewhere.

AppCache, especially in the phone/"save to home screen" contexts, and since other storage is so often used to get around its "quirks". @jaffathecake has already done a lot of the work around it.

http://alistapart.com/article/application-cache-is-a-douchebag
http://flailingmonkey.com/application-cache-not-a-douchebag/
https://developer.mozilla.org/en-US/docs/Web/HTML/Using_the_application_cache
https://developer.mozilla.org/en-US/docs/Web/API/Window/applicationCache
https://msdn.microsoft.com/en-us/library/ie/hh771715%28v=vs.85%29.aspx
http://appcache.offline.technology/

In save to homescreen mode (Chrome) there will be no difference today. Other browsers though, would be interesting to know

I know that iOS Safari drops indexeddb support when launched from homescreen

Is launched from homescreen a different use case than using the WebView?

@pp-koch: UIWebView, WKWebView, homescreen app, iOS Safari, and even Chrome on Safari (!): none of the above should be considered equivalent. I've seen WebSQL/IndexedDB issues that occur in only one of the above. The Raymond Camden article has them all listed in the comments.