- Introduction to IndexedDB Promised Library
- Getting Started with Indexed DB
- More IDB
- Cleaning IDB
- Cache Photos
- Cleaning Photo Cache
- Caching Avatars
It is a web browser based database for storing data; supported by most modern browsers. It's a No-SQL db meaning that the data is not relational. IndexedDB allows for multiple databases, which each can have multiple object stores, e.g. ""collections" of "posts"--or literally collections from mongoDB--for each kind of thing you want to store.
Using object stores IndexedDB can store different items or data types: objects, strings or arrays. Items in the object's store can have an primary key (out-of-line keys) or you can assign the property of the values to be the key (in-line keys). Can have out-of-line keys where there is a key value pair or in-line keys where the data stored is a javascript object and the key is the property of the object. Any key must be unique as it becomes the way to identify an item. Items can also be indexed, offering different views of data ordered by particular properties.
All read-write operations are part of a transaction. If any of the operations in a transaction fail the whole transaction is like it never happened. This is similar to SQL commits as much as promises are similar to event-sourcing.
The API to access the database is clumsy and dense. The API asynchronous but has its own kind of events, plus its prone to causing spaghetti code
This library is a small library to use instead of the default API to access the database. It offers access to promises which makes using IndexedDB more manageable.
If you don't want to use the default API then you can use the IndexedDB Promised library. The library can be found here: https://www.npmjs.com/package/idb
idb.open('name' (string), version (number), callback)This will open a connection database.
- idb.open() will return a promise which can then be used to enter transactions in the DB.
- name is the name of the database
- version is a number that definse the version of this database
- The callback functions is what will be called if the database is not found or the version number is different. This is the only place you can remove or add object stores (collections) or indexes. In order to maintain db integrity, you can only create objects stores and indexes within the upgrade function.
The open() method takes a callback function that will be called if the database doesn't exist or the version number doesn't exist. The callback will be passed an object. This object can be used to "upgrade the db".
createObjectStore('storeName', {optionalParameter} ) - can be used to add a new Object Store to the database. For database integrity this can only be done at the open step. The optional parameter is an object. It can take a key-path key that will point to the key path in case you're not using object value pairs. If you want to enter objects as the values into this object store you will have to specify the key that will be used as the index key.
db.transaction(storeNames, mode) - This method is used to create a new transaction in the database. It can be multiple Object Stores as an ARRAY of strings or one. The second parameter is the mode. By default the mode is set to read, this is the fastest access mode but it does not allow writing to the database. The second mode is readwrite wich does allow writing to the database.
tx.ObjectStore('objectStore') - This can be used to return a object Store object. You have to specify the objectStore you'd like to work with.
objectStore.get("key") - This method wil be used to retrieve a value from the object store. It takes one parameter, the key of the value you'd like to retrieve.
objectStore.put('value', 'key') - After an object store is created this can be used to add key value pairs to the object store. The first parameter is the value and the second is the key.
tx.complete - This returns a promise that will be fullfilled if the whole transaction succeeded.
// import the library
import idb from 'idb';
// Open the database text-db version 1, if it's not available it will be created and the callback function will be called.
var dbPromise = idb.open('text-db',1 , function(upgradeDB) {
// Create a Object Store called keyval (in mongoDB this would be a called a collection)
var keyValStore = upgradeDB.createObjectStore('keyval');
// Now that we have an object-store we can work with it. Here we can add an item to the keyval object-store. The item will have a value of world and a key of hello.
keyValStore.put('world', 'hello')
});
dbPromise.then(function(db) {
// Transaction needs to be set up first. You need to pass in the object stores that will be used in the transaction. It's possible to have multiple.
var tx = db.transaction('keyval');
// Next we need to set up the objectStore object. In this case it's the keyal object store.
var keyValStore = tx.objectStore('keyval');
// The next line were using the ObjectStore object and using .get() to retreive the value for the key hello. It will return a promise.
return keyValStore.get('hello');
}).then(function(val) {
console.log("The value of 'hello' is: " + val);
});
dbPromise.then(function(db) {
// Creating a new transaction that will be writing to the database. readwrite is the mode we have to set. keyval is the objectStore we'll be working with.
var tx = db.transaction('keyval', 'readwrite');
// We will need to access the object store of keyval so were storeing that oject in the variable keyValStore.
var keyValStore = tx.objectStore('keyval');
// Now that we have the Object Store object we can call put on it, this will enter the
keyValStore.put('bar', 'foo');
// Because we can have multiple operations we use tx.complete to make sure everythign works. tx.complete returns a promise that will be fullfilled when the whole transaction succeeds or error out if any part of the transaction failed.
return tx.complete;
}).then(function() {
console.log("Added foo:bar to keyval");
})
;Because we want to group data in different ways, we will be adding new objects stores to the database. Important to note that we have to change the version. If we don't, the browser will stop when trying to 'upgradeDb.createObjectStore('keyeval')' because it already exists. So, if we change the version but don't add a case statement on the oldVersion property it will produce an error.
var dbPromise = idb.open('test-db', 2, function(upgradeDb) {
// Switch the built in oldVersion and run each statement if it matches.
switch(upgradeDb.oldVersion) {
// Version 0 already exists so this statement wil not run.
case 0:
var keyValStore = upgradeDb.createObjectStore('keyval');
keyValStore.put("world", "hello");
// Version 1 does not exist in the database at this point so the below code will run.
case 1:
// Create an object store that will be a objects store of objects. They index will be 'name', it should be a unique key that we are using.
upgradeDb.createObjectStore('people', { keyPath: 'name' });If we want to save object to the database we have to create a new objects store and pass it a keyPath. The keyPath will be the index.
upgradeDb.createObjectStore('people', { keyPath: 'name' });- In this case the objects store will be called people
- The key on this objects store will be name, this means every person should have name key/val pair.
This method is similar to get() in that it returns all the key value pairs. However it does not take a key parameter, it will return all objects stored in a particular Object Store.
Indexes can only be created during the updateDB step with a version Upgrade.
// db upgrades have their own transaction object found at upgradeDB.transaction
var exampleStore = upgradeDB.transaction.objectStore('storeName');
exampleStore.createIndex('indexName', 'key');** storeName - The name you want to give the objects store
indexName - The name you want to give to the index, will be used later to retrieve information by this index.
key - The key you want the index to be sorted by.
/// enter into browser console, and will remove the database
indexedDb.delete.database('objects store name')Can be used to iterate over index or object stores. It's beneficial because you can modify or remove objects as you are iterating over them.
someStore.openCursor() - Return a a promise for the first cursor item.
cursor.advance(num) - Skip a couple iterations.
cursor.continue() - Go to the next iteration.
cursor.delete() - Used to delete the item in the current cursor position
cursor.update(newVal) - Used to update the cursor with a new value.
dbPromise.then(function(db) {
var tx = db.transaction('people');
var peopleStore = tx.objectStore('people');
var ageIndex = peopleStore.index('age');
// Rather than getAll() we can
return ageIndex.openCursor();
}).then(function(cursor) {
if (!cursor) return;
return cursor.advance(2);
}).then(function logPerson(cursor) {
if (!cursor) return;
console.log("Cursored at:", cursor.value.name);
// I could also do things like:
// cursor.update(newValue) to change the value, or
// cursor.delete() to delete this entry
return cursor.continue().then(logPerson);
}).then(function() {
console.log('Done cursoring');
});import idb from 'idb';
var dbPromise = idb.open('test-db', 4, function(upgradeDb) {
switch(upgradeDb.oldVersion) {
case 0:
var keyValStore = upgradeDb.createObjectStore('keyval');
keyValStore.put("world", "hello");
case 1:
upgradeDb.createObjectStore('people', { keyPath: 'name' });
case 2:
var peopleStore = upgradeDb.transaction.objectStore('people');
peopleStore.createIndex('animal', 'favoriteAnimal');
case 3:
var peopleStore = upgradeDb.transaction.objectStore('people');
peopleStore.createIndex('age', 'age');
}
});
// get the value of hello key
dbPromise.then(function(db) {
var tx = db.transaction('keyval');
var keyValStore = tx.objectStore('keyval');
return keyValStore.get('hello');
}).then(function(val) {
console.log('The value of "hello" is:', val);
});
// set "foo" to be "bar" in "keyval"
dbPromise.then(function(db) {
var tx = db.transaction('keyval', 'readwrite');
var keyValStore = tx.objectStore('keyval');
keyValStore.put('bar', 'foo');
return tx.complete;
}).then(function() {
console.log('Added foo:bar to keyval');
});
dbPromise.then(function(db) {
var tx = db.transaction('keyval', 'readwrite');
var keyValStore = tx.objectStore('keyval');
keyValStore.put('cat', 'favoriteAnimal');
return tx.complete;
}).then(function() {
console.log('Added favoriteAnimal:cat to keyval');
});
// add people to "people"
dbPromise.then(function(db) {
var tx = db.transaction('people', 'readwrite');
var peopleStore = tx.objectStore('people');
peopleStore.put({
name: 'Sam Munoz',
age: 25,
favoriteAnimal: 'dog'
});
peopleStore.put({
name: 'Susan Keller',
age: 34,
favoriteAnimal: 'cat'
});
peopleStore.put({
name: 'Lillie Wolfe',
age: 28,
favoriteAnimal: 'dog'
});
peopleStore.put({
name: 'Marc Stone',
age: 39,
favoriteAnimal: 'cat'
});
return tx.complete;
}).then(function() {
console.log('People added');
});
// list all cat people
dbPromise.then(function(db) {
var tx = db.transaction('people');
var peopleStore = tx.objectStore('people');
var animalIndex = peopleStore.index('animal');
return animalIndex.getAll('cat');
}).then(function(people) {
console.log('Cat people:', people);
});
// list all people ordered by age
dbPromise.then(function(db) {
var tx = db.transaction('people');
var peopleStore = tx.objectStore('people');
var ageIndex = peopleStore.index('age');
return ageIndex.getAll();
}).then(function(people) {
console.log("People by Age:", people);
});We don't want to continuously add posts, this will cause our DB to run out of memory. Instead we should remove old posts from the database. In the following exmaple we are removing all old posts and only keeping the 30 latest posts.
// Use the index store, this way the items will be arranged by oldest to newest.
// openCursor(null, 'prev') starts the cursor at the last item (in this case the last item is the newst post).
store.index('by-date').openCursor(null, 'prev').then(function(cursor) {
// Skip the first 30 items, we don't want to delete those.
return cursor.advance(30);
// Next we want to start deleting. We are setting up a recursive operation here.
// Once the first item is deleted it will then call cursor.continue().then(deleteRest) which will delete the next item
// And so on and so on until the cursor is empty and there are no more items left.
// At that point the line if (!cursor) return; will be run and the recursive stack will end and the original function will return.
}).then(function deleteRest(cursor) {
if (!cursor) return;
cursor.delete();
cursor.continue().then(deleteRest);
});So far we are cacheing all our static content, the page skeleton and some icons that are used on the page. We are also saving posts to indexedDB but we are not cacheing the photos that come with posts. We need to add those to the cache as each reposne is made from the browser.
Our request/response is used up once we do something with it so we need to clone it before messing with it. This way we can do two differetnt things with our response. We can for example, send it to the cache, and then send it to the web site.
We have to add a new cache and make sure that we are not deleting it in our activate event.
// Were saving the cache name in a variable for use later.
var contentImgCache = 'wittr-content-imgs';
// Also creating an array of caches that we want to keep. This will be used in the activate event below to make sure we're not deleting these two caches when doing our clean up.
var allCaches = [staticCacheName, contentImgCache];In activate event:
self.addEventListener('activate', function(event) {
event.waitUntil(
caches.keys().then(function(cacheNames) {
return Promise.all(
cacheNames.filter(function(cacheName) {
// If the cache starts with 'wittr-' and is not one of our cacheNames listed in an array then it will be returned by filer and later delete in the .map function.
// the includes method is ES2016 method on arrays or strings. It can be used to check if an array has a certain value.
return cacheName.startsWith('wittr-') &&
!allCaches.includes(cacheName);
}).map(function(cacheName) {
return caches.delete(cacheName);
})
);
})
);
});Next we have to add a new check to for all fetch events that start with /photos/. If the front end makes a fetch request for a URL that has /photos/ in it, we want to intercept that fetch request, check if it is in the cache. If the request is in the cache we want to serve it up from the cache, otherwise we can get it from the network/server and then cache it and send it back to the browser/client.
// If the request URL starts wit photos we want to intercept it.
if (requestUrl.pathname.startsWith('/photos/')) {
// We'll respond to the request with whatever is returned by servePhoto() function. Well pass our request to that function.
event.respondWith(servePhoto(event.request));
return;
}The servePhoto function:
function servePhoto(request) {
// URL look like /phots/9-892340-23422jiojiofsj-fwfsdf-fsdffsd-800px.jpg
// Since we want to save the URL without the size information
var storageUrl = request.url.replace(/-\d+px\.jpg$/,'');
//Open the cache - the return statement is needed because whatever will be returned back from this whole code will be returned back to the respondWith() function and sent to the client.
return caches.open(contentImgCache).then(function(cache) {
// Next we'll do a match on the storageURL. If we alraedy have the image cached we want to serve that up.
return cache.match(storageUrl).then(function(response) {
// If we get a reponse from the match then we return the respone and we're done.
if (response) return response;
// Otherwise we have to go to the network and fetch the image.
return fetch(request).then(function(networkResponse) {
// Now that we got the reaponse we can put it into the cache. We'll save it with the storegeUrl as the request. This way it will not include the image size so that when we do the match above it be found and served. We also have to do a clone() on the response from the network because requests and responses are streams they can only be used once. Cloning it allows us to do two things with them. 1. Save to cache and 2. return back to respondWith().
cache.put(storageUrl, networkResponse.clone());
// And we have to return the original response back to respondWith().
return networkResponse;
});
});
});
}We can't just keep adding photos to cache. We cne
cache.delete(request) - You can use this method to pass in a specific request to delete the cache for that request.
Returns a promise that gives as all requests for entreis in the cache.
cache.keys().then(function(requests)) {
// ..
}The following code is grabbing all of the witter informatoin from indexedDB and then comparing the posts with photos with the image cache. Any photos that do not exist in indexedDB are deleted.
It's also a good idea to wrap the code in a set interval so that it runs every 5 minutes. This makes sure that if the user has the page open for a long time, the cache does not run out of space.
setInterval(function() {
indexController._cleanImageCache();
}, 1000 * 60 * 5);IndexController.prototype._cleanImageCache = function() {
return this._dbPromise.then(function(db) {
if (!db) return;
// Create an array to hold all the images we want to keep.
var imagesNeeded = [];
var tx = db.transaction('wittrs');
// Get all witter posts in IndexedDB
return tx.objectStore('wittrs').getAll()
.then(function(messages) {
// For each post we need to check if it has a photo
messages.forEach(function(message) {
// If the post has a photo attached to it we want to push it to our array of photos we want to keep.
if (message.photo) {
imagesNeeded.push(message.photo);
}
});
// Now that we have all the photos we want to keep we ahve to open the cache of photos. We put return here because once this function ends we want the whole function to return.
return caches.open('wittr-content-imgs').then(function(cache) {
// We use cache.keys to pull all requests available in the cache.
return cache.keys().then(function(requests) {
// Now we have to compare each request in the cache with the images we want to keep.
requests.forEach(function(request) {
// We have to convert the URL to a URL object so we can access the pathname property.
var url = new URL(request.url);
// if the imagesNeeded does not have the request then we can delete it. We're using .pathname becuase the URL in the cache is the full URL and the URl in our images needed is the relative path. To compare the two we need them to beoth be paths.
if (!imagesNeeded.includes(url.pathname)) {
cache.delete(request);
}
});
});
});
});
});
};First we have to add a check in our fetch event handler on the service worker to make sure that any fetch request for an avatar is intercepted by the fetch handler on the service worker. Once the request is handled it'll call serveAvatar fucntion ang whatever is resturned from that function will be the response to the client.
if (requestUrl.pathname.startsWith('/avatars/')) {
event.respondWith(serveAvatar(event.request));
return;
}Next the serveAvatar function will take in the request from the the page and pull the avatar from the cache if it exists. It will also make a request to the network and pull it from the network and update the cache. This will make sure we always have the latest avatar.
function serveAvatar(request) {
// Avatar urls look like:
// avatars/sam-2x.jpg
// But storageUrl has the -2x.jpg bit missing.
// Use this url to store & match the image in the cache.
// This means you only store one copy of each avatar.
var storageUrl = request.url.replace(/-\dx\.jpg$/, '');
// Opne the cache
return caches.open(contentImgsCache).then(function(cache) {
// Check if the cache has a match fro the URL
return cache.match(storageUrl).then(function(response) {
// Next we're going to do a fetch to the network for the avatar
var networkFetch = fetch(request).then(function(networkResponse) {
// we will put the image fro mthe network into the cache
cache.put(storageUrl, networkResponse.clone());
// and we'll return the network reponse from the promise
return networkResponse;
});
// If the image was found in the cache it will not be null and we'll return that, otherwise we'll return the networkFetch (image from the network).
return response || networkFetch;
});
});
}