Skip to content

Instantly share code, notes, and snippets.

@loilo
Last active April 22, 2024 12:17
Show Gist options
  • Star 70 You must be signed in to star a gist
  • Fork 15 You must be signed in to fork a gist
  • Save loilo/ed43739361ec718129a15ae5d531095b to your computer and use it in GitHub Desktop.
Save loilo/ed43739361ec718129a15ae5d531095b to your computer and use it in GitHub Desktop.
Back up and restore an IndexedDB database

Back up and restore an IndexedDB database

This gist is an ES module which provides functions to import and export data from an IndexedDB database as JSON. It's based on Justin Emery's indexeddb-export-import package, but applies some adjustments that reflect better on the current browser landscape (i.e. better developer ergonomics but no support for Internet Explorer).

Usage

For each of the provided functionalities, you need a connected IDBDatabase instance.

Export Data

import { idb } from 'some-database'

import { exportToJson } from 'idb-backup-and-restore.mjs'

exportToJson(idb)
  .then(result => {
    console.log('Exported JSON string:', result)
  })
  .catch(error => {
    console.error('Something went wrong during export:', error)
  })

Import Data

import { idb } from 'some-database'
import { serializedData } from 'some-serialized-data'

import { importFromJson } from 'idb-backup-and-restore.mjs'

importFromJson(idb, serializedData)
  .then(() => {
    console.log('Successfully imported data')
  })
  .catch(error => {
    console.error('Something went wrong during import:', error)
  })

Clear Database

Depending on your use case, it can be reasonable to clear a database before importing serialized data:

import { idb } from 'some-database'
import { serializedData } from 'some-serialized-data'

import { importFromJson, clearDatabase } from 'idb-backup-and-restore.mjs'

clearDatabase(idb)
  .then(() => importFromJson(idb, serializedData))
  .then(() => {
    console.log('Successfully cleared database and imported data')
  })
  .catch(error => {
    console.error('Could not clear & import database:', error)
  })
/**
* Export all data from an IndexedDB database
*
* @param {IDBDatabase} idbDatabase The database to export from
* @return {Promise<string>}
*/
export function exportToJson(idbDatabase) {
return new Promise((resolve, reject) => {
const exportObject = {}
if (idbDatabase.objectStoreNames.length === 0) {
resolve(JSON.stringify(exportObject))
} else {
const transaction = idbDatabase.transaction(
idbDatabase.objectStoreNames,
'readonly'
)
transaction.addEventListener('error', reject)
for (const storeName of idbDatabase.objectStoreNames) {
const allObjects = []
transaction
.objectStore(storeName)
.openCursor()
.addEventListener('success', event => {
const cursor = event.target.result
if (cursor) {
// Cursor holds value, put it into store data
allObjects.push(cursor.value)
cursor.continue()
} else {
// No more values, store is done
exportObject[storeName] = allObjects
// Last store was handled
if (
idbDatabase.objectStoreNames.length ===
Object.keys(exportObject).length
) {
resolve(JSON.stringify(exportObject))
}
}
})
}
}
})
}
/**
* Import data from JSON into an IndexedDB database.
* This does not delete any existing data from the database, so keys may clash.
*
* @param {IDBDatabase} idbDatabase Database to import into
* @param {string} json Data to import, one key per object store
* @return {Promise<void>}
*/
export function importFromJson(idbDatabase, json) {
return new Promise((resolve, reject) => {
const transaction = idbDatabase.transaction(
idbDatabase.objectStoreNames,
'readwrite'
)
transaction.addEventListener('error', reject)
var importObject = JSON.parse(json)
for (const storeName of idbDatabase.objectStoreNames) {
let count = 0
for (const toAdd of importObject[storeName]) {
const request = transaction.objectStore(storeName).add(toAdd)
request.addEventListener('success', () => {
count++
if (count === importObject[storeName].length) {
// Added all objects for this store
delete importObject[storeName]
if (Object.keys(importObject).length === 0) {
// Added all object stores
resolve()
}
}
})
}
}
})
}
/**
* Clear a database
*
* @param {IDBDatabase} idbDatabase The database to delete all data from
* @return {Promise<void>}
*/
export function clearDatabase(idbDatabase) {
return new Promise((resolve, reject) => {
const transaction = idbDatabase.transaction(
idbDatabase.objectStoreNames,
'readwrite'
)
transaction.addEventListener('error', reject)
let count = 0
for (const storeName of idbDatabase.objectStoreNames) {
transaction
.objectStore(storeName)
.clear()
.addEventListener('success', () => {
count++
if (count === idbDatabase.objectStoreNames.length) {
// Cleared all object stores
resolve()
}
})
}
})
}
@jimecox2
Copy link

An awesome set of functions. Thank You very much for sharing.

@tomsaleeba
Copy link

tomsaleeba commented Jul 8, 2020

For anyone else like me who doesn't reguarly use the raw IDB API, you can get an IDBDatabase instance and call one of these function with this:

var conn = indexedDB.open('some-db', 1) // change the name and version as needed
connection.onsuccess = e => {
  var database = e.target.result
  exportToJson(database).then(console.log).catch(console.error)
}

BTW, thanks for the useful functions πŸŽ‰

@sombhattacharyya1983
Copy link

I am looking to export a json from a indexeddb. Wondering how well this will work for a ~200 MB size dump. I am being able to do it for 60 MB. Have you tried a similar use case?

@xmedeko
Copy link

xmedeko commented Sep 23, 2020

Very nice, thanks. Just the bare export/import function should return the export object instead of JSON string. The user may do JSON stuff by herself, then (or make wrappers). E.g. to clone a database one does not need to make JSON.

@shellking4
Copy link

Great ! Thank you so much for these helpful functions. I just can't figure out how to use import statement in the browser. How do I run the functions in chrome console for example ?

@loilo
Copy link
Author

loilo commented Mar 1, 2021

@shellking4 What do you refer to with "import statement"? Importing stuff from the idb-backup-and-restore.mjs file or calling the importFromJson function?

@shellking4
Copy link

@shellking4 What do you refer to with "import statement"? Importing stuff from the idb-backup-and-restore.mjs file or calling the importFromJson function?

Yes you get it right. How can I do it in my browser console ?

@loilo
Copy link
Author

loilo commented Mar 1, 2021

No, I'm sorry, I don't get it right: What do you want to do in the console?
Do you mean
a) importing stuff from the idb-backup-and-restore.mjs or
b) calling the importFromJson function?

@shellking4
Copy link

No, I'm sorry, I don't get it right: What do you want to do in the console?
Do you mean
a) importing stuff from the idb-backup-and-restore.mjs or
b) calling the importFromJson function?

well... I think wanna do both. The whole point is that I have data in indexedDB and in a few weeks I will wipe the system out because of some graphical problems. But I'd really like to backup the data I have in indexedDB and restore them later on. In my search for solutions I luckily landed here.

@loilo
Copy link
Author

loilo commented Mar 2, 2021

Okay, that should be relatively straightforward, under one condition: You'll still have the IndexedDB on the new machine, even though it may be empty. This is necessary because the export script from above can only export the DB's data, not its strucure.

For both exporting and importing data, you'll need an IndexedDB instance. Say you're wanting to back up the foo database, then you could get an instance in the console like this:

var db
var request = window.indexedDB.open('foo')
request.onerror = () => console.error('Could not open database')
request.onsuccess = () => { db = request.result }

Export:

  1. Copy the exportToJson function (without the prepended export statement), paste it into the DevTools Console and hit Enter.
  2. Run this: await exportToJson(db)
  3. If you see some JSON output in the console it means that the export worked. Run copy($_) to copy the return value of the previous command (= the exported data) to your clipboard.
  4. Save that exported data somewhere.

Import:

  1. Copy the importFromJson function (without the prepended export statement), paste it into the DevTools Console and hit Enter.

  2. Invoke it with the IndexedDB instance and the exported data as parameters: await importFromJson(db, exportedString).

    Note that you actually have to pass the exported data as a string, not as a JSON object. For example: If you exported and empty database {}, you'd have to pass into the function the following: await importFromJson(db, '{}').


This should be it. Probably test it before you actually rely on it. πŸ™‚

@shellking4
Copy link

shellking4 commented Mar 2, 2021

thank you very much @loilo for the quick and clear answer. I followed the steps and did as you said. ran it since a while but no json output yet. I really don't know if it's working or not.

@loilo
Copy link
Author

loilo commented Mar 2, 2021

Mh, I cannot say from the distance what your problem might be, but I have created a Gist for you which includes the whole workflow for exporting a database. Copy the script as a whole and paste it into your browser console β€” that should clearly state whether it worked or not:

https://gist.github.com/loilo/ddfdb3c54fa474a89f71ce0660cd38b7

@shellking4
Copy link

shellking4 commented Mar 2, 2021

Mh, I cannot say from the distance what your problem might be, but I have created a Gist for you which includes the whole workflow for exporting a database. Copy the script as a whole and paste it into your browser console β€” that should clearly state whether it worked or not:

https://gist.github.com/loilo/ddfdb3c54fa474a89f71ce0660cd38b7

Thank you very much for helping me !
I pasted the gist in the console and hit enter then the console output '' Promise {} ''
After it says '' Uncaught RangeError: Invalid string length at JSON.stringify ().

I have up to 63 stores in the database. Would it be the cause ? Too much data ?

@loilo
Copy link
Author

loilo commented Mar 2, 2021

It seems like it, at least the error message is usually linked to that problem.

For these amounts of data you should probably back up your whole browser's user data. For Google Chrome, those are located at %userdata%\AppData\Local\Google\Chrome\User Data (Windows) respectively ~/Library/Application Support/Google/Chrome (macOS). Other browsers use very similar paths.

You should be able to back up that whole folder and restore it in the same place on a new machine. I have definitely done this successfully before, but, again, you better verifiy it to work with another PC or in a VM. (If I remember correctly, there was also the restriction that the new device needed to have the same user/password credentials to be able to read the backed up data on Windows β€” but I'm not sure on that. That might only affect stored passwords.)

@shellking4
Copy link

It seems like it, at least the error message is usually linked with that problem.

For these amounts of data you should probably back up your whole browser's user data. For Google Chrome, those are located at %userdata%\AppData\Local\Google\Chrome\User Data (Windows) respectively ~/Library/Application Support/Google/Chrome (macOS). Other browsers use very similar paths.

You should be able to back up that whole folder and restore it in the same place on a new machine. I have definitely done this successfully before, but, again, you better verifiy it to work with another PC or in a VM. (If I remember correctly, there was also the restriction that the new device needed to have the same user/password credentials to be able to read the backed up data on Windows β€” but I'm not sure on that.)

Thank you soooooooooooooo much, you're a life saver
Thanks a lot πŸ™‚πŸ˜Š.

@shellking4
Copy link

It seems like it, at least the error message is usually linked with that problem.

For these amounts of data you should probably back up your whole browser's user data. For Google Chrome, those are located at %userdata%\AppData\Local\Google\Chrome\User Data (Windows) respectively ~/Library/Application Support/Google/Chrome (macOS). Other browsers use very similar paths.

You should be able to back up that whole folder and restore it in the same place on a new machine. I have definitely done this successfully before, but, again, you better verifiy it to work with another PC or in a VM. (If I remember correctly, there was also the restriction that the new device needed to have the same user/password credentials to be able to read the backed up data on Windows β€” but I'm not sure on that.)

Thank you soooooooooooooo much, you're a life saver
Thanks a lot πŸ™‚πŸ˜Š.

@loilo
Copy link
Author

loilo commented Mar 2, 2021

Glad I could help. πŸ™‚ Have a nice day! πŸ‘‹

@shellking4
Copy link

Glad I could help. πŸ™‚ Have a nice day! πŸ‘‹

Thank you ! Have a nice day too

@dan-osull
Copy link

Mh, I cannot say from the distance what your problem might be, but I have created a Gist for you which includes the whole workflow for exporting a database. Copy the script as a whole and paste it into your browser console β€” that should clearly state whether it worked or not:

https://gist.github.com/loilo/ddfdb3c54fa474a89f71ce0660cd38b7

Thanks very much for this script, it was just what I was looking for!

@sheunaluko
Copy link

sheunaluko commented May 15, 2021

Thanks so much @loilo! In case anyone is wondering, the callback function that handles the cursor can also extract the key corresponding to each object in the store, if you want that information (as I did). The modification in exportToJson simply becomes:

allObjects.push([cursor.key, cursor.value])

and the corresponding modification in importFromJson within the loop:

 let [k,value] = toAdd ; 
 const request = transaction.objectStore(storeName).add(value,k)

Hope that helps someone and thanks again! Web tech is so dope , more info on cursors here

@liudonghua123
Copy link

Okay, that should be relatively straightforward, under one condition: You'll still have the IndexedDB on the new machine, even though it may be empty. This is necessary because the export script from above can only export the DB's data, not its strucure.

For both exporting and importing data, you'll need an IndexedDB instance. Say you're wanting to back up the foo database, then you could get an instance in the console like this:

var db
var request = window.indexedDB.open('foo')
request.onerror = () => console.error('Could not open database')
request.onsuccess = () => { db = request.result }

Export:

  1. Copy the exportToJson function (without the prepended export statement), paste it into the DevTools Console and hit Enter.
  2. Run this: await exportToJson(db)
  3. If you see some JSON output in the console it means that the export worked. Run copy($_) to copy the return value of the previous command (= the exported data) to your clipboard.
  4. Save that exported data somewhere.

Import:

  1. Copy the importFromJson function (without the prepended export statement), paste it into the DevTools Console and hit Enter.
  2. Invoke it with the IndexedDB instance and the exported data as parameters: await importFromJson(db, exportedString).
    Note that you actually have to pass the exported data as a string, not as a JSON object. For example: If you exported and empty database {}, you'd have to pass into the function the following: await importFromJson(db, '{}').

This should be it. Probably test it before you actually rely on it. πŸ™‚

Hi, Thanks for the script, But when I tried to dump the two emscripten_filesystem and gscyclone database on https://play.wo1wan.com/jjnext/play?id=540&mode=0&gstate=def, the script just copied {"FILES":[]} and then the page redirected to about:blank.

I also tried the script on https://gist.github.com/loilo/ddfdb3c54fa474a89f71ce0660cd38b7, however, it's the same problems.

I could not found any error in the console.

@moniuch
Copy link

moniuch commented Aug 31, 2023

Am I correct that the importFromJson function requires that the target db has all the necessary stores created beforehand?

@loilo
Copy link
Author

loilo commented Aug 31, 2023

@moniuch Yes, if I remember correctly, this is the case.

@xmedeko
Copy link

xmedeko commented Aug 31, 2023

@moniuch It simple to modify the code to create stores by JSON before the import.

@bertkdowns
Copy link

bertkdowns commented Oct 13, 2023

I may have found a bug - I had a few object stores that were empty. When importing, because there were no records, delete importObject[storeName] was never run. I fixed it by adding a check before the for loop:

      const importObject = JSON.parse(json)
      for (const storeName of idbDatabase.objectStoreNames) {
        // ignore empty stores (we have to do this explicitly or the keys will never be deleted, and the promise will not resolve)
        if(importObject[storeName].length == 0){
            delete importObject[storeName];
            continue;
        }
        let count = 0
        for (const toAdd of importObject[storeName]) {
....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment