Note that all steps are being perform on Ubuntu linux with the following packages installed:
binutils-x86-64-linux-gnufor thestringscommand line tool.- Note that llvm also provides a
stringstool, for example thellvm-14package providesllvm-strings-14which is equivalent for our purposes.
- Note that llvm also provides a
coreutilsfor theodcommand line tool.entfor theentcommand line tool whose homepage is https://www.fourmilab.ch/random/ (but whose debian package has diverged).uuidfor theuuidcommand line tool.
I am using the --profile PROFILE-PATH command line argument combined with
-no-remote for testing purposes.
Because I am testing in a build where dom.cache.privateBrowsing.enabled defaults
to false, I first manually configure the profile to set the preference to true,
then restart.
Steps:
- Open a PB window.
- Open the URL https://firefox-storage-test.glitch.me/ and ensure that it
reports that:
- LocalStorage, QuotaManager, IndexedDB, and Cache API are "Good: Totally Working."
- Validate that PROFILE/storage/private is created and change into that
directory. Future steps will refer to this directory as
PBSTORAGE. - Validate that there is a directory that starts with
uuid+++and is followed by a UUID. For example,uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03. All characters in the UUID will match the regex character range[-0-9a-z].- It's important to ensure that the UUID is a version 4 random UUID that is variant 1. We
can confirm this by matching against the pattern
xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxxand ensuring that M is4and N has a bit pattern of10xxwhich means it matches the regex[89ab]. We can also use command line tools to parse likeuuid -d THEUUIDwill sayversion: 4 (random data based). Other tools likeuuidparsealso exist.
- It's important to ensure that the UUID is a version 4 random UUID that is variant 1. We
can confirm this by matching against the pattern
- Run
/usr/bin/find . -printf "%p %sB\n"in PBSTORAGE order to dump the entirety of the tree, including hidden dot files. For my run, I got:
. 4096B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03 4096B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/cache 4096B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/cache/morgue 4096B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/cache/morgue/172 4096B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/cache/morgue/172/{8511b9e6-c48f-4a23-a561-4639c1fb8bac}.final 12288B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/cache/morgue/36 4096B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/cache/morgue/36/{28b9798c-4714-4150-8b75-8edbfdbfa424}.final 4096B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/cache/caches.sqlite 131072B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/cache/caches.sqlite-wal 254728B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/cache/context_open.marker 0B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/cache/.padding 8B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/idb 4096B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/idb/742c9a20-94f1-42ec-a196-ba9dd8c03f96.files 4096B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/idb/742c9a20-94f1-42ec-a196-ba9dd8c03f96.sqlite-wal 0B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/idb/742c9a20-94f1-42ec-a196-ba9dd8c03f96.sqlite 90112B
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/.metadata-v2 73B
- We expect to find in the sole directory:
- A
.metadata-v2file which may be removed in the future. - The
idbIndexedDB database directory.- A database of the form
UUID.sqliteas well as a pairedUUID.filesdirectory using the same UUID which holds Blobs stored in the database or very large structured clone values. - We now always expect to find
.sqlite-journalor.sqlite-walor.sqlite-shmsuffixed files because we have changed Firefox to retain the files for performance reasons. This means they will have a non-zero size when the database is actively open, but once it has been closed, the files should be truncated to zero length. For https://firefox-storage-test.glitch.me/ we expect the database to be closed in a timely fashion, so you may want to re-run the find command to see if those files have gone away.
- A database of the form
- The
cachesCache API directory.- The
morguedirectory, a poorly named directory that contains the Response bodies using an additional tier of sub-directories to shard the UUID-based file names. We expect a total of 2 Response body files with filenames ending in.final. One should be 4 KiB (4096 B) and one should be 12 KiB (12288 B). - The
caches.sqliteand its Write-Ahead-Log counterpart,caches.sqlite-wal. When the cache is open, we expect the.sqlite-walfile to have a non-zero size (254,728 B i this case), but to shrink to 0 B when the cache is closed. In this case the.sqlitefile was 128 KiB (131,072 B) both when open and after checkpointing. context_open.markera marker file that's present when the cache is open, with a 0 length..paddingis always present and always 8 B in size. It conveys the amount of padding for no-cors responses without us having to open the database.
- The
- A
- Verify that the
.metadata-v2file contains the UUID and not the original origin name. Note that I will be using relative paths from the root of PBSTORAGE in my example, but it's fine to change into the directory you are investigating and just use the file names.- Run the
stringscommand to dump the runs of printable characters at least 4 characters long on the metadata file. Verify that the string corresponds to the UUID. For example, my run looks like:
- Run the
$ strings ./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/.metadata-v2
+uuid://4ae1fe8a-9f6e-428d-9eee-4a57707a0a03
- Run the
od -t x1zcommand to dump the entire contents of the file and verify there are no other notable pieces of information that look concerning. The data should largely look like the following where the first 8 bytes are a timestamp (which will change), followed by a bunch of zeroes. The sanitized origin that starts withuuid://will also be prefixed by a length byte which will render as+because all the UUIDs have the same length. My run looked like:
$ /usr/bin/od -t x1z ./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/.metadata-v2
0000000 00 06 09 9c 14 24 ec 26 00 00 00 00 00 00 00 00 >.....$.&........<
0000020 00 00 00 00 00 00 00 00 00 00 00 00 2b 75 75 69 >............+uui<
0000040 64 3a 2f 2f 34 61 65 31 66 65 38 61 2d 39 66 36 >d://4ae1fe8a-9f6<
0000060 65 2d 34 32 38 64 2d 39 65 65 65 2d 34 61 35 37 >e-428d-9eee-4a57<
0000100 37 30 37 61 30 61 30 33 01 >707a0a03.<
0000111
- Verify that each SQLite database appears encrypted by using the
entcommand on the database to verify that there is a high about of entropy per bit. Note that a hard-coded SQLite header is used in the clear at the very beginning of the file, so using thefilecommand will correctly identify the that the file is a SQLite database. Because the ciphertext should look like a high-quality stream of (pseudo)random data, we expect to see an entropy close to 8 bits per byte. This is what myentinvocations look like when using find to run ent on all the relevant non-zero-length files at once using/usr/bin/find . -name \*.sqlite\* -size +0 -printf "%p %sB\n" -exec ent '{}' \; -exec echo -e "\n---\n" \;
Entropy = 7.511740 bits per byte.
Optimum compression would reduce the size
of this 131072 byte file by 6 percent.
Chi square distribution for 131072 samples is 534588.14, and randomly
would exceed this value less than 0.01 percent of the times.
Arithmetic mean value of data bytes is 111.7017 (127.5 = random).
Monte Carlo value for Pi is 3.232593271 (error 2.90 percent).
Serial correlation coefficient is 0.271889 (totally uncorrelated = 0.0).
---
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/cache/caches.sqlite-wal 254728B
Entropy = 7.998382 bits per byte.
Optimum compression would reduce the size
of this 254728 byte file by 0 percent.
Chi square distribution for 254728 samples is 614.91, and randomly
would exceed this value less than 0.01 percent of the times.
Arithmetic mean value of data bytes is 127.3905 (127.5 = random).
Monte Carlo value for Pi is 3.133273661 (error 0.26 percent).
Serial correlation coefficient is 0.006572 (totally uncorrelated = 0.0).
---
./uuid+++4ae1fe8a-9f6e-428d-9eee-4a57707a0a03/idb/742c9a20-94f1-42ec-a196-ba9dd8c03f96.sqlite 90112B
Entropy = 7.996701 bits per byte.
Optimum compression would reduce the size
of this 90112 byte file by 0 percent.
Chi square distribution for 90112 samples is 418.14, and randomly
would exceed this value less than 0.01 percent of the times.
Arithmetic mean value of data bytes is 127.4524 (127.5 = random).
Monte Carlo value for Pi is 3.170861633 (error 0.93 percent).
Serial correlation coefficient is 0.001525 (totally uncorrelated = 0.0).
---
- A potential sanity check at this time while we currently do not encrypt IDB
databases for non-private browsing mode is to also run
entagainst the same database for non-private-browisng. This is an example invocation I ran from the directoryPBPROFILE/storage/default/https+++firefox-storage-test.glitch.menoting that we also have LocalStorage present in this case, whereas for private browsing it is only retained in memory and not currently written to disk.
$ /usr/bin/find . -name \*.sqlite\* -size +0 -printf "%p %sB\n" -exec ent '{}' \; -exec echo -e "\n---\n" \;
./cache/caches.sqlite 98304B
Entropy = 1.906636 bits per byte.
Optimum compression would reduce the size
of this 98304 byte file by 76 percent.
Chi square distribution for 98304 samples is 16278142.56, and randomly
would exceed this value less than 0.01 percent of the times.
Arithmetic mean value of data bytes is 16.1374 (127.5 = random).
Monte Carlo value for Pi is 3.998291016 (error 27.27 percent).
Serial correlation coefficient is 0.893347 (totally uncorrelated = 0.0).
---
./idb/1984578081ptenrestis.sqlite 49152B
Entropy = 0.757212 bits per byte.
Optimum compression would reduce the size
of this 49152 byte file by 90 percent.
Chi square distribution for 49152 samples is 10818176.36, and randomly
would exceed this value less than 0.01 percent of the times.
Arithmetic mean value of data bytes is 5.8571 (127.5 = random).
Monte Carlo value for Pi is 3.999511719 (error 27.31 percent).
Serial correlation coefficient is 0.910430 (totally uncorrelated = 0.0).
---
./ls/data.sqlite 6144B
Entropy = 1.405298 bits per byte.
Optimum compression would reduce the size
of this 6144 byte file by 82 percent.
Chi square distribution for 6144 samples is 1157584.75, and randomly
would exceed this value less than 0.01 percent of the times.
Arithmetic mean value of data bytes is 10.9212 (127.5 = random).
Monte Carlo value for Pi is 4.000000000 (error 27.32 percent).
Serial correlation coefficient is 0.825137 (totally uncorrelated = 0.0).
---
- An additional step we can take is to run the
stringscommand against the encrypted database as just an additional level of sanity checking. As noted above, we do expect to find the stringSQLite format 3in the clear because it is part of the hardcoded header that is stored in the clear. In order to reduce false positives it is possible to pass an-nargument likestrings -n8to only show sequences of 8-or-more printable characters which will produce fewer lines of output. Because the random output should be uniformly distributed over all values in the range 0-255, we do expect to see multiple runs of (gibberish) printable characters. Our revised "find" invocation is:/usr/bin/find . -name \*.sqlite\* -size +0 -printf "%p %sB\n" -exec strings -n8 '{}' \; -exec echo -e "\n---\n" \;and produces the expected result of "SQLite format 3" followed by a random set of random printable characters.
At this point we have verified that:
- The file/directory names do not include the (real) origin name in the clear.
- That the metadata file likewise does not contain the (real) origin name in the clear.
- That the file is encrypted although it includes the expected SQLite header.
Our Cache API implementation stores the Response bodies as individual files under the "morgue" directory.
Ent invocation:
/usr/bin/find . -name \*.final -size +0 -printf "%p %sB\n" -exec ent '{}' \; -exec echo -e "\n---\n" \;