Read LMDB in python

create_lmdb.sh

CAFFE_ROOT=/home/wyang/Data/Code/caffe-wk

$CAFFE_ROOT/build/tools/convert_imageset -backend lmdb /home/wyang/code/pose/chen-nips14-pose/ LMDB_val_patches.txt tmp_LMDB

lmdb_reader.py

import caffe
import lmdb
import numpy as np
import matplotlib.pyplot as plt
from caffe.proto import caffe_pb2
# Wei Yang 2015-08-19
# Source
#   Read LevelDB/LMDB
#   ==================
#       http://research.beenfrog.com/code/2015/03/28/read-leveldb-lmdb-for-caffe-with-python.html
#   Plot image
#   ==================
#       http://www.pyimagesearch.com/2014/11/03/display-matplotlib-rgb-image/
#   Creating LMDB in python
#   ==================
#       http://deepdish.io/2015/04/28/creating-lmdb-in-python/

lmdb_file = "/home/wyang/code/pose/chen-nips14-pose/cache/lsp/LMDB_val"
lmdb_env = lmdb.open(lmdb_file)
lmdb_txn = lmdb_env.begin()
lmdb_cursor = lmdb_txn.cursor()
datum = caffe_pb2.Datum()

for key, value in lmdb_cursor:
    datum.ParseFromString(value)

    label = datum.label
    data = caffe.io.datum_to_array(datum)
    im = data.astype(np.uint8)
    im = np.transpose(im, (2, 1, 0)) # original (dim, col, row)
    print "label ", label

    plt.imshow(im)
    plt.show()

LMDB_val_patches.txt

./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000001.jpg 1
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000002.jpg 1823
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000003.jpg 2903
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000004.jpg 4470
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000005.jpg 4698
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000006.jpg 4797
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000007.jpg 4917
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000008.jpg 5072
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000009.jpg 5177
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000010.jpg 5271
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000011.jpg 6617
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000012.jpg 7592
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000013.jpg 7754
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000014.jpg 7892
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000015.jpg 7984
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000016.jpg 8109
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000017.jpg 8190
./cache/flic_win/LMDB/LMDB_val_patches/casino-royale-00039871_00000001_000018.jpg 8341
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000001.jpg 1
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000002.jpg 900
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000003.jpg 2585
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000004.jpg 4529
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000005.jpg 4713
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000006.jpg 4819
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000007.jpg 4921
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000008.jpg 5095
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000009.jpg 5160
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000010.jpg 5278
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000011.jpg 5916
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000012.jpg 7624
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000013.jpg 7654
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000014.jpg 7833
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000015.jpg 7985
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000016.jpg 8006
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000017.jpg 8212
./cache/flic_win/LMDB/LMDB_val_patches/million-dollar-baby-disc-00106821_00000002_000018.jpg 8346

I keep getting lmdb.Error: ****t.lmdb: Not a directory should we provide folder path or file path?

I keep getting lmdb.Error: ****t.lmdb: Not a directory should we provide folder path or file path?

You should specify the directory as specified in the documentation for lmdb.open(path):
path:
Location of directory (if subdir=True) or file prefix to store the database.

https://lmdb.readthedocs.io/en/release/#lmdb.lmdb.open
https://lmdb.readthedocs.io/en/release/#lmdb.Environment

I have tried by providing the folder path that contains a single file (filename_wallet.lmdb) but still I cannot read any line out of it using your approach, seems like it is not possible.

I have tried by providing the folder path that contains a single file (filename_wallet.lmdb) but still I cannot read any line out of it using your approach, seems like it is not possible.

Try setting the file path lmdb.open(filename_wallet.lmdb, subdir=False)
Maybe the file is broken?

Thanks seems like I was able to get some key and values but returned value is single character:

for key, value in lmdb_cursor:
    print(key)
    print(value)

# output:
b'Header'
b'\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\x00\x00\x00\x00'

---

Do you think using with caffe might get additional data (because original file size is much larger)?