-
-
Save mdouze/46d6bbbaabca0b9778fca37ed2bcccf6 to your computer and use it in GitHub Desktop.
Your dataset requires 1 TiB ram! Try to reduce the dimension as well as the nlist
, or you will never get this index trained.
My dataset isnt getting trained on cpu rather that gpu inspite of using
index_ivf = faiss.extract_index_ivf(index2) clustering_index = faiss.index_cpu_to_all_gpus(faiss.IndexFlatL2(64)) index_ivf.clustering_index = clustering_index
Can any one help me
Can we use the GPU version of the Binary Flat index as the clustering index for the binary indexes? Like below:
faiss.index_cpu_to_all_gpus(faiss.IndexBinaryFlat(d))
My dataset isnt getting trained on cpu rather that gpu inspite of using
index_ivf = faiss.extract_index_ivf(index2) clustering_index = faiss.index_cpu_to_all_gpus(faiss.IndexFlatL2(64)) index_ivf.clustering_index = clustering_index
Can any one help me
Do you have faiss-gpu installed? What do you get for faiss.get_num_gpu()?
Hi @mdouze , is it correct that something like:
"PCAR1024,IVF262144_HNSW32,PQ512x8" (with inner product)
cannot be trained on the GPU?
https://gist.github.com/mdouze/46d6bbbaabca0b9778fca37ed2bcccf6?permalink_comment_id=3111847#gistcomment-3111847
You mention that PQ quantisation is not implemented?
@mdouze
Getting memory error when adding 140M data points, which is the size of my dataset.
Any tips to overcome this? Even if I add 50% of the data 70M points, I get memory error