| (py310) ishan@gpu-workstation ~/code/lucene-raft/cuda (ishan/integrating-raft-and-jni) $ mvn package | |
| [INFO] Scanning for projects... | |
| [INFO] | |
| [INFO] ------< com.searchscale.lucene.vectorsearch:cagra-searcher-cuda >------- | |
| [INFO] Building cagra-searcher-cuda 0.1 | |
| [INFO] from pom.xml | |
| [INFO] --------------------------------[ jar ]--------------------------------- | |
| [INFO] | |
| [INFO] --- cmake:3.27.7-b1:generate (cmake-generate) @ cagra-searcher-cuda --- | |
| [INFO] Downloading binaries to /home/ishan/code/lucene-raft/cuda/target/dependency/cmake |
| #include <cstdint> | |
| #include <raft/core/device_mdarray.hpp> | |
| #include <raft/core/device_resources.hpp> | |
| #include <raft/neighbors/cagra.cuh> | |
| #include <raft/random/make_blobs.cuh> | |
| #include <iostream> | |
| #include <fstream> | |
| #include <string> | |
| #include <rmm/mr/device/device_memory_resource.hpp> |
| { | |
| "responseHeader":{ | |
| "zkConnected":true, | |
| "status":0, | |
| "QTime":5, | |
| "params":{ | |
| "q":"*:*", | |
| "facet.field":"unknownField", | |
| "indent":"true", | |
| "q.op":"OR", |
Employ the use of a slow hard disk (local or network attached) for persistent storage and a fast SSD (preferably local) for caching. This combination can be used as data directory for Apache Solr.
In this example, we have a hard disk at /dev/sdb and the SSD at /dev/nvme0n1. The entire hard disk (/dev/sdb) can be used for persistent storage, but in this example we are using just a single partition (/dev/sdb1) as the persistent store.
| root@asrock-new:/home/ishan# hdparm -tT /dev/nvme0n1p2 | |
| /dev/nvme0n1p2: | |
| Timing cached reads: 58452 MB in 1.99 seconds = 29301.60 MB/sec | |
| Timing buffered disk reads: 4180 MB in 3.00 seconds = 1392.81 MB/sec | |
| root@asrock-new:/home/ishan# hdparm -tT /dev/sda2 | |
| /dev/sda2: | |
| Timing cached reads: 56250 MB in 2.00 seconds = 28195.14 MB/sec | |
| Timing buffered disk reads: 346 MB in 3.02 seconds = 114.75 MB/sec |
| #include <errno.h> | |
| #include <fcntl.h> | |
| #include <unistd.h> | |
| #include <stdio.h> | |
| #include <stdlib.h> | |
| #include <string.h> | |
| #include <sys/types.h> | |
| #include <sys/mman.h> | |
| #include <sys/stat.h> | |
| #include <time.h> |
| [junit4] Suite: org.apache.solr.ltr.TestLTROnSolrCloud | |
| [junit4] 1> 07:14:54.420 [jetty-launcher-32-thread-1] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete. | |
| [junit4] 1> 07:14:59.071 [qtp1845186159-135] ERROR org.apache.solr.handler.RequestHandlerBase - org.apache.solr.common.SolrException: Unable to reload core [collection1_shard3_replica_n4] | |
| [junit4] 1> at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1757) | |
| [junit4] 1> at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1676) | |
| [junit4] 1> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$2(CoreAdminOperation.java:134) | |
| [junit4] 1> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:367) | |
| [junit4] 1> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397) | |
| [junit4] 1> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody( |
| [ishan@computadora code] $ tar -xf solr-8.7.0.tgz | |
| [ishan@computadora code] $ cd solr-8.7.0/ | |
| [ishan@computadora solr-8.7.0] $ bin/solr -c | |
| *** [WARN] *** Your open file limit is currently 1024. | |
| It should be set to 65000 to avoid operational disruption. | |
| If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh | |
| Waiting up to 180 seconds to see Solr running on port 8983 [\] | |
| Started Solr server on port 8983 (pid=723496). Happy searching! | |
| [ishan@computadora solr-8.7.0] $ curl http://localhost:8983/api/node/health |
| # This program is equivalent of this shell command, but using multple threads | |
| # for i in {14566365..24090104}; do wget -q https://hacker-news.firebaseio.com/v0/item/$i.json; done | |
| import threading | |
| import urllib2 | |
| import time | |
| def fetch_url(url): | |
| urlHandler = urllib2.urlopen("https://hacker-news.firebaseio.com/v0/item/" + str(url) + ".json") | |
| html = urlHandler.read() | |
| with open(str(url) + ".json", "w") as text_file: |
| ishan@instance-4:~/solr-bench$ unzip -l logs-2020-16-06_14.35.26.zip | |
| Archive: logs-2020-16-06_14.35.26.zip | |
| Length Date Time Name | |
| --------- ---------- ----- ---- | |
| 133120 2020-06-16 14:35 solrlogs-solr-1.tar | |
| 40960 2020-06-16 14:35 solrlogs-solr-2.tar | |
| --------- ------- | |
| 174080 2 files |