Skip to content

Instantly share code, notes, and snippets.

@Inkimar

Inkimar/gist:1e186a98726d212846161dee5bcafd80

Created November 9, 2022 14:09
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Inkimar/1e186a98726d212846161dee5bcafd80 to your computer and use it in GitHub Desktop.
Save Inkimar/1e186a98726d212846161dee5bcafd80 to your computer and use it in GitHub Desktop.
Download ZIP
2021-11-9, SNIC - biocache -> dr632
Raw
gistfile1.txt
1. load has been done
2. biocache> process -dr dr632
@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

Error after process

98411f57f9ba 2022-11-09 13:14:05,988 INFO : [RecordProcessor] - RecordProcessor is instantiated. processTime:2022-11-09T13:14:05Z
98411f57f9ba 2022-11-09 13:14:06,306 INFO : [AttributionDAO] - Calling web service for dr632
98411f57f9ba 2022-11-09 13:14:06,306 INFO : [AttributionDAO] - Calling web service for dr632
98411f57f9ba 2022-11-09 13:14:06,320 INFO : [AttributionDAO] - Calling web service for dr632
98411f57f9ba 2022-11-09 13:14:06,320 INFO : [AttributionDAO] - Calling web service for dr632
98411f57f9ba 2022-11-09 13:14:06,903 WARN : [WebServiceLoader] - Unable to load https://collections.biodiversitydata.se/ws/dataResource/dr632.json : Server returned HTTP response code: 500 for URL: https://collections.biodiversitydata.se/ws/dataResource/dr632.json
98411f57f9ba 2022-11-09 13:14:06,923 WARN : [WebServiceLoader] - Unable to load https://collections.biodiversitydata.se/ws/dataResource/dr632.json : Server returned HTTP response code: 500 for URL: https://collections.biodiversitydata.se/ws/dataResource/dr632.json
98411f57f9ba 2022-11-09 13:14:06,926 WARN : [WebServiceLoader] - Unable to load https://collections.biodiversitydata.se/ws/dataResource/dr632.json : Server returned HTTP response code: 500 for URL: https://collections.biodiversitydata.se/ws/dataResource/dr632.json
98411f57f9ba 2022-11-09 13:14:06,955 INFO : [Vocab] - Reading

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

Action ->

  1. run artillery quick -count 10 - n 50 https://collections.biodiversitydata.se/ws/datasource/632.json from another terminal
  2. then run process again - biocache> process -dr dr632
  3. NO ERROR
biocache> process -dr dr632
98411f57f9ba 2022-11-09 14:11:54,379 INFO : [ProcessRecords] - Starting processing...
There is an incremental row key file for this resource.  Would you like to perform an incremental process (y/n) ?
y
98411f57f9ba 2022-11-09 14:11:56,877 INFO : [RecordProcessor] - RecordProcessor is instantiated. processTime:2022-11-09T14:11:56Z
98411f57f9ba 2022-11-09 14:11:57,214 INFO : [AttributionDAO] - Calling web service for dr632
98411f57f9ba 2022-11-09 14:11:57,214 INFO : [AttributionDAO] - Calling web service for dr632
98411f57f9ba 2022-11-09 14:11:57,214 INFO : [AttributionDAO] - Calling web service for dr632
98411f57f9ba 2022-11-09 14:11:57,214 INFO : [AttributionDAO] - Calling web service for dr632
98411f57f9ba 2022-11-09 14:11:57,892 INFO : [Vocab] - Reading vocab file: /data/biocache/vocab/dwc.txt
98411f57f9ba 2022-11-09 14:11:58,102 INFO : [AttributionDAO] - Looking up collectory web service for KTH|KTH-2013-BALTIC-16S
98411f57f9ba 2022-11-09 14:11:58,103 INFO : [AttributionDAO] - Looking up collectory web service for KTH|KTH-2013-BALTIC-16S
98411f57f9ba 2022-11-09 14:11:58,105 INFO : [AttributionDAO] - Looking up collectory web service for KTH|KTH-2013-BALTIC-16S
98411f57f9ba 2022-11-09 14:11:58,107 INFO : [AttributionDAO] - Looking up collectory web service for KTH|KTH-2013-BALTIC-16S
98411f57f9ba 2022-11-09 14:11:58,178 INFO : [Vocab] - Reading vocab file: /data/biocache/vocab/kingdoms.txt
98411f57f9ba 2022-11-09 14:11:58,396 INFO : [Cassandra3PersistenceManager] - #######  Using full replication, so splitting token ranges based on node number 0
98411f57f9ba 2022-11-09 14:11:58,396 INFO : [Cassandra3PersistenceManager] - #######  Total number of token ranges for cluster 1025
98411f57f9ba 2022-11-09 14:11:58,408 INFO : [Cassandra3PersistenceManager] - Starting threads...number of callables 1025
98411f57f9ba 2022-11-09 14:12:00,674 INFO : [Cassandra3PersistenceManager] - All threads have completed paging
98411f57f9ba 2022-11-09 14:12:00,678 INFO : [Cassandra3PersistenceManager] - #######  Using full replication, so splitting token ranges based on node number 0
98411f57f9ba 2022-11-09 14:12:00,678 INFO : [Cassandra3PersistenceManager] - #######  Total number of token ranges for cluster 1025
98411f57f9ba 2022-11-09 14:12:00,680 INFO : [Cassandra3PersistenceManager] - Starting threads...number of callables 1025
98411f57f9ba 2022-11-09 14:12:02,890 INFO : [Cassandra3PersistenceManager] - All threads have completed paging
98411f57f9ba 2022-11-09 14:12:02,892 INFO : [Cassandra3PersistenceManager] - #######  Using full replication, so splitting token ranges based on node number 0
98411f57f9ba 2022-11-09 14:12:02,892 INFO : [Cassandra3PersistenceManager] - #######  Total number of token ranges for cluster 1025
98411f57f9ba 2022-11-09 14:12:02,898 INFO : [Cassandra3PersistenceManager] - Starting threads...number of callables 1025
98411f57f9ba 2022-11-09 14:12:04,790 INFO : [Cassandra3PersistenceManager] - All threads have completed paging
98411f57f9ba 2022-11-09 14:12:04,792 INFO : [Cassandra3PersistenceManager] - #######  Using full replication, so splitting token ranges based on node number 0
98411f57f9ba 2022-11-09 14:12:04,793 INFO : [Cassandra3PersistenceManager] - #######  Total number of token ranges for cluster 1025
98411f57f9ba 2022-11-09 14:12:04,795 INFO : [Cassandra3PersistenceManager] - Starting threads...number of callables 1025
98411f57f9ba 2022-11-09 14:12:06,757 INFO : [Cassandra3PersistenceManager] - All threads have completed paging
98411f57f9ba 2022-11-09 14:12:06,771 INFO : [Vocab] - Reading vocab file: /data/biocache/vocab/basisOfRecord.txt
98411f57f9ba 2022-11-09 14:12:06,829 INFO : [Vocab] - Reading vocab file: /data/biocache/vocab/datums.txt
98411f57f9ba 2022-11-09 14:12:07,436 INFO : [ENGINE] - dataFileCache open start
98411f57f9ba 2022-11-09 14:12:07,517 INFO : [ENGINE] - Checkpoint start
98411f57f9ba 2022-11-09 14:12:07,518 INFO : [ENGINE] - Checkpoint end
98411f57f9ba 2022-11-09 14:12:07,776 INFO : [SpatialLayerDAO] - Loading Layer information from .....https://spatial.biodiversitydata.se/ws
98411f57f9ba 2022-11-09 14:12:09,372 INFO : [SpatialLayerDAO] - Number of layers loaded .....75
98411f57f9ba 2022-11-09 14:12:09,658 INFO : [SpatialLayerDAO] - Number of fields loaded .....76
98411f57f9ba 2022-11-09 14:12:09,660 INFO : [SpatialLayerDAO] - SDS enabled .....true
98411f57f9ba 2022-11-09 14:12:09,660 INFO : [SpatialLayerDAO] - Loaded layers required by SDS.....
98411f57f9ba 2022-11-09 14:12:09,660 INFO : [SpatialLayerDAO] - Retrieving list from https://sds.biodiversitydata.se/ws/layers
98411f57f9ba 2022-11-09 14:12:09,707 INFO : [SpatialLayerDAO] - Loading SDS layer.....cl10038
98411f57f9ba 2022-11-09 14:12:09,713 INFO : [SpatialLayerDAO] - Geospatial ID...cl10038 - available: true - /data/biocache/layers/sweden-boundary.shp, field name: NAME_0
98411f57f9ba 2022-11-09 14:12:09,729 INFO : [DBFRecords] - reading shapefile: /data/biocache/layers/sweden-boundary.dbf
98411f57f9ba 2022-11-09 14:12:09,809 INFO : [SpatialLayerDAO] - Geospatial ID...cl10097 - available: false - /data/biocache/layers/lan.shp, field name: LnNamn
98411f57f9ba 2022-11-09 14:12:09,813 INFO : [SpatialLayerDAO] - Geospatial ID...cl10087 - available: false - /data/biocache/layers/world-countries.shp, field name: NAME_ENGLI
98411f57f9ba 2022-11-09 14:12:09,874 INFO : [Vocab] - Reading vocab file: /data/biocache/vocab/stateProvinces.txt
98411f57f9ba 2022-11-09 14:12:09,897 INFO : [Vocab] - Reading vocab file: /data/biocache/vocab/countries.txt
98411f57f9ba 2022-11-09 14:12:15,415 INFO : [ProcessRecords] - 1000 >> Last key : 4840f76f-f82c-4f65-b276-ae432b79eaec, records per sec: 53.943253
98411f57f9ba 2022-11-09 14:12:17,876 INFO : [ProcessRecords] - 2000 >> Last key : 50cb9031-f2ab-4d49-9eeb-334ce2943953, records per sec: 407.49796
98411f57f9ba 2022-11-09 14:12:20,179 INFO : [ProcessRecords] - 3000 >> Last key : 8cc4c95a-db5d-4428-93d0-f1a4e85c71d9, records per sec: 434.21625
98411f57f9ba 2022-11-09 14:12:22,204 INFO : [ProcessRecords] - 4000 >> Last key : dcc58958-561b-404e-99a0-f4ec286abbdc, records per sec: 493.82715
98411f57f9ba 2022-11-09 14:12:23,947 INFO : [ProcessRecords] - 5000 >> Last key : 07214217-36cf-4da8-81c0-0be5ab070a1f, records per sec: 574.3825
98411f57f9ba 2022-11-09 14:12:25,934 INFO : [ProcessRecords] - 6000 >> Last key : 522f08a8-9ce9-47e7-b589-5c2c710dfa81, records per sec: 503.27127
98411f57f9ba 2022-11-09 14:12:27,735 INFO : [ProcessRecords] - 7000 >> Last key : 524e2ff9-b0f3-423a-b5d9-13b1fe7ba6fa, records per sec: 555.55554
98411f57f9ba 2022-11-09 14:12:29,203 INFO : [ProcessRecords] - 8000 >> Last key : 613d9f3c-6f84-4e3a-a3ca-5ec8d4c935a3, records per sec: 681.66327
98411f57f9ba 2022-11-09 14:12:30,567 INFO : [ProcessRecords] - 9000 >> Last key : a11d46be-0227-4711-b4f7-75a8f6462474, records per sec: 733.6757
98411f57f9ba 2022-11-09 14:12:31,978 INFO : [ProcessRecords] - 10000 >> Last key : cced1cdc-77ff-4c33-bf4b-1ef18d6b5f41, records per sec: 708.7172
98411f57f9ba 2022-11-09 14:12:33,389 INFO : [ProcessRecords] - 11000 >> Last key : f214a31d-8b8a-4c01-9a2f-98db86dc630c, records per sec: 708.7172
98411f57f9ba 2022-11-09 14:12:33,849 INFO : [ProcessRecords] - Total records processed: 11387

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

sample -dr dr632

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022 

biocache> sample -dr dr632
There is an incremental row key file for this resource.  Would you like to perform an incremental process (y/n) ?
y
98411f57f9ba 2022-11-09 14:16:27,367 INFO : [Counter] - Sampling : dr632
98411f57f9ba 2022-11-09 14:16:27,368 INFO : [Sampling] - Creating distinct list of coordinates for row keys in /data/tmp/row_key_dr632.csv
98411f57f9ba 2022-11-09 14:17:01,333 INFO : [Sampling] - Created distinct list of coordinates for row keys in /data/tmp/row_key_dr632.csv. Number of coordinates: 22
98411f57f9ba 2022-11-09 14:17:01,333 INFO : [Sampling] - Writing to file: /data/tmp/loc-dr632.txt
98411f57f9ba 2022-11-09 14:17:01,350 INFO : [Sampling] - Finished writing to file: /data/tmp/loc-dr632.txt
98411f57f9ba 2022-11-09 14:17:01,350 INFO : [Counter] - Running sampling...writing to file: /data/tmp/sampling-dr632.txt
98411f57f9ba 2022-11-09 14:17:01,351 INFO : [Sampling] - ********* START - TEST BATCH SAMPLING FROM FILE ***************
98411f57f9ba 2022-11-09 14:17:01,533 INFO : [Config] - Number of fields to sample: 75
98411f57f9ba 2022-11-09 14:17:01,533 INFO : [Config] - Fields to sample: el10000,el10001,el10002,el10003,el10004,el10005,el10006,el10007,el10008,el10009,el10010,el10011,el10012,el10013,el10014,el10015,el10016,el10017,el10018,el10019,el10020,el10021,el10022,el10023,el10024,el10025,el10026,el10027,el10028,el10029,el10030,el10031,el10032,el10033,el10034,el10035,el10036,el10044,cl10067,cl10066,cl10065,cl10064,cl10063,cl10061,cl10059,cl10058,cl10057,cl10042,cl10041,cl10040,cl10038,cl10074,cl10073,cl10071,cl10070,cl10068,cl10055,cl10054,cl10053,cl10052,cl10051,cl10050,cl10048,cl10047,cl10046,cl10082,cl10083,cl10084,cl10087,cl10089,cl10090,cl10097,cl10101,cl10102,cl10104
98411f57f9ba 2022-11-09 14:17:01,544 INFO : [PointsReader] - Loading points from file: /data/tmp/loc-dr632.txt
98411f57f9ba 2022-11-09 14:17:18,661 INFO : [Sampling] - Total points sampled so far : 21
98411f57f9ba 2022-11-09 14:17:18,663 INFO : [PointsReader] - Loading points from file: /data/tmp/loc-dr632.txt
98411f57f9ba 2022-11-09 14:17:18,665 INFO : [LoadSamplingConsumer] - Loading the sampling into the database: /data/tmp/sampling-dr632.txt
98411f57f9ba 2022-11-09 14:17:18,680 INFO : [Sampling] - Total points sampled : 21, output file: /data/tmp/sampling-dr632.txt point file: /data/tmp/loc-dr632.txt
98411f57f9ba 2022-11-09 14:17:18,680 INFO : [Sampling] - ********* END - TEST BATCH SAMPLING FROM FILE ***************
98411f57f9ba 2022-11-09 14:17:18,752 INFO : [LoadSamplingConsumer] - Finished loading: /data/tmp/sampling-dr632.txt in 86ms
98411f57f9ba 2022-11-09 14:17:18,780 INFO : [Counter] - Loading sampling into occ table
98411f57f9ba 2022-11-09 14:17:18,781 INFO : [Counter] - Starting loading sampling for /data/tmp/row_key_dr632.csv
98411f57f9ba 2022-11-09 14:17:19,241 INFO : [Counter] - 1009 >> Loading sampling records per sec: 2183.406, lastkey: 4840f76f-f82c-4f65-b276-ae432b79eaec
98411f57f9ba 2022-11-09 14:17:19,567 INFO : [Counter] - 2006 >> Loading sampling records per sec: 3076.923, lastkey: 50cb9031-f2ab-4d49-9eeb-334ce2943953
98411f57f9ba 2022-11-09 14:17:19,998 INFO : [Counter] - 3003 >> Loading sampling records per sec: 2325.5813, lastkey: 696978cb-183a-4387-9357-3be6e8420098
98411f57f9ba 2022-11-09 14:17:20,452 INFO : [Counter] - 4007 >> Loading sampling records per sec: 2202.6433, lastkey: dcc58958-561b-404e-99a0-f4ec286abbdc
98411f57f9ba 2022-11-09 14:17:20,793 INFO : [Counter] - 5007 >> Loading sampling records per sec: 2941.1765, lastkey: 07214217-36cf-4da8-81c0-0be5ab070a1f
98411f57f9ba 2022-11-09 14:17:21,431 INFO : [Counter] - 6008 >> Loading sampling records per sec: 1567.3981, lastkey: 522f08a8-9ce9-47e7-b589-5c2c710dfa81
98411f57f9ba 2022-11-09 14:17:21,748 INFO : [Counter] - 7009 >> Loading sampling records per sec: 3154.5742, lastkey: 524e2ff9-b0f3-423a-b5d9-13b1fe7ba6fa
98411f57f9ba 2022-11-09 14:17:22,062 INFO : [Counter] - 8006 >> Loading sampling records per sec: 3184.7134, lastkey: 613d9f3c-6f84-4e3a-a3ca-5ec8d4c935a3
98411f57f9ba 2022-11-09 14:17:22,360 INFO : [Counter] - 9007 >> Loading sampling records per sec: 3367.0034, lastkey: a11d46be-0227-4711-b4f7-75a8f6462474
98411f57f9ba 2022-11-09 14:17:22,681 INFO : [Counter] - 10006 >> Loading sampling records per sec: 3115.2646, lastkey: cced1cdc-77ff-4c33-bf4b-1ef18d6b5f41
98411f57f9ba 2022-11-09 14:17:23,053 INFO : [Counter] - 11007 >> Loading sampling records per sec: 2688.1719, lastkey: f214a31d-8b8a-4c01-9a2f-98db86dc630c
98411f57f9ba 2022-11-09 14:17:23,202 INFO : [Counter] - Completed loading sampling into occ table

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

index -dr -dr632

biocache> index -dr dr632
There is an incremental row key file for this resource.  Would you like to perform an incremental process (y/n) ?
y
98411f57f9ba 2022-11-09 14:19:57,149 INFO : [IndexRecords] - Starting the reindex by row key....
98411f57f9ba 2022-11-09 14:19:57,161 INFO : [SolrIndexDAO] - Initialising the solr server zookeeper1:2181,zookeeper2:2181 cloudserver:null solrServer:null
98411f57f9ba 2022-11-09 14:19:57,961 INFO : [TaxonSpeciesListDAO] - Loading list information...
98411f57f9ba 2022-11-09 14:19:58,097 INFO : [TaxonSpeciesListDAO] - Loading data from dr174
98411f57f9ba 2022-11-09 14:19:58,097 INFO : [TaxonSpeciesListDAO] - Downloading list data from https://lists.biodiversitydata.se/speciesListItem/downloadList/dr174?fetch=%7BkvpValues%3Dselect%7D
98411f57f9ba 2022-11-09 14:19:58,513 INFO : [TaxonSpeciesListDAO] - Loading data from dr204
98411f57f9ba 2022-11-09 14:19:58,513 INFO : [TaxonSpeciesListDAO] - Downloading list data from https://lists.biodiversitydata.se/speciesListItem/downloadList/dr204?fetch=%7BkvpValues%3Dselect%7D
98411f57f9ba 2022-11-09 14:20:24,204 INFO : [TaxonSpeciesListDAO] - Loading list information...completed.
98411f57f9ba 2022-11-09 14:20:24,235 INFO : [SpeciesGroups] - Using default legacy species groups
98411f57f9ba 2022-11-09 14:20:24,812 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://solr2:8983/solr/biocache: No such path /schema/fields/organismQuantity, retry? 0
98411f57f9ba 2022-11-09 14:20:24,815 INFO : [SolrIndexDAO] - Field not in schema: organismQuantity
98411f57f9ba 2022-11-09 14:20:24,823 INFO : [SolrIndexDAO] - Adding field: organismQuantity
98411f57f9ba 2022-11-09 14:20:25,501 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://solr2:8983/solr/biocache: No such path /schema/fields/materialSampleID, retry? 0
98411f57f9ba 2022-11-09 14:20:25,501 INFO : [SolrIndexDAO] - Field not in schema: materialSampleID
98411f57f9ba 2022-11-09 14:20:25,501 INFO : [SolrIndexDAO] - Adding field: materialSampleID
98411f57f9ba 2022-11-09 14:20:27,359 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://solr4:8983/solr/biocache: No such path /schema/fields/organismQuantityType, retry? 0
98411f57f9ba 2022-11-09 14:20:27,359 INFO : [SolrIndexDAO] - Field not in schema: organismQuantityType
98411f57f9ba 2022-11-09 14:20:27,359 INFO : [SolrIndexDAO] - Adding field: organismQuantityType
98411f57f9ba 2022-11-09 14:20:28,573 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://solr1:8983/solr/biocache: No such path /schema/fields/sampleSizeValue, retry? 0
98411f57f9ba 2022-11-09 14:20:28,573 INFO : [SolrIndexDAO] - Field not in schema: sampleSizeValue
98411f57f9ba 2022-11-09 14:20:28,573 INFO : [SolrIndexDAO] - Adding field: sampleSizeValue
98411f57f9ba 2022-11-09 14:20:29,439 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://solr4:8983/solr/biocache: No such path /schema/fields/sampleSizeUnit, retry? 0
98411f57f9ba 2022-11-09 14:20:29,439 INFO : [SolrIndexDAO] - Field not in schema: sampleSizeUnit
98411f57f9ba 2022-11-09 14:20:29,439 INFO : [SolrIndexDAO] - Adding field: sampleSizeUnit
98411f57f9ba 2022-11-09 14:20:34,505 INFO : [IndexRecords] - 1000 >> Last key : 4840f76f-f82c-4f65-b276-ae432b79eaec, records per sec: 26.77161
98411f57f9ba 2022-11-09 14:20:38,589 INFO : [IndexRecords] - 2000 >> Last key : 50cb9031-f2ab-4d49-9eeb-334ce2943953, records per sec: 244.85797
98411f57f9ba 2022-11-09 14:20:42,537 INFO : [IndexRecords] - 3000 >> Last key : 696978cb-183a-4387-9357-3be6e8420098, records per sec: 253.29282
98411f57f9ba 2022-11-09 14:20:46,524 INFO : [IndexRecords] - 4000 >> Last key : dcc58958-561b-404e-99a0-f4ec286abbdc, records per sec: 250.87807
98411f57f9ba 2022-11-09 14:20:50,649 INFO : [IndexRecords] - 5000 >> Last key : 07214217-36cf-4da8-81c0-0be5ab070a1f, records per sec: 242.42424
98411f57f9ba 2022-11-09 14:20:54,548 INFO : [IndexRecords] - 6000 >> Last key : 522f08a8-9ce9-47e7-b589-5c2c710dfa81, records per sec: 256.476
98411f57f9ba 2022-11-09 14:20:58,510 INFO : [IndexRecords] - 7000 >> Last key : 524e2ff9-b0f3-423a-b5d9-13b1fe7ba6fa, records per sec: 252.39778
98411f57f9ba 2022-11-09 14:21:01,829 INFO : [IndexRecords] - 8000 >> Last key : 613d9f3c-6f84-4e3a-a3ca-5ec8d4c935a3, records per sec: 301.29556
98411f57f9ba 2022-11-09 14:21:05,066 INFO : [IndexRecords] - 9000 >> Last key : a11d46be-0227-4711-b4f7-75a8f6462474, records per sec: 308.928
98411f57f9ba 2022-11-09 14:21:08,780 INFO : [IndexRecords] - 10000 >> Last key : cced1cdc-77ff-4c33-bf4b-1ef18d6b5f41, records per sec: 269.2515
98411f57f9ba 2022-11-09 14:21:12,804 INFO : [IndexRecords] - 11000 >> Last key : f214a31d-8b8a-4c01-9a2f-98db86dc630c, records per sec: 248.57074
98411f57f9ba 2022-11-09 14:21:14,151 INFO : [SolrIndexDAO] - Performing index commit....
98411f57f9ba 2022-11-09 14:21:16,868 INFO : [SolrIndexDAO] - Performing index commit....done
98411f57f9ba 2022-11-09 14:21:16,913 INFO : [SolrIndexDAO] - >>>> Document count of index: 16343
98411f57f9ba 2022-11-09 14:21:16,914 INFO : [SolrIndexDAO] - Shutting down the indexing...
98411f57f9ba 2022-11-09 14:21:17,220 INFO : [SolrIndexDAO] - Finalise finished.

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

Stop -> /opt/sbdi/bin/stop_docker_app solrcloud
Start -> /opt/sbdi/bin/start_docker_app solrcloud

docker service logs -f solrcloud_solr1

solrcloud_solr1.1.nx0sl798dufs@uat-solr-1    | Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 within 30000 ms
solrcloud_solr1.1.nx0sl798dufs@uat-solr-1    | 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:250)
solrcloud_solr1.1.nx0sl798dufs@uat-solr-1    | 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:193)
solrcloud_solr1.1.nx0sl798dufs@uat-solr-1    | 	... 54 more

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022 •
edited

Failed to connect to Zookeeper

solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | java.net.UnknownHostException: zookeeper3: Name or service not known
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.InetAddress.getByName(InetAddress.java:1077)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:181)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:591)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:958)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 2022-11-09 14:28:11,236 [myid:1] - INFO  [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@847] - Notification time out: 12800

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

Try to restart with this command:

  1. docker service update --force solrcloud_solr1
  2. docker service update --force solrcloud_solr4
  3. docker service update --force solrcloud_solr2
  4. docker service update --force solrcloud_solr3

Error when it comes to mounting the disk
docker service update --force solrcloud_zookeeper1
solrcloud_zookeeper1
overall progress: 0 out of 1 tasks
1/1: failed to mount local volume: mount /docker/var/volumes/solrcloud/data_zoo…
service update paused: update paused due to failure or early termination of task jt8o9waf5o7hprxhmzmclmwsk

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

2022-11-09

ingimar@uat-manager-1:~$ docker service ps solrcloud_zookeeper1
ID             NAME                         IMAGE              NODE            DESIRED STATE   CURRENT STATE             ERROR                              PORTS
71cr8nk8eqp3   solrcloud_zookeeper1.1       zookeeper:3.4.14   uat-solr-1      Running         Running 26 minutes ago                                       *:3888->3888/tcp,*:3888->3888/tcp,*:2888->2888/tcp,*:2888->2888/tcp,*:2181->2181/tcp,*:2181->2181/tcp
x0bvmny1q5ts    \_ solrcloud_zookeeper1.1   zookeeper:3.4.14   uat-manager-3   Shutdown        Rejected 26 minutes ago   "failed to mount local volume:…"   
zbo6mq2m4n6y    \_ solrcloud_zookeeper1.1   zookeeper:3.4.14   uat-manager-3   Shutdown        Rejected 26 minutes ago   "failed to mount local volume:…"   
qo17l309mlwb    \_ solrcloud_zookeeper1.1   zookeeper:3.4.14   uat-solr-4      Shutdown        Rejected 26 minutes ago   "failed to mount local volume:…"   
ql8rh45jsg5t    \_ solrcloud_zookeeper1.1   zookeeper:3.4.14   uat-solr-4      Shutdown        Rejected 26 minutes ago   "failed to mount local volume:…"

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

Jag kan se att disken finns på uat-solr1

                          Size  Used Avail Use% Mounted on
udev                                            16G     0   16G   0% /dev
tmpfs                                          3.2G  1.5M  3.2G   1% /run
/dev/vda1                                       20G   12G  7.3G  63% /
tmpfs                                           16G     0   16G   0% /dev/shm
tmpfs                                          5.0M     0  5.0M   0% /run/lock
tmpfs                                           16G     0   16G   0% /sys/fs/cgroup
/dev/loop1                                      56M   56M     0 100% /snap/core18/2620
/dev/mapper/vg_docker_local-lv_var_lib_docker   79G   19G   56G  25% /docker_var_lib
/dev/loop0                                      56M   56M     0 100% /snap/core18/2566
/dev/loop3                                      64M   64M     0 100% /snap/core20/1634
/dev/loop4                                      71M   71M     0 100% /snap/lxd/19647
/dev/loop5                                      68M   68M     0 100% /snap/lxd/22753
/dev/loop6                                      48M   48M     0 100% /snap/snapd/17336
/dev/loop7                                      48M   48M     0 100% /snap/snapd/17029
/dev/mapper/vg_docker_local-lv_docker_local    200G  1.7G  199G   1% /docker_local
/dev/vda15                                     105M  5.2M  100M   5% /boot/efi
uat-manager-1:/docker                          796G  491G  306G  62% /docker
/dev/loop8                                      64M   64M     0 100% /snap/core20/1695
tmpfs                                          3.2G     0  3.2G   0% /run/user/1700

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

Fungerar nu klockan 16:35 - 2021-11-09

biocache> index -dr dr632
There is an incremental row key file for this resource.  Would you like to perform an incremental process (y/n) ?
y
7e1638f6c05d 2022-11-09 15:33:48,009 INFO : [IndexRecords] - Starting the reindex by row key....
7e1638f6c05d 2022-11-09 15:33:48,102 INFO : [SolrIndexDAO] - Initialising the solr server zookeeper1:2181,zookeeper2:2181 cloudserver:null solrServer:null
7e1638f6c05d 2022-11-09 15:33:49,477 INFO : [TaxonSpeciesListDAO] - Loading list information...
7e1638f6c05d 2022-11-09 15:33:49,958 INFO : [TaxonSpeciesListDAO] - Loading data from dr174
7e1638f6c05d 2022-11-09 15:33:49,959 INFO : [TaxonSpeciesListDAO] - Downloading list data from https://lists.biodiversitydata.se/speciesListItem/downloadList/dr174?fetch=%7BkvpValues%3Dselect%7D
7e1638f6c05d 2022-11-09 15:33:50,301 INFO : [TaxonSpeciesListDAO] - Loading data from dr204
7e1638f6c05d 2022-11-09 15:33:50,301 INFO : [TaxonSpeciesListDAO] - Downloading list data from https://lists.biodiversitydata.se/speciesListItem/downloadList/dr204?fetch=%7BkvpValues%3Dselect%7D
7e1638f6c05d 2022-11-09 15:34:16,035 INFO : [TaxonSpeciesListDAO] - Loading list information...completed.
7e1638f6c05d 2022-11-09 15:34:16,343 INFO : [SpeciesGroups] - Using default legacy species groups
7e1638f6c05d 2022-11-09 15:34:24,219 INFO : [IndexRecords] - 1000 >> Last key : 4840f76f-f82c-4f65-b276-ae432b79eaec, records per sec: 27.617443
7e1638f6c05d 2022-11-09 15:34:29,020 INFO : [IndexRecords] - 2000 >> Last key : 50cb9031-f2ab-4d49-9eeb-334ce2943953, records per sec: 208.33333
7e1638f6c05d 2022-11-09 15:34:33,188 INFO : [IndexRecords] - 3000 >> Last key : 696978cb-183a-4387-9357-3be6e8420098, records per sec: 239.92322
7e1638f6c05d 2022-11-09 15:34:37,362 INFO : [IndexRecords] - 4000 >> Last key : dcc58958-561b-404e-99a0-f4ec286abbdc, records per sec: 239.63576
7e1638f6c05d 2022-11-09 15:34:41,972 INFO : [IndexRecords] - 5000 >> Last key : 07214217-36cf-4da8-81c0-0be5ab070a1f, records per sec: 216.9668
7e1638f6c05d 2022-11-09 15:34:46,001 INFO : [IndexRecords] - 6000 >> Last key : 522f08a8-9ce9-47e7-b589-5c2c710dfa81, records per sec: 248.20056
7e1638f6c05d 2022-11-09 15:34:50,078 INFO : [IndexRecords] - 7000 >> Last key : 524e2ff9-b0f3-423a-b5d9-13b1fe7ba6fa, records per sec: 245.33855
7e1638f6c05d 2022-11-09 15:34:54,052 INFO : [IndexRecords] - 8000 >> Last key : 613d9f3c-6f84-4e3a-a3ca-5ec8d4c935a3, records per sec: 251.63564
7e1638f6c05d 2022-11-09 15:34:57,989 INFO : [IndexRecords] - 9000 >> Last key : a11d46be-0227-4711-b4f7-75a8f6462474, records per sec: 254.0005
7e1638f6c05d 2022-11-09 15:35:01,950 INFO : [IndexRecords] - 10000 >> Last key : cced1cdc-77ff-4c33-bf4b-1ef18d6b5f41, records per sec: 252.4615
7e1638f6c05d 2022-11-09 15:35:06,087 INFO : [IndexRecords] - 11000 >> Last key : f214a31d-8b8a-4c01-9a2f-98db86dc630c, records per sec: 241.7795
7e1638f6c05d 2022-11-09 15:35:07,607 INFO : [SolrIndexDAO] - Performing index commit....
7e1638f6c05d 2022-11-09 15:35:11,397 INFO : [SolrIndexDAO] - Performing index commit....done
7e1638f6c05d 2022-11-09 15:35:11,544 INFO : [SolrIndexDAO] - >>>> Document count of index: 16343
7e1638f6c05d 2022-11-09 15:35:11,545 INFO : [SolrIndexDAO] - Shutting down the indexing...
7e1638f6c05d 2022-11-09 15:35:11,663 INFO : [SolrIndexDAO] - Finalise finished.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment