-
-
Save moziauddin/c7ba94ca89bcc404e08246fbd1a193c2 to your computer and use it in GitHub Desktop.
BNTI env setup, full-automation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| git repo name | SHA / tag | build command | version | artifact name | |
|---|---|---|---|---|---|
| https://github.com/bio-org-au/nsl-domain-plugin.git | 34e671f818c83dffba672a1938c060faa2d01db9 | ./build.sh | 1.10 | nsl-domain-plugin-1.10.zip | |
| https://github.com/bio-org-au/services.git | e251bdd5f20e73832ec81501ae530b924e1dda2f | ./build.sh | 1.0206 | nxl#services##1.0206.war | |
| https://github.com/bio-org-au/mapper.git | 138d1ddd8e71c7a79c7405d3269fd6ceb00aa87f | ./build.sh | 1.0022 | nsl#mapper##1.0022.war | |
| https://github.com/bio-org-au/nsl-editor.git | 9675e53469f352fcf4a439b0d8eeacbd91f12285 | ./build.sh | 1.67 | nxl#editor##1.67.war |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # BEFORE running this script: sudo systemctl stop tomcat.service | |
| # | |
| # TODO: - this assumes /tmp/tblbiota_20180717.csv | |
| # - replace all hard-coded 'tblbiota_20180717.csv' with at least 'tblbiota.csv' | |
| # - parametrize/templatize the hard-coded hostname (for example: dev50-icn.oztaxa.com) | |
| # - etc | |
| dropdb nsl && createdb nsl | |
| psql -d nsl -c "create schema mapper;" | |
| psql -f dawr_ddl.sql -d nsl | |
| psql -f dawr_import.sql -d nsl | |
| psql -f mapper-ddl.sql -d nsl | |
| psql -d nsl -c "INSERT INTO mapper.db_version (id, version) VALUES (1, 5);" | |
| psql -d nsl -c "INSERT INTO mapper.host (id, host_name, preferred) VALUES (1, 'dev51-icn.oztaxa.com/nxl/mapper', true);" | |
curl -s -L 'https://dev55-icn.oztaxa.com/nxl/services/search?product=BPNI&tree.id=&name=&inc._scientific=&inc.scientific=on&inc._cultivar=&inc._other=&max=7500&display=apni&search=true' > dev55-all-records_test.html
grep 'https://dev55-icn.oztaxa.com' ./dev55-all-records_test.html | wc -l
7222
# and the main test/verification, for each https://dev55-icn.oztaxa.com linke do:
curl -s -I -L 'https://dev55-icn.oztaxa.com/nxl/editor/search?query=id:40003844&query_field=name-instances&query_on=instance'
HTTP/2 302
date: Tue, 07 Aug 2018 07:44:01 GMT
content-type: text/html;charset=utf-8
location: https://dev55-icn.oztaxa.com/nxl/editor/sign_in
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
cache-control: no-cache
set-cookie: _nsl_editor_session_test=WkY0WnFFbXFhcVdhSkdNWnRoNi9KZm1FQTVJK0pybnozYitiaWRwUG9FSjdEY1JScm50Q2hmOFF6YTdlMFFSeHVoMkdYUVUxa3daSDlaMkl6R3VUT3pZNGJJNFZQblJBYXdHbEZHVzkwdUorUzQ1NmpwY0NxWHdwWHBORi8vUDNvRG4xQzcxb0xpNHJHd05PUmxveWp5V3dGY2hyT2JjSXpRUkFScWowR2E5bytqRnBQbjFMYmgya2pFeUkwYVNYeDRqY2IzRm5sS3lPZXozaWdna3ZaZ2hsMzVOK0NLcDV4Nk9MOGFZSlNSUmNuclRjaUR0NldMQk9JTHJaYlRUUkNnK1NZeW9EV25lVkZteU13b1F0RFVwOEVKMVR0MTIwQWpEYTZSMEh0K2tJZkkyR3ZJNUtTcVduTEh6OFlEMHkxSlphVEI4enJ1aE9rUnBjMGlPendBPT0tLUdCc0xLQzV5MzRvckcwWHJ1M1lCeGc9PQ%3D%3D--15d000c5e933f2eb7f7495e5a848a25a15a46104; path=/; expires=Tue, 07 Aug 2018 10:44:01 -0000; HttpOnly
x-request-id: 4035fb74-79cf-418c-8278-a7dd8d60d719
x-runtime: 0.026098
HTTP/2 200
date: Tue, 07 Aug 2018 07:44:02 GMT
content-type: text/html;charset=utf-8
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
etag: W/"260f5c0e17b8c07fb449fa080e60db0a"
cache-control: max-age=0, private, must-revalidate
set-cookie: _nsl_editor_session_test=RkNNVCtpMnNSNWdaMTJvc256SHJLR3FsMHY1dDdkYlpvN0hPWFpoK2tQZm5Pc0hpTEZsNTRadTBmM2JzQTRreUV0YnVtQTlHNVR4NGNyazdER0lPaS9YNlQ5RUdleHBZNnZqSllmQUVPQytZalpVaDJzdXVVZW0zcUNUNm9sRC9XUlNnMW8yMGp4YU9PSzRFRXN0T0RBPT0tLVJaZEdFdGltR0hhSkwxVm4wanBjb2c9PQ%3D%3D--e21144a4523975258be6f2a74f3e39759fa5895b; path=/; expires=Tue, 07 Aug 2018 10:44:02 -0000; HttpOnly
x-request-id: 36a581c2-3451-4407-9c3b-3bb274b93bdc
x-runtime: 0.105420
- java version: ORACLE jdk1.8.0_171 (use latest stable in 1.8)
- jruby: 9.1.13.0
- tomcat version: 8.5.13 (use latest stable in 8.5)
- postgresql version: 9.5.13 see locale WARNING/notes bellow
initdb --locale=en_AU - apacheds: 2.0.0-M17
ANBG do build and upload their artifacts into a "private" maven repo that is not accessible to the outside world, and the build of nsl-domain-plugin fails, for now we use this simple workaround in order to build the plugin ourselves (TODO: make proper fix):
diff --git a/grails-app/conf/BuildConfig.groovy b/grails-app/conf/BuildConfig.groovy
index ec5c446..37b0bdb 100644
--- a/grails-app/conf/BuildConfig.groovy
+++ b/grails-app/conf/BuildConfig.groovy
@@ -46,8 +46,9 @@ grails.project.dependency.resolution = {
log "warn" // log level of Ivy resolver, either 'error', 'warn', 'info', 'debug' or 'verbose'
repositories {
-
+
mavenLocal()
+ mavenRepo("https://nexus.ala.org.au/content/groups/public/")
mavenRepo(id: "anbg", url: "http://appsdev1-ibis.it.csiro.au:8085/repository/maven-central/")
mavenRepo(id: "anbg", url: "http://appsdev1-ibis.it.csiro.au:8085/repository/grails-plugins/")
grailsHome()The plugin will be (for now) installed in the maven repo/cache:
mbohun@linux-khr1:~> find ~/.m2/repository -name "*.zip" | grep nsl-domain
/home/mbohun/.m2/repository/au/org/biodiversity/grails/plugins/nsl-domain-plugin/1.11-SNAPSHOT/nsl-domain-plugin-1.11-SNAPSHOT.zip
/home/mbohun/.m2/repository/au/org/biodiversity/grails/plugins/nsl-domain-plugin/1.10/nsl-domain-plugin-1.10.zip
-
add/fix this in the jenkins build
cp nsl#mapper##1.0022.war nxl#mapper##1.0022.war -
sudo systemctl stop tomcat.service
-
./BNTi-icn_load-data.sh > /tmp/BNTi-icn_load-data.log 2>&1
-
sudo systemctl start tomcat.service
-
login to https://dev51-icn.oztaxa.com/nxl/services/ go to admin -> "Reconstruct name strings"
-
psql -f dawr_tree_import.sql -d nsl > /tmp/dawr_tree_import.log 2>&1
-
psql -f Add_the_mapper_IDs_for_the_tree_elements_created.sql -d nsl > /tmp/add_mapper_ids.log 2>&1
-
more sql fixes/mods:
# 2018-08-01 # update/adjust tree host_name update tree set host_name='https://dev51-icn.oztaxa.com/nxl/mapper'; # mapper host_name update update mapper.host set host_name='dev51-icn.oztaxa.com/nxl/mapper'; # name_space adjustment (this will be templetized) update mapper.identifier set name_space ='bpni' where name_space ='apni'; # update all tree-s group_name to 'treebuilder' update tree set group_name='treebuilder'; # updating tree mapper links update tree_element set instance_link = regexp_replace(instance_link, 'https://dev51-icn.oztaxa.com', 'https://dev51-icn.oztaxa.com/nxl/mapper'), name_link = regexp_replace(name_link, 'https://dev51-icn.oztaxa.com', 'https://dev51-icn.oztaxa.com/nxl/mapper');
BNTi-icn_load-data.sh BASH script REPLACES (ALMOST) ALL BELLOW THIS POINT
This BASH script represent-s the outstanding/missing step that had to be executed manually after the end of the installation:
#!/bin/bash
# NOTE/TODO: - we are cleaning this up
# - for now this script assumes that you did:
# 1. install and start tomcat
# 2. deploy the 4 war files
# - /opt/tomcat/webapps/nsl#mapper##1.0022.war
# - /opt/tomcat/webapps/nxl#editor##1.65.war
# - /opt/tomcat/webapps/nxl#services##1.0204.war
# the war-s / webapps will throw exceptions and fail to start, BUT this step creates/installs:
# /opt/apache-tomcat-8.5.13/webapps/nxl#services##1.0204/plugins/nsl-domain-plugin-1.10/sql
# with .sql script-s (update-to-25.sql) we have to run further bellow.
# 3. stop tomcat (sudo systemctl restart tomcat)
#
# NOTE: if you do 'sudo su postgress' then you do *NOT* need to pass
# '-h localhost -p 5432 -U nsldev -d nsl' to psql
#
sudo su postgress
psql -h localhost -p 5432 -U nsldev -d nsl -f /tmp/icn-nsl-ddl.sql
cd /opt/apache-tomcat-8.5.13/webapps/nxl#services##1.0204/plugins/nsl-domain-plugin-1.10/sql
psql -h localhost -p 5432 -U nsldev -d nsl -f ./update-to-25.sql
# FIX: mapper-ddl-pulled_form_prod.sql is not in git, given to us by ANBG dev-s on #Slack
#
psql -h localhost -p 5432 -U nsldev -d nsl -f /tmp/mapper-ddl-pulled_form_prod.sql
psql -h localhost -p 5432 -U nsldev -d nsl -f /tmp/mapper-ddl.sql
# FIX: mapper.db_version table created, but empty
psql -d nsl -c "INSERT INTO mapper.db_version (id, version) VALUES (1, 5);"
# FIX: always make sure the mapper points to the localhost mapper
# from: psql -d nsl -c "update mapper.host set host_name = 'localhost:7070/nsl-mapper' where preferred = true;"
psql -d nsl -c "INSERT INTO mapper.host (id, host_name, preferred) VALUES (1, 'localhost:7070/nsl-mapper', true);"
# back to linux user ubuntu
exit
# for changes to files owned/used by linux user tomcat
sudo su tomcat
# FIX: editor, add missing property
# TODO: this same property (value) was just set/written into the DB mapper.host table, do we really insist on
# reading the same property from multiple places?
echo 'Rails.configuration.nsl_linker = "http://localhost:7070/nsl-mapper/"' >> $CATALINA_HOME/.nsl/editor-config.rb
# exit as user tomcat to become ubuntu again (default)
exit
# as user ubuntu:
sudo systemctl restart tomcat
- replace dev42 in
*.sql4 entriespostgres@ip-192-168-1-237:/home/ubuntu/2017-07-30_BNTi-icn-data_load$ ls -lahF total 8.3M drwxrwxr-x 3 ubuntu ubuntu 4.0K Jul 31 11:10 ./ drwxr-xr-x 6 ubuntu ubuntu 4.0K Jul 31 11:10 ../ -rw-rw-r-- 1 ubuntu ubuntu 232K Jul 23 13:05 dawr_ddl.sql -rw-rw-r-- 1 ubuntu ubuntu 21K Jul 31 11:04 dawr_import.sql -rw-rw-r-- 1 ubuntu ubuntu 1.8K Jul 24 14:25 dawr_readme.md -rw-rw-r-- 1 ubuntu ubuntu 3.9K Jul 31 11:04 dawr_tree_import.sql -rw-rw-r-- 1 ubuntu ubuntu 4.0K Jul 31 11:09 mapper-ddl.sql drwxrwxr-x 8 ubuntu ubuntu 4.0K Jul 31 11:08 nsl-infra/ -rw-r--r-- 1 ubuntu ubuntu 1.7M Jul 30 11:18 services.zip -rw-rw-r-- 1 ubuntu ubuntu 6.4M Jul 23 14:11 tblBiota_20180717.csv
- name: Create nsl.public tables, views etc
become: true
become_user: postgres
command: psql -f /tmp/"{{ shard_nsl_ddl_file }}" nsl
- name: Create nsl.mapper tables, views etc
become: true
become_user: postgres
command: psql -f /tmp/mapper-ddl.sql nsl
This psql line came from ANBG's bin/start-nsl-dev.sh
mbohun@linux-khr1:~/src/nsl-dev-env> cat bin/start-nsl-dev.sh
#! /bin/bash
~/apps/apacheds-2.0.0-M17/bin/apacheds.sh start
# always make sure the mapper points to the localhost mapper
psql -d nsl -c "update mapper.host set host_name = 'localhost:7070/nsl-mapper' where preferred = true;"
cd ~/devel/anbg/nsl/apache-tomcat-8.5.13
bin/startup.sh
tail -f logs/catalina.out
# always make sure the mapper points to the localhost mapper
psql -h localhost -p 5432 -U nsldev -d nsl -c "update mapper.host set host_name = 'localhost:7070/nsl-mapper' where preferred = true;"
This was added manually, after the DB was created:
nsl=# \dt mapper.
List of relations
Schema | Name | Type | Owner
--------+-----------------------+-------+--------
mapper | db_version | table | nsldev
mapper | host | table | nsldev
mapper | identifier | table | nsldev
mapper | identifier_identities | table | nsldev
mapper | match | table | nsldev
mapper | match_host | table | nsldev
(6 rows)
nsl=# select * from mapper.db_version;
id | version
----+---------
(0 rows)
nsl=# INSERT INTO mapper.db_version (id, version) VALUES (1, 5);
INSERT 0 1
nsl=# select * from mapper.db_version;
id | version
----+---------
1 | 5
(1 row)
nsl=# select * from mapper.host
nsl-# ;
id | host_name | preferred
----+-----------+-----------
(0 rows)
nsl=# update mapper.host set host_name = 'localhost:7070/nsl-mapper' where preferred = true;
UPDATE 0
nsl=# select * from mapper.host
;
id | host_name | preferred
----+-----------+-----------
(0 rows)
nsl=# INSERT INTO mapper.host (id, host_name, preferred) VALUES (0, 'localhost:7070/nsl-mapper', true);
INSERT 0 1
nsl=# select * from mapper.host
;
id | host_name | preferred
----+---------------------------+-----------
0 | localhost:7070/nsl-mapper | t
(1 row)
The services startup at the moment crashes/throws-exception:
2018-07-16 13:13:58.118 [DEBUG] grails.app.services.au.org.biodiversity.nsl.ConfigService:128 - read config: classification tree key: APC
2018-07-16 13:13:58.210 [ INFO] grails.app.services.au.org.biodiversity.nsl.NslDomainService:152 - updating to version 25
2018-07-16 13:13:58.215 [ INFO] grails.app.services.au.org.biodiversity.nsl.NslDomainService:152 - found nslDomainPlugin with path /plugins/nsl-domain-plugin-1.10
2018-07-16 13:13:58.216 [ INFO] grails.app.services.au.org.biodiversity.nsl.NslDomainService:152 - nsl-ddl.sql file path /opt/apache-tomcat-8.5.13/webapps/nxl#services##1.0204/plugins/nsl-domain-plugin-1.10/sql/update-to-25.sql
2018-07-16 13:13:58.323 [ERROR] org.codehaus.groovy.grails.web.context.GrailsContextLoaderListener:213 - Error initializing the application: No such property: scheme for class: SimpleTemplateScript1
groovy.lang.MissingPropertyException: No such property: scheme for class: SimpleTemplateScript1
at SimpleTemplateScript1.run(SimpleTemplateScript1.groovy:1)
at au.org.biodiversity.nsl.NslDomainService.replaceParams(NslDomainService.groovy:111)
at au.org.biodiversity.nsl.NslDomainService.updateToCurrentVersion(NslDomainService.groovy:63)
at BootStrap$_closure1.doCall(BootStrap.groovy:37)
at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:327)
at grails.util.Environment.executeForEnvironment(Environment.java:320)
at grails.util.Environment.executeForCurrentEnvironment(Environment.java:296)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-07-16 13:13:58.325 [ERROR] org.codehaus.groovy.grails.web.context.GrailsContextLoaderListener:213 - Error initializing Grails: No such property: scheme for class: SimpleTemplateScript1
groovy.lang.MissingPropertyException: No such property: scheme for class: SimpleTemplateScript1
at SimpleTemplateScript1.run(SimpleTemplateScript1.groovy:1)
at au.org.biodiversity.nsl.NslDomainService.replaceParams(NslDomainService.groovy:111)
at au.org.biodiversity.nsl.NslDomainService.updateToCurrentVersion(NslDomainService.groovy:63)
at BootStrap$_closure1.doCall(BootStrap.groovy:37)
at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:327)
at grails.util.Environment.executeForEnvironment(Environment.java:320)
at grails.util.Environment.executeForCurrentEnvironment(Environment.java:296)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-07-16 13:13:58.331 [ WARN] org.codehaus.groovy.grails.lifecycle.ShutdownOperations:189 - Error occurred running shutdown operation: BeanFactory not initialized or already closed - call 'refresh' before accessing beans via the ApplicationContext
java.lang.IllegalStateException: BeanFactory not initialized or already closed - call 'refresh' before accessing beans via the ApplicationContext
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-07-16 13:13:58.332 [ INFO] org.codehaus.groovy.grails.web.context.GrailsContextLoaderListener:152 - Destroying Spring parent WebApplicationContext Root WebApplicationContext
- I ran manually
./update-to-25.sql(do I need/have to run./nsl-ddl.sqlBEFORE./update-to-25.sql) see the log/output/errors bellow ?cd /opt/apache-tomcat-8.5.13/webapps/nxl#services##1.0204/plugins/nsl-domain-plugin-1.10/sql psql -h localhost -p 5432 -U nsldev -d nsl -f ./update-to-25.sql sudo systemctl restart tomcat- After this nsl-services started automatically without reporting any error-s/exception-s (
./update-to-26.sql)
$CATALINA_HOME/.nsl/editor-config.rbneeds to be added:Rails.configuration.nsl_linker = "http://localhost:7070/nsl-mapper/"
- plus the whole installer has to be searched/checked for typo/wrong hostname
icn.dev42.oztaxa.com(it should bedev42-icn.oztaxa.com) - nginx needs HTTP -> HTTPS
BAD
postgres@ip-192-168-1-237:/home/ubuntu/2017-07-30_BNTi-icn-data_load$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
GOOD
This is what it should be (affects date format/interpretation, and psql sorting, etc):
[pmcneil@nerdbrick ~]$ locale
LANG=en_AU.utf8
LC_CTYPE="en_AU.utf8"
LC_NUMERIC="en_AU.utf8"
LC_TIME="en_AU.utf8"
LC_COLLATE="en_AU.utf8"
LC_MONETARY="en_AU.utf8"
LC_MESSAGES="en_AU.utf8"
LC_PAPER="en_AU.utf8"
LC_NAME="en_AU.utf8"
LC_ADDRESS="en_AU.utf8"
LC_TELEPHONE="en_AU.utf8"
LC_MEASUREMENT="en_AU.utf8"
LC_IDENTIFICATION="en_AU.utf8"
LC_ALL=
env details:
tomcat@ip-192-168-1-236:/opt$ ls -lahF
total 9.0M
drwxr-xr-x 4 root root 4.0K Jul 16 11:30 ./
drwxr-xr-x 23 root root 4.0K Jul 16 11:42 ../
drwxr-xr-x 11 tomcat tomcat 4.0K Jul 16 11:59 apache-tomcat-8.5.13/
-rw-r--r-- 1 root root 9.0M Jul 10 15:09 apache-tomcat-8.5.13.tar.gz
lrwxrwxrwx 1 root root 12 Jul 16 11:30 jdk -> jdk1.8.0_171/
drwxr-xr-x 8 root root 4.0K Mar 29 11:18 jdk1.8.0_171/
tomcat@ip-192-168-1-236:/opt$ cat apache-tomcat-8.5.13/bin/setenv.sh
#!/bin/bash
JAVA_HOME=/opt/jdk
JRE_HOME=$JAVA_HOME/jre
export PATH=$JAVA_HOME/bin:$PATH
export SECRET_KEY_BASE="[REMOVED]"
tomcat@ip-192-168-1-236:/opt$ which java
/opt/jdk/bin/java
tomcat@ip-192-168-1-236:/opt$ which javac
/opt/jdk/bin/javac
tomcat@ip-192-168-1-236:/opt$ echo $JAVA_HOME
/opt/jdk
tomcat@ip-192-168-1-236:/opt$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
tomcat@ip-192-168-1-236:/opt$ javac -version
javac 1.8.0_171
postgres@ip-192-168-1-237:/home/ubuntu/2017-07-30_BNTi-icn-data_load$ psql
psql (9.5.13)
Type "help" for help.
postgres=# select '25/03/03 12:54' :: timestamp with time zone;
ERROR: date/time field value out of range: "25/03/03 12:54"
LINE 1: select '25/03/03 12:54' :: timestamp with time zone;
^
HINT: Perhaps you need a different "datestyle" setting.
postgres=# select '03/25/03 12:54' :: timestamp with time zone;
timestamptz
------------------------
2003-03-25 12:54:00+11
(1 row)
postgres=#
This is part is scripted in BNTi-icn_load-data.sh
To import data from a CSV file like tblbiota_20180717.csv
- Make sure you have the appropriate NSL users set up on your postgresql database.
- drop and create your test database (don't drop if you don't have it...)
dropdb dtest && createdb dtest
- create the new schema and default data from the dawr_ddl.sql (note there are some modifications to the nsl.ddl in this file, mainly the shard config and the ID range for the global nsl sequence.) PLEASE DON'T CHANGE THE SEQUENCE RANGE as this makes it easier if we need to detangle data at a later date.
psql -f dawr_ddl.sql -d dtest
- clean up your CSV file by making all the ychryearofpub data integers by removing stray
)and ranges like1882/3and removing trailing spaces (Hint: use a regex to replace*,with,) - copy your cleaned up CSV file to /tmp then run the import:
psql -f dawr_import.sql -d dtest
- Run up your mapper, LDAP and services.
- Log into the services, go to the admin page, click the reconstruct all names. Watch the log output and wait for it
to complete. once finished the services will run name checks via notifications in the background. Since this may take
some time and has no real use here you may just delete the notifications using sql
delete from notification;. - Finally run the tree import sql.
psql -f dawr_tree_import.sql -d dtest
- if you now go to the classifications page and click on
BPCyou should see your tree in draft. You can publish it if you like.
- This installs to a database named
dtestchange that name in the commands above to the database name you wish to use. - you can edit the details in the shard_config table inserted in the dawr_ddl.sql file, but make sure the namespace matches what you have in the import files.
These are runtime erros/exceptions notes, if cofirmed/relevant/etc. we can officially create issues for them in their github repos.
2018-07-18 16:14:29.807 [DEBUG] grails.app.services.au.org.biodiversity.nsl.ConfigService:128 - read config: classification tree key: APC
2018-07-18 16:14:29.848 [DEBUG] grails.app.controllers.au.org.biodiversity.nsl.api.SearchController:128 - doing search
2018-07-18 16:14:29.856 [DEBUG] grails.app.services.au.org.biodiversity.nsl.SearchService:128 - select distinct(n), n.sortName, n.nameRank.sortOrder from Name n where n.instances.size > 0 and (n.nameType.scientific = true) order by n.sortName, n.nameRank.sortOrder asc
2018-07-18 16:14:29.857 [DEBUG] grails.app.services.au.org.biodiversity.nsl.SearchService:128 - {}
2018-07-18 16:14:30.027 [DEBUG] grails.app.services.au.org.biodiversity.nsl.SearchService:128 - query took 167ms
2018-07-18 16:14:30.589 [ERROR] org.codehaus.groovy.grails.web.errors.GrailsExceptionResolver:213 - NoSuchElementException occurred when processing request: [GET] /nxl/services/search
Cannot access first() element from an empty List. Stacktrace follows:
org.codehaus.groovy.grails.web.pages.exceptions.GroovyPagesException: Error processing GroovyPageView: Error executing tag <st:randomName>: Cannot access first() element from an empty List
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at com.brandseye.cors.CorsFilter.doFilter(CorsFilter.java:82)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.codehaus.groovy.grails.web.taglib.exceptions.GrailsTagException: Error executing tag <st:randomName>: Cannot access first() element from an empty List
at gsp_services_searchsearch_gsp$_run_closure2.doCall(gsp_services_searchsearch_gsp.groovy:136)
at gsp_services_searchsearch_gsp.run(gsp_services_searchsearch_gsp.groovy:160)
... 13 more
Caused by: java.util.NoSuchElementException: Cannot access first() element from an empty List
at services.ServiceTagLib$_closure26.doCall(ServiceTagLib.groovy:353)
... 15 more
2018-07-18 16:14:30.609 [DEBUG] grails.app.controllers.au.org.biodiversity.nsl.api.ErrorController:128 - Error controller: error is org.codehaus.groovy.grails.web.pages.exceptions.GroovyPagesException: Error processing GroovyPageView: Error executing tag <st:randomName>:
Cannot access first() element from an empty List
This looks like an BUG/un-finished customization:
- although the source/template of the editor HTML does support a customizable
window.relative_url_rootthrough setting theRails.application.config.relative_url_rootproperty:<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content=""> <meta name="author" content=""> <title> <%= page_title %> </title> <script> window.relative_url_root = "<%= Rails.application.config.relative_url_root %>"; window.NG_TEMPLATE_URL="<%= @ng_template_path %>"; taxoTree = "<%= ShardConfig.classification_tree_key %>"; </script> <% if Rails.env == "development" || params[:debug] == 'true' %> <script> debugSwitch = true; </script> <% else %> <script> debugSwitch = false; </script> <% end %> <%= stylesheet_link_tag 'application', media: 'all', 'data-turbolinks-track' => false %> <%= javascript_include_tag 'application', 'data-turbolinks-track' => false %> <%= csrf_meta_tags %> <%= favicon_link_tag 'favicon.png?v=1', :rel => 'shortcut icon' %> <% if development? %>
- The URLs of CSS, and .js files right bellow are NOT constructed with
window.relative_url_rootresulting in generating HTML source with URLs with diff/inconsistent relative root, for examplehref="/nsl/editor/assets/application-15121e9f759d8c1bbe071d54fc09fe3f.css":<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content=""> <meta name="author" content=""> <title> NSL Editor </title> <script> window.relative_url_root = "/nxl/editor"; window.NG_TEMPLATE_URL=""; taxoTree = "APC"; </script> <script> debugSwitch = false; </script> <link rel="stylesheet" media="all" href="/nsl/editor/assets/application-15121e9f759d8c1bbe071d54fc09fe3f.css" data-turbolinks-track="false" /> <script src="/nsl/editor/assets/application-4614ecd9e3edfc6fd419aa215019264d.js" data-turbolinks-track="false"></script> <meta name="csrf-param" content="authenticity_token" /> <meta name="csrf-token" content="1XAUjxGMEpwZAcklSIimKHhMU7qzGGEGuqwzG9uF/jwWK3UeRC+GQDcGGJRDJoHYrEcquXgfYpF5bCwSRzuoXg==" /> <link rel="shortcut icon" type="image/x-icon" href="/nsl/editor/assets/favicon-2c74ce22698b130a7e669effa4699004.png?v=1" />
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment