Barnard College ISLE docker-compose.yml - alpha version, in dev - Example migration of existing islandora stack -> docker stack environment.

docker-compose.yml

version: '2'
# ISLE Alpha 2, for Development and Testing.
# Usernames and passwords to login to tomcat on services fedora and solr is: admin,ild_tc_adm_2018 - this is specified in the default tomcat-users.xml and can be replaced if you desire (SHOULD BE IN PROD)
services:

  # PHPMyAdmin is a MySQL administration GUI.  It's helpful for me!
  # Connect to this container @ http://0.0.0.0:8081 - the sql host is "mysql" and user is "root" with password set in the mysql service ENV.
  myadmin:
    image: phpmyadmin/phpmyadmin:latest
    container_name: phpmyadmin
    environment:
      - PMA_ARBITRARY=1
    networks:
      - int_network
    ports: 
      - 8081:80
    volumes:
      - /sessions
    depends_on:
      - mysql

  mysql:
    image: islandoracollabgroup/isle-mysql:alpha2
    container_name: isle-a2-mysql
    hostname: mysql
    environment:
      - MYSQL_ROOT_PASSWORD=barnard123
    networks:
      - int_network
    ports:
      - "3306:3306"
    volumes:
      - ./bcol/mysql/data:/sql_databases_for_import # I placed my sql.gz files in here and used phpmyadmin or mysql -uroot -p<MYSQL_ROOT_PASSWORD> <database> < drupal.sql
      # A note about the line above.  Once your data is imported it is placed into the volume called "db_data" described below. 
      - db_data:/var/lib/mysql
      # The above data volume WILL persist accross images - and you are NOT required to repopulate your DBs if you stop or rm the container.
      - ./bcol/mysql/config/my.cnf:/etc/alternatives/my.cnf:ro

  fedora:
    image: islandoracollabgroup/isle-fedora:alpha2
    container_name: isle-a2-fedora
    hostname: fedora
    networks:
      - int_network
    ports:
      - "8080:8080"
      - "8777:80"
    tty: true
    depends_on:
      - mysql
      - solr
    volumes:
    - /mnt/barnard/islandora/fedora/data:/usr/local/fedora/data # Barnard's Fedora data folder and all subfolders from bare metal instance.
    - ./bcol/fedora/config/fedora.fcfg:/usr/local/fedora/server/config/fedora.fcfg # Make sure your jbdc mysql string is correct, username, and password.
    - ./bcol/fedora/config/fedora-users.xml:/usr/local/fedora/server/config/fedora-users.xml # Our Fedora Users
    - ./bcol/fedora/config/filter-drupal.xml:/usr/local/fedora/server/config/filter-drupal.xml # hostname for mysql server is "mysql"
    - ./bcol/fedoragsearch/config/fgsconfigFinal/fedoragsearch.properties:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/fedoragsearch.properties
    - ./bcol/fedoragsearch/config/fgsconfigFinal/fgsconfigObjects.properties:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/fgsconfigObjects.properties
    - ./bcol/fedoragsearch/config/fgsconfigFinal/index/FgsIndex/index.properties:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/index.properties
    - ./bcol/fedoragsearch/config/fgsconfigFinal/repository/FgsRepos/repository.properties:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/repository/FgsRepos/repository.properties
    

  solr:
    image: islandoracollabgroup/isle-solr:alpha2
    container_name: isle-a2-solr
    hostname: solr
    networks:
      - int_network
    ports:
      - "8091:8080"
      - "8983:8983"
    tty: true
    depends_on:
      - mysql 
    volumes:
    - ./bcol/solr/data/collection1/data:/usr/local/solr/collection1/data  # A COMPLETE OVERWRITE WITH BC'S DATA
    - ./bcol/solr/data/collection1/conf/schema.xml:/usr/local/solr/collection1/conf/schema.xml  # A COMPLETE OVERWRITE WITH BC'S DATA


  apache:
    image: islandoracollabgroup/isle-apache:alpha2
    container_name: isle-a2-apache
    hostname: digitalcollections
    networks:
      - int_network
#       - ext_network
    ports:
      - "80:80"
    tty: true
    depends_on:
      - mysql
      - fedora
      - solr
    volumes:
      - ./bcol/httpd/data:/var/www/html

# Defined networks
networks:
  int_network:
  ext_network:

# Defined and saved
volumes:
  db_data:

STEPS:

Gathering info:

from webapps\fedoragsearch\WEB-INF\classes\fgsconfigFinal\

fedoragsearch.soapUser				= fedoraAdmin
fedoragsearch.soapPass				= XXXXXXXXX

TODO: I will need to create this user in the fedora-users.xml for my data to work because it is unique. Nevermind - I will map my entire fedora-users.xml and that will deprecate this need.

...continued from webapps\fedoragsearch\WEB-INF\classes\fgsconfigFinal:

fgsindex.indexDir				= /usr/local/fedora/solr/collection1/data/index

hello! this is stating a folder in solr needs to be accessible to FEDORA will need to copy that data to a new home and change the config value!
TODO: I will need to copy this content or review what needs to be changed.

===

fedora\server\config\fedora.fcfg

 <param name="jdbcURL" value="jdbc:mysql://localhost/fedora3?useUnicode=true&amp;amp;characterEncoding=UTF-8&amp;amp;autoReconnect=true">
      <comment>The JDBC connection URL.</comment>
 </param>

TODO: update the config to point from "localhost" to "mysql" in the JDBC.

...continuing from from fedora\server\config\fedora.fcfg:

 <param name="dbUsername" value="XXXXXXXXX">
 </param>
 <param name="dbPassword" value="XXXXXXXXX">
 </param>

TODO: Create these users in MySQL. Copy in old FEDORA db. (read on)

fedora\server\config\filter-drupal.xml

<connection server="localhost" port="3306" dbname="drupal_digital" user="XXXXXXXXX" password="XXXXXXXXX">

TODO: update the config to point from "localhost" to "mysql" in the connection.

DRUPAL SITE MIGRATION:

Drupal MySQL settings from sites/default/settings.php - easiest but there are other sources (e.g., filter-drupal.xml)
TODO: Create those users in MySQL. Copy in old SITE db. (read on)

==

HUGE myssql transactions failed so:

attempted:
increased in my.cnf:

	innodb_buffer_pool_size = 256M
	innodb_log_buffer_size = 10M
	innodb_log_file_size = 128M

STILL failed - when I looked at our production instance my.conf: over 4GB total was dedicated to SQL! Can't do that in the alpha - so instead I installed PHPMYADMIN and imported my tables that way. +1 to docker for the flexibility of adding phpmyadmin to the stack easily!

Returning back to this Sunday with great hope to finish Fedora.

todo:

• snapshot Barnard dataset (zfs snapshot grandpool/backupd-edu.barnard.digitalcollections@`date -Iminutes` )
• working via vboxsf to mount to Docker host (which is running in Virtualbox) avoid uid/gid rw conflicts and proactively chmod 777 -R grandpool/backupd-edu.barnard.digitalcollections. Recall we took a snapshot for a reason, all in.
• look through our existing configuration files looking for usernames, passwords, hostnames, file/folder locations, etc. Note users/passwords. Change HOSTNAMES to point to appropriate CONTAINER_NAMES in the ISLE stack. Our files will replaces the defaults in the base image.
• Drop existing containers, and possibly even pull the images again (no need if there is no change upstream - CHECK ON THIS BEN)
  • Who forgot to check upstream?
• Verify that the fgsfinalconfig files and folders are not so fragile that moving it to a new instance renders them broken.

Fedora operational
usr/local/fedora/server/config# vim fedora.fcfg
search 'resourceIndex' - check that path and set it if in err.
<param name="path" value="usr/local/fedora/data/resourceIndex" isFilePath="true"> <comment>The local path to the main triplestore directory.</comment> </param>
Here I set it to an absolute path (NOTE THIS ISNT AN ABSOLUTE PATH IN NORMAL PARLANCE, caught myself up on this: i.e.: not /usr/local... just usr/local/...) because I knew where my data was - @todo: review this and determine the appropriate original RI path! Comply with that if possible for parity.

Fedora is now running. RI index R/W error resolved (nothing to do with VBOXSF as initially thought - yay!)

This is more to do with mounting the appropriate folders and persisting volumes. Ignore this and follow the updated gist.

SQL operational

• Import DB's and users.

Tomcat(s) operational
Did not change a thing on the Solr box save for mounting my collection.
Replaced fedora.fcfg with Docker-ready version. See area dedicated to "Fedora Operation"
Replaced tomcat-users and fedora-users.xml with versions to reflect Barnard's environment.

Fedora Operation
Fedora working and serving files.

• See this gist's comment above about checking path to RI - this was the short delay in starting Fedora.
  (Note: the logs were helpful, it reports "cannot open for read/write" and it meant more like "RI doesn't exist at specified location" in my situation (because I'm migrating).

FGS

• Last hold out. Haven't looked at this yet.

Solr operational

• error in Apache in Solr. ServerAlias - I recall this was resolved by @g7morris and will be in next release.
• check and triple checked. Is Solr like... the best thing ever? No serious changes made - I don't even think... no, didn't change any other settings upon review.

Apache Operational
nominal:

• website loads - no muss mount orig /var/www to /var/www in container
• Fedora talks and serves - complete.
• Solr searches - nice!

Service names in Drupal configs after migration:

fedora base url: http://isle-fedora:8080/fedora [admin/islandora/configure]
solr [base] url: http://isle-solr:8080/solr [admin/islandora/search/islandora_solr]

• more to come.

FGS Dedicated Comment

Updated paths to any XSLTs that may have moved from DGI's default (/usr/local/fedora/tomcat to /usr/local/tomcat/. Else consider ln -s

• fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/foxmlToSolr.xslt

  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/DC_to_solr.xslt"/>
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/QDC_to_solr.xslt"/>
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/RELS-EXT_to_solr.xslt"/>
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/RELS-INT_to_solr.xslt"/>
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/FOXML_properties_to_solr.xslt"/>
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/datastream_info_to_solr.xslt"/>
  <!--<xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/MODS_to_solr.xslt"/>-->
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/slurp_all_MODS_to_solr.xslt"/>
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/slurp_all_ead_to_solr.xslt"/>
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/EACCPF_to_solr.xslt"/>
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/TEI_to_solr.xslt"/>
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/text_to_solr.xslt"/>
  <!--<xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/XML_to_one_solr_field.xslt"/>-->
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/XML_text_nodes_to_solr.xslt"/>
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/MADS_to_solr.xslt"/>
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/WORKFLOW_to_solr.xslt"/>
  <xsl:include href="/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/classes/fgsconfigFinal/index/FgsIndex/islandora_transforms/slurp_all_chemicalML_to_solr.xslt"/>

CHOWN of mounted www to www-data:islandora