How we publish packages and generate environments
This is a quick writeup for ootbee to document how we (parashift) publish and manage packages. It's a little different from the standard Alfresco eco-system, but most components are open source or could be if there is interest.
A modest Proposal
If you find this setup useful, I am happy to work towards opening up some of these components for ootbee. I understand that this is quite an opinionated approach and may not gel well with what exists currently. But, if you feel that there could be some use for some of these components, let's work together.
In the beginning
When I first started there was no real automated way to bring up an Alfresco environment with specific module versions in a quick an easy fashion.
You would need to:
- Install Alfresco
- Copy Over the amps
- Stop Alfresco
- Run apply_amps
- Hope you haven't forgotten any other steps
This was fraught with mistakes, and time consuming. We needed a quick way to build up an environment with a given configuration.
We started off by automating the install process using salt-stack as a base.
The formulas we've written for salt-stack allow alfresco to be configured via what's called a pillar file. This is a simple file written in YAML which describes the configuration of a VM/Container/Server.
Over time we have added in different formulas and configurations, including ldap, kerberos, reverse ssl proxies, breaking of server into different roles (seperate hosts for db/solr, etc..) and pretty much every environment configuration we support. These formulas are mostly public, with the open ones published in our gitlab.
The salt formulas can be deployed to either redhat/centos
7 or ubuntu
14.04/16.04 without having to change the pillar configuration. This means you can migrate between OSes easily or upgrade to new OSes without breaking your Alfresco setup.
We also have a docker formula, so you can spool up an instance of Alfresco with whatever modules you want inside a docker container.
There is some support for windows environments, but salt stack is a bit spotty under windows.
There is a readme in the repo which goes into detail, but to describe an environment, you can use the following yaml pillar config:
roles: - alfresco alfresco: ss: True version: 5.2.f alfresco_modules: - ootbee:support-tools:1.0.0 - parashift:onlyoffice:1.3.2
This will create a fully fledged alfresco environment with two modules installed, The order of the bee support tools & our onlyoffice module. It will install Community version
5.2.f and Alfresco Search Services (as dictated by
ss: True), alongside postgres, libreoffice, etc...
The modules themselves are pulled down from our packages server, including alfresco itself.
The Packages Server
The packages server is simply a REST server written with Spring Boot and stores some info in a Postgres database, alongside all binary content in a simple directory.
We wanted something a little different from a standard maven repo, mainly to support that there are
share components, and the standard maven triplet doesn't really address this. That: and we wanted to be able to have paid subscriptions to specific modules, etc...
The structure and layout of the packages server is rather simple:
- Vendors: the vendor or group that this relates to i.e,
- Modules: the module itself, such as
- Versions: the versions of the module, with a flag as to whether they've passed QA or are still dev versions. We tend to use semantic versioning for our modules.
- Artifacts: Whether there is a
repo, or other artifact relating to this.
The main API points are for downloading, and verifying checksums of downloaded amps (If you publish an amp with the same version, the checksum will still be compared and updated)
For instance, to get the md5 sum of the onlyoffice module's repo amp:
The URL dictates the different components:
To publish an amp or to download an amp, you need to provide a token, which can be generated by an admin.
An example of publishing:
curl -X POST -H TOKEN:1a2bb234598lkjslwporiuwe -F "firstname.lastname@example.org" http://localhost:8080/packages/module/apache/solr/4.10.3/repo
While this is easy, it's also a bit tedious to do so manually, hence why we have invented a few more tools.
A possible front end
Here are a couple of example screenshots of what that might look like:
The home page, showing recently updated modules, plus instructions on how to get started:
A module page, whereby the README (written in markdown) is displayed inline along with download links:
This frontend was very much a WIP and would need a bit more work to see it to fruition.
Paramp - Parashift Amp Extractor
Paramp is our tool to replace alfresco mmt for constructing environments. There are a few reasons that we invented it, mainly because mmt doesn't handle inter-dependencies well, and doesn't allow you to downgrade versions. Basically it got in the way quite a lot.
One of the (non-documented.. sorry) things about this is the ability to contact the packages server, and pull it down directly. This is what's used in our salt formula: paramp pulls down the amps from the packages server, and then builds out an instance.
It can also check to make sure you have the latest of everything.
I.e, given the following yaml file:
alfresco_modules: - ootbee:support-tools:1.0.0 - parashift:onlyoffice:1.3.1
You can run
paramp -c file.yml and get the following output:
Checking versions Module 'parashift:onlyoffice:1.3.1' can be upgraded to version '1.3.2' Paste the following into your yaml file: alfresco_modules: - ootbee:support-tools:1.0.0 - parashift:onlyoffice:1.3.2
Manually publishing stuff is time consuming, so we invented a command line tool that would publish amps directly to the packages server (given the right auth token). We invented a tool called parelease that makes it a bit easier to automatically publish stuff.
We have a tool called
parelease which is rather simple to use once configured, but does align with our gradle builds, rather than the maven builds.
To publish something, we simply run
parelease -g clean amp which will compile and then publish the
$ parelease -g clean amp BUILD SUCCESSFUL in 4s 5 actionable tasks: 5 executed BUILD SUCCESSFUL in 0s 4 actionable tasks: 4 executed Attempting to submit path: repo/build/amp/onlyoffice-alfresco-repo-1.3.2.amp to: https://example.com/module/parashift/onlyoffice/1.3.2/repo Submitted! Checksum is: cbfcf7874c6702bedc476d167159bb4e Attempting to submit path: share/build/amp/onlyoffice-alfresco-share-1.3.2.amp to: https://example.com/module/parashift/onlyoffice/1.3.2/share Submitted! Checksum is: 9019b9dea9ead4fc8b49fa7267e41502