Instantly share code, notes, and snippets.

@cetra3 /packages.md Secret
Last active Jun 28, 2018

Embed
What would you like to do?
Order of the Bee Packages info

How we publish packages and generate environments

This is a quick writeup for ootbee to document how we (parashift) publish and manage packages. It's a little different from the standard Alfresco eco-system, but most components are open source or could be if there is interest.

A modest Proposal

If you find this setup useful, I am happy to work towards opening up some of these components for ootbee. I understand that this is quite an opinionated approach and may not gel well with what exists currently. But, if you feel that there could be some use for some of these components, let's work together.

In the beginning

When I first started there was no real automated way to bring up an Alfresco environment with specific module versions in a quick an easy fashion.

You would need to:

  • Install Alfresco
  • Copy Over the amps
  • Stop Alfresco
  • Run apply_amps
  • Hope you haven't forgotten any other steps

This was fraught with mistakes, and time consuming. We needed a quick way to build up an environment with a given configuration.

Salt Stack

We started off by automating the install process using salt-stack as a base.

The formulas we've written for salt-stack allow alfresco to be configured via what's called a pillar file. This is a simple file written in YAML which describes the configuration of a VM/Container/Server.

Over time we have added in different formulas and configurations, including ldap, kerberos, reverse ssl proxies, breaking of server into different roles (seperate hosts for db/solr, etc..) and pretty much every environment configuration we support. These formulas are mostly public, with the open ones published in our gitlab.

The salt formulas can be deployed to either redhat/centos 7 or ubuntu 14.04/16.04 without having to change the pillar configuration. This means you can migrate between OSes easily or upgrade to new OSes without breaking your Alfresco setup.

We also have a docker formula, so you can spool up an instance of Alfresco with whatever modules you want inside a docker container.

There is some support for windows environments, but salt stack is a bit spotty under windows.

There is a readme in the repo which goes into detail, but to describe an environment, you can use the following yaml pillar config:

roles:
  - alfresco

alfresco:
 ss: True
 version: 5.2.f

alfresco_modules:
  - ootbee:support-tools:1.0.0
  - parashift:onlyoffice:1.3.2

This will create a fully fledged alfresco environment with two modules installed, The order of the bee support tools & our onlyoffice module. It will install Community version 5.2.f and Alfresco Search Services (as dictated by ss: True), alongside postgres, libreoffice, etc...

The modules themselves are pulled down from our packages server, including alfresco itself.

The Packages Server

The packages server is simply a REST server written with Spring Boot and stores some info in a Postgres database, alongside all binary content in a simple directory.

We wanted something a little different from a standard maven repo, mainly to support that there are repo and share components, and the standard maven triplet doesn't really address this. That: and we wanted to be able to have paid subscriptions to specific modules, etc...

The structure and layout of the packages server is rather simple:

  • Vendors: the vendor or group that this relates to i.e, parashift or ootbee
  • Modules: the module itself, such as support-tools
  • Versions: the versions of the module, with a flag as to whether they've passed QA or are still dev versions. We tend to use semantic versioning for our modules.
  • Artifacts: Whether there is a share, repo, or other artifact relating to this.

The main API points are for downloading, and verifying checksums of downloaded amps (If you publish an amp with the same version, the checksum will still be compared and updated)

For instance, to get the md5 sum of the onlyoffice module's repo amp:

curl http://localhost:8080/packages/module/parashift/onlyoffice/1.3.2/repo.md5

The URL dictates the different components:

http://localhost:8080/packages/module/{{vendor}}/{{module}}/{{version}}/{{artifact}}

To publish an amp or to download an amp, you need to provide a token, which can be generated by an admin.

An example of publishing:

curl -X POST -H TOKEN:1a2bb234598lkjslwporiuwe -F "file=@solr.war" http://localhost:8080/packages/module/apache/solr/4.10.3/repo

While this is easy, it's also a bit tedious to do so manually, hence why we have invented a few more tools.

A possible front end

A rebuild was taking place of the packages server to give it a nice web interface. The idea was copying things like npm or crates with a bit of a search.

Here are a couple of example screenshots of what that might look like:

The home page, showing recently updated modules, plus instructions on how to get started:

A module page, whereby the README (written in markdown) is displayed inline along with download links:

This frontend was very much a WIP and would need a bit more work to see it to fruition.

Paramp - Parashift Amp Extractor

Paramp is our tool to replace alfresco mmt for constructing environments. There are a few reasons that we invented it, mainly because mmt doesn't handle inter-dependencies well, and doesn't allow you to downgrade versions. Basically it got in the way quite a lot.

One of the (non-documented.. sorry) things about this is the ability to contact the packages server, and pull it down directly. This is what's used in our salt formula: paramp pulls down the amps from the packages server, and then builds out an instance.

It can also check to make sure you have the latest of everything.

I.e, given the following yaml file:

alfresco_modules:
  - ootbee:support-tools:1.0.0
  - parashift:onlyoffice:1.3.1

You can run paramp -c file.yml and get the following output:

Checking versions

Module 'parashift:onlyoffice:1.3.1' can be upgraded to version '1.3.2'

Paste the following into your yaml file:

alfresco_modules:
  - ootbee:support-tools:1.0.0
  - parashift:onlyoffice:1.3.2

Parelease

Manually publishing stuff is time consuming, so we invented a command line tool that would publish amps directly to the packages server (given the right auth token). We invented a tool called parelease that makes it a bit easier to automatically publish stuff.

We have a tool called parelease which is rather simple to use once configured, but does align with our gradle builds, rather than the maven builds.

To publish something, we simply run parelease -g clean amp which will compile and then publish the share and repo subdirectories:

$ parelease -g clean amp

BUILD SUCCESSFUL in 4s
5 actionable tasks: 5 executed

BUILD SUCCESSFUL in 0s
4 actionable tasks: 4 executed

Attempting to submit path: repo/build/amp/onlyoffice-alfresco-repo-1.3.2.amp to: https://example.com/module/parashift/onlyoffice/1.3.2/repo
Submitted! Checksum is: cbfcf7874c6702bedc476d167159bb4e
Attempting to submit path: share/build/amp/onlyoffice-alfresco-share-1.3.2.amp to: https://example.com/module/parashift/onlyoffice/1.3.2/share
Submitted! Checksum is: 9019b9dea9ead4fc8b49fa7267e41502
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment