Skip to content

Instantly share code, notes, and snippets.

@m3t
Last active July 18, 2023 13:10
Show Gist options
  • Star 5 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save m3t/df29ec4e0aae167f99c8 to your computer and use it in GitHub Desktop.
Save m3t/df29ec4e0aae167f99c8 to your computer and use it in GitHub Desktop.
Git - Submodules in Travis CI

Supplement to the continuous deployment over SSH with Tavis CI.

git --version
# git version 2.5.3

Example: Hexo with gulp-starter

Goal: Do some postprocessing like minifying with Gulp+Webpack on generated Hexo website before deployment.

gulp-starter is an opinionated boilerplate for web development like Google's Web Starter Kit. To add another task to your gulp workflow, it's worth a look at the official Gulp recipes directory.

Forking repo

Adding fork as submodule

⚠ In this context we use submodules for ourselves only (and Travis CI as consumer). Mastering submodules in a collaborative project takes much more in depth knowlegde to avoid serious damage! ⚠

$ git submodule add

Since the URL in the .gitmodules file is what Travis CI will try to fetch from, make sure to use a URL it can access (NOT SSH).

git submodule add --branch hexo https://github.com/GH_USER/gulp-starter.git gulp-starter
# Cloning into 'gulp-starter'...
git status
# On branch master
# Your branch is up-to-date with 'origin/master'.
# 
# Changes to be committed:
#  (use "git reset HEAD <file>..." to unstage)
#
#	new file:   .gitmodules
#	new file:   gulp-starter

git diff --cached --submodule

diff --git a/.gitmodules b/.gitmodules
new file mode 100644
index 0000000..a3732db
--- /dev/null
+++ b/.gitmodules
@@ -0,0 +1,4 @@
+[submodule "gulp-starter"]
+       path = gulp-starter
+       url = https://github.com/GH_USER/gulp-starter.git
+       branch = hexo
Submodule gulp-starter 0000000...d331ffc (new submodule)

Although gulp-starter is a subdirectory in your working directory, Git sees it as a submodule and doesn’t track its contents when you’re not in that directory. Submodules are composed from a so-called gitlink tree entry in the main repository that refers to a particular commit object within the inner repository that is completely separate.

This means, when Travis CI init and update the repository, the submodule path 'gulp-starter' is checked out to that particular commit. But with update --remote the latest commit from the remote's branch will be checked out.

submodule.<name>.branch

If the submodule is intended to track a branch other than master, you set it in your .gitmodules file:

git config -f .gitmodules submodule.gulp-starter.branch <branch>

In the first place, on the checkout to submodule's SHA-1 it's HEAD is detached. You can checkout a branch in all active submodules at once, of course.

# http://stackoverflow.com/a/18799234
git submodule foreach -q --recursive \
    'branch="$(git config -f $toplevel/.gitmodules submodule.$name.branch)"; \
     [ -z "$branch" ] && \
     git checkout master || git checkout $branch'

$ git submodule update --remote

If submodule's repository has been modified, but superproject's gitlink hasn't pointed to that new SHA-1, Travis CI will still checkout submodule's new commit in case --remote will be used to update.

SHA-1 will be prefixed with "-" if the submodule is not initialized

git submodule status
# -9f4094cb7a9dba2eff72f30a9a3be8c6602de86e gulp-starter
  1. Submodule’s remote repository is being fetched
  2. Submodule's HEAD refers to the latest commit from the branch submodule.<name>.branch in detached state
git submodule update --init --remote --recursive
# Submodule 'gulp-starter' (https://github.com/GH_USER/gulp-starter.git) registered for path 'gulp-starter'
# Cloning into 'gulp-starter'...
# ...
# Submodule path 'gulp-starter': checked out '99d13f81d574dee80df94500bff645452b05b79c'

SHA-1 will be prefixed with "+" if the currently checked out submodule commit does not match the SHA-1 found in the index of the containing repository (superproject)

git submodule status
# +99d13f81d574dee80df94500bff645452b05b79c gulp-starter

gulp-starter/

git remote add upstream https://github.com/vigetlabs/gulp-starter.git

 Useful side note referring to github-as-GH_USER in the following SSH-URL

git remote set-url --push origin git@github-as-GH_USER:GH_USER/gulp-starter.git
git remote set-url --push upstream no-pushing
git remote -v
# origin	https://github.com/GH_USER/gulp-starter.git (fetch)
# origin	git@github-as-GH_USER:GH_USER/gulp-starter.git (push)
# upstream	https://github.com/vigetlabs/gulp-starter (fetch)
# upstream	no-pushing (push)

Tracking a branch

git pull # almost equal to "update --remote" in superproject
git checkout hexo
git status
# On branch hexo

Now you're able to bring your fork's branch into sync with the upstream repository on demand, without losing your local changes.

git fetch upstream master
git merge upstream/master

Integrating submodule's content

⚠ Utterly opinionated, grossly incomplete ⚠

ln -s gulp-starter/gulpfile.js gulpfile.js

gulpfile.js/

 More detailed information about modifcation

gulpfile.js/tasks/

Disable unused tasks

Accordingly less npm-packages are needed and this leads to less build-time in Travis CI (due to caching node_modules, this is not of importance).

for f in *.js; do mv -- "$f" "${f%.js}.js.off"; done
cd iconFont
for f in *.js; do mv -- "$f" "${f%.js}.js.off"; done

Then enable some tasks again.

mv production.js.off production.js
mv html.js.off html.js
mv images.js.off images.js
mv css.js.off css.js
mv webpackProduction.js.off webpackProduction.js

themes/landscape/source/

mkdir javascripts
touch page1.js page2.js
mv *.js javascripts/
page1.js
require('../fancybox/jquery.fancybox.pack.js');
require('../js/script.js');

themes/landscape/layout/_partial

cp after-footer.ejs after-footer.ejs.off
after-footer.ejs diff
--- after-footer.ejs.off
+++ after-footer.ejs
@@ -18,7 +18,6 @@
 
 <% if (theme.fancybox){ %>
   <%- css('fancybox/jquery.fancybox') %>
-  <%- js('fancybox/jquery.fancybox.pack') %>
+  <%- js('javascripts/page1') %>
 <% } %>
 
-<%- js('js/script') %>

Updating GitHub in the right order

"Because lifecycles are separate, updating a submodule inside its container project (NB superproject) requires two commits and two pushes." - Christophe Porteneuve

  1. Submodule (fork of gulp-starter): Commit and push gulp workflow
  2. Main repo/Superproject (Hexo): Commit and push (triggers Travis CI) i.a. new referenced SHA1 (created in 1.) of the submodule

Troubleshooting

Removing a submodule permanently

"Removing a submodule requires several commands and tweaks, some of which are manual and unassisted." - Christophe Porteneuve

Unregister: Remove submodule from .git/config together with it's directory, which remains empty.

git submodule deinit path/to/submodule

Clean reference in .gitmodules and working directory

git rm path/to/submodule

Reusing a submodule's directory after deletion

error: A git directory for 'submodule.<name>' is found locally with remote(s)

  1.  Remove a submodule permanently

  2. "(..) submodules these days keep all their Git data in the top project’s .git directory, so (..) destroying a submodule directory won’t lose any commits or branches that you had." - Chapter 7.11 of Pro Git book.

    rm -rf .git/modules/path/to/submodule

Footnotes

  • 1: Even if the HostName and User are identical for different SSH-connections, they can still be distinguished in ~/.ssh/config:

    "HostName specifies the real host name to log into. This can be used to specify nicknames or abbreviations for hosts."

    "Any identities represented by the authentication agent will be used for authentication unless IdentitiesOnly is set." - ssh_config(5) manual page


Creative Commons License
This work by m3t (96bd6c8bb869fe632b3650fb7156c797ef8c2a055d31dde634565f3edda485ba) <mlt [at] posteo [dot] de> is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

# Created by "git submodule init" / "git submodule update --init"
[submodule "gulp-starter"]
url = https://github.com/GH_USER/gulp-starter.git
update = checkout
# https://git-scm.com/docs/gitmodules
# https://git-scm.com/docs/git-config
[submodule "gulp-starter"]
path = gulp-starter
# http://docs.travis-ci.com/user/customizing-the-build/#Use-Public-URLs-For-Submodules
# Alternative solution: https://gist.github.com/iedemam/9830045
url = https://github.com/GH_USER/gulp-starter.git
# Prefer "rebase" for collaborative projects
# https://medium.com/@porteneuve/getting-solid-at-git-rebase-vs-merge-4fa1a48c53aa
update = checkout # default: checkout
# Checkout the commit recorded in the superproject on a detached HEAD in the submodule.
# Used by update --remote
branch = hexo # default: master
# When not to show a submodule as modified
ignore = dirty # default: none
{
"name": "hexo-site",
"version": "0.0.0",
"private": true,
"hexo": {
"version": "3.1.1"
},
"scripts": {
"build": "hexo generate && gulp production"
},
"dependencies": {
"hexo": "^3.1.0",
"hexo-deployer-git": "0.0.4",
...
},
"devDependencies": {
"babel-core": "^5.8.24",
"babel-loader": "^5.3.2",
"del": "^2.0.2",
"fs": "0.0.2",
"gulp": "^3.9.0",
"gulp-autoprefixer": "^3.1.0",
"gulp-changed": "^1.3.0",
"gulp-data": "^1.2.0",
"gulp-filter": "^3.0.1",
"gulp-htmlmin": "^1.1.4",
"gulp-if": "^2.0.0",
"gulp-imagemin": "^2.3.0",
"gulp-minify-css": "^1.2.1",
"gulp-notify": "^2.2.0",
"gulp-rev": "^6.0.1",
"gulp-rev-napkin": "^0.1.0",
"gulp-rev-replace": "^0.4.2",
"gulp-sequence": "^0.4.1",
"gulp-sizereport": "^1.1.3",
"gulp-uglify": "^1.4.0",
"gulp-util": "^3.0.6",
"lodash": "^3.10.1",
"path": "^0.12.7",
"require-dir": "^0.3.0",
"webpack": "^1.12.1",
"webpack-glob-entries": "^1.0.1"
}
}
Host github-as-GH_USER
HostName github.com
User git
IdentityFile /home/USER/.ssh/id_rsa_GH_USER
IdentitiesOnly yes
# http://docs.travis-ci.com/user/customizing-the-build/#Git-Submodules
git:
submodules: false
before_install:
# https://gist.github.com/iedemam/9830045
# - sed -i 's/git@github.com:/git:\/\/github.com\//' .gitmodules
#
# https://git-scm.com/docs/git-submodule#_options:
# --remote
# Instead of using the superproject’s recorded SHA-1 to update the submodule,
# use the status of the submodule’s remote-tracking (branch.<name>.remote) branch (submodule.<name>.branch).
# --recursive
# https://github.com/travis-ci/travis-ci/issues/4099
- git submodule update --init --remote --recursive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment