Skip to content

Instantly share code, notes, and snippets.

@Paraphraser

Paraphraser/IOTstackPullRequest.md

Last active January 28, 2023 17:26
Show Gist options
  • Star 5 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save Paraphraser/818bf54faf5d3b3ed08d16281f32297d to your computer and use it in GitHub Desktop.
Save Paraphraser/818bf54faf5d3b3ed08d16281f32297d to your computer and use it in GitHub Desktop.
Download ZIP
Preparing IOTstack Pull Requests
Raw
IOTstackPullRequest.md

Preparing a Pull Request for IOTstack

If you want to fix a bug or propose an enhancement for IOTstack, you will need to prepare a Pull Request (PR).

Please don't try to do anything in this gist inside an ~/IOTstack folder on a Raspberry Pi that is also being used to run your Docker containers. It's easy to get confused and you could accidentally break your own working IOTstack.

You can do everything on the same Raspberry Pi that is running your IOTstack but it's usually easier to work on a desktop or laptop. If you accept this advice, there's no risk of breaking your running IOTstack.

This guide barely scratches the surface of Pull Requests. There is lots of advice on GitHub and Google will find plenty of hits for any question you might have.

Table of contents

Understand how IOTstack works

Please don't skip this section or skim-read. It's really important.

Structure of a clean clone

Start by making a clean clone of IOTstack:

$ git clone https://github.com/SensorsIot/IOTstack.git CleanIOTstack

Notes:

  • The folder name CleanIOTstack is not important. I'm just making the point that it should not be ~/IOTstack.
  • It doesn't matter what your working directory is when you start. It could be your home directory, your desktop, your documents folder. It's your choice.

Explore the result:

$ ls -A1F CleanIOTstack
.bash_aliases ✅
.git/ ❎
.github/ ❎
.gitignore ❎
.native/ ✅
.templates/ ✅
.tmp/ ❎
LICENSE ❎
README.md ✅
docs/ ✅
duck/ ✅
menu.sh* ✅
mkdocs.yml ✅
scripts/ ✅

Observe what is not there:

  • There is no docker-compose.yml
  • There is no backups directory
  • There is no services directory
  • There is no volumes directory

General principles:

  1. Stay away from anything with a ❎.
  2. The vast majority of Pull Requests will involved changes to the files or folders marked with ✅.
  3. Don't add any new top-level folders without first discussing it with a maintainer.
  4. Do not ever do anything that causes a backups, services or volumes folder, or a docker-compose.yml file to be added to the repository on GitHub.

Service implementation

When you choose a service in menu.sh, this is (conceptually) what happens:

  1. Services are defined in the .templates folder. For example, the definition for Node-Red is in:

    ~/IOTstack/.templates/nodered

  2. menu.sh copies the service's essential components from the .templates folder to a folder of the same name in the services directory. Using Node-Red as the example, the destination folder is:

    ~/IOTstack/services/nodered

    menu.sh creates ~/IOTstack/services when it is needed. Choosing "Pull full service from template" in the menu repeats this copying operation, overwriting whatever was there before.

  3. Each service has a service.yml in its services directory. For Node-Red, that's:

    ~/IOTstack/services/nodered/service.yml

    That file is appended to docker-compose.yml. The first time this happens, menu.sh initialises docker-compose.yml with a header.

  4. If there is a build.sh script in the .templates folder for your service, it is run. Its output is a Dockerfile which winds up in the services directory.

  5. A .templates folder can also have a static Dockerfile. If it does, that is copied to the services folder.

  6. If a directoryfix.sh script exists, it is run. Typical operations include:

    • creating placeholder files and/or folders in the ~/IOTstack/volumes directory (possibly also creating ~/IOTstack/volumes)
    • changing permissions and ownership on files and/or folders in the volumes directory.

After menu.sh finishes its work, you are told to do this:

$ cd ~/IOTstack
$ docker-compose up -d

The "up" tells docker-compose to process docker-compose.yml, pulling down base images as needed.

If a Dockerfile is defined for a service, that is run to build a custom image atop the base image.

docker-compose also auto-creates any missing file system structures defined in volumes statements listed in docker-compose.yml.

docker-compose does not auto-create any missing file system structures for env or devices entries; those cause errors.

Finally, docker-compose instantiates the images as running containers.

Key insights

Once you get your mind around this basic process, it should be self-evident that a clean checkout from GitHub is the factory for all IOTstack installations, while the contents of backups, services, volumes and docker-compose.yml represent each user's individual choices, configuration options and data. It should then be obvious why you must not use a Pull Request to:

  • create the services directory or anything inside it. You must work with the .templates system.
  • create the volumes directory or anything inside it. You must work with the directoryfix.sh system.
  • create the backups directory or anything inside it. This folder is the exclusive domain of backup and restore scripts.
  • shoehorn stuff into containers any old which-way. You must work with either a static Dockerfile or the build.sh system, both of which are in .templates.

If you break these rules and your changes don't get caught by a maintainer before they get into production, you risk stuffing up everyone else's IOTstack.

Preparation (one-time)

GitHub

Create GitHub account

If you don't already have an account on GitHub, create one. Point your browser to:

https://github.com/

and go through the signup process.

You will also find it helpful review the information at:

https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github

The sheer quantity of information on that page may seem daunting and you can get by without doing any of it. You'll just be entering your credentials a fair bit.

Hint: It really is a good idea to set up SSH key-pairs.

Fork the SensorsIot/IOTstack repository

Open your browser at:

https://github.com/SensorsIot/IOTstack

Click the "Fork" button near the top, right of the page and follow your nose. All this does is make an exact copy of SensorsIot/IOTstack. The only real difference is that you have write privileges on your fork.

You only need to fork SensorsIot/IOTstack once. Your fork can be used to prepare as many Pull Requests as you like.

It is possible to delete a fork but GitHub makes it hard to do (lots of "are you sure?"). You should interpret that resistance as saying, "Keep your fork!"

Once GitHub creates your fork, click the big green "Code" button. That opens a popup menu. Click the widget next to the URL. That puts the URL of your fork on the clipboard.

Paste the URL into a text editor window and save it somewhere. You will need it later (see prepare_IOTstack_clone script).

Local computer

This is the bare minimum of what is really a huge scope. I can't promise that this is all that is needed to get the job done. Google is your friend.

Create a global ignores file

Copy the following text to your clipboard:

# General
.DS_Store
Desktop.ini
.AppleDouble
.LSOverride

# Icon must end with two \r
Icon


# Thumbnails
._*
Thumbs.db

# Files that might appear in the root of a volume
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns
.com.apple.timemachine.donotpresent

# Directories potentially created on remote AFP share
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk

# Compiled Python files
*.pyc

# Compiled C++ files
*.out

# Application specific files
venv
node_modules
.sass-cache

# VSCode
.vscode
*.code-workspace

then, using the text editor of your choice, paste it into a file with the following path name:

~/.gitignore_global

Set Git options

See which Git options (if any) are in place already:

$ git config --global --list

If you get an error like this:

fatal: unable to read config file '~/.gitconfig'

then execute these commands, substituting the email address you use to login to GitHub:

$ git config --global user.email 'user@domain'
$ git config --global --list

Make sure you have at least the following in place:

user.name=Your Name
user.email=user@domain
core.excludesfile=~/.gitignore_global
core.pager=less -r
core.editor='bbedit' --wait

You can fix any omissions with these commands:

$ git config --global user.name 'Your Name'
$ git config --global user.email 'user@domain'
$ git config --global core.excludesfile '~/.gitignore_global'
$ git config --global core.pager "less -r"
$ git config --global core.editor "'bbedit' --wait"

Notes:

  • Your Name is associated with commits but is otherwise not significant.
  • user@domain should be how you sign-in to GitHub.
  • You can use any pager you like.
  • The --wait flag on the core.editor is appropriate for an external GUI editor like bbedit. If you want to use a traditional in-terminal editor like vim then omit the --wait flag.

If you have created a key-pair for signing your commits then you can also add:

$ git config --global user.signingkey «fingerprint»

Helper scripts

I use two helper scripts. I urge you to put them into your bin folder or equivalent. Copy the text of each to your clipboard, paste into a text editor window, and save the file with the recommended name.

Don't forget to make the scripts executable, like this:

$ chmod u+x prepare_github_clone_from_fork
$ chmod u+x prepare_IOTstack_clone

prepare_github_clone_from_fork

This script is generic in the sense that it will work for any GitHub repository, not just IOTstack. The script expects three parameters:

  • the URL of your fork on GitHub
  • the URL of the upstream repository on GitHub (the repository that was forked to make the first URL); and
  • the name of a folder on your local computer into which the cloned copy of the fork should be placed.
#!/usr/bin/env bash

if [ "$#" -ne 3 ]; then
	echo "Usage: $(basename "$0") forkURL upstreamURL repo"
	echo "       (repo = eg \"IOTstack\" or \"osx-cpu-temp\""
	exit -1
fi

# $1=forkURL
# $2=upstreamURL
# $3=repo

mkdir -p "$3"
git clone "$1" "$3"
cd "$3"
git remote add -f upstream "$2"
git checkout master

prepare_IOTstack_clone

This script is specific to IOTstack. It takes no parameters and just calls prepare_github_clone_from_fork with appropriate arguments.

In the earlier Fork the SensorsIot/IOTstack repository step, you saved the URL of your fork. You will need that now.

Copy the following text into a text editor:

#!/usr/bin/env bash

prepare_github_clone_from_fork \
	"«URL OF YOUR IOTSTACK FORK»" \
	"https://github.com/SensorsIot/IOTstack.git" \
	"IOTstack"

Replace the «URL OF YOUR IOTSTACK FORK» with the URL you saved before, then save the file into your bin with the name prepare_IOTstack_clone.

Creating a Pull Request

Open a browser window to your fork on GitHub

Always start by opening a browser window pointing to your fork on GitHub.

If you just forked SensorsIot/IOTstack then your browser window will already be open to the correct page.

Check the status line. You are looking for this:

This branch is even with SensorsIot:master.

If you see that message, go onto prepare a commit message.

However, if you see a message indicating that your fork is "behind" SensorsIot/IOTstack, then you will need to synchronise your fork first.

Synchronise your fork with SensorsIot/IOTstack

Any time the upstream changes, it will be "ahead" of your fork. That means your fork will be "behind" and has to be brought up-to-date.

This is the incantation:

$ mkdir sync_my_fork
$ cd sync_my_fork
$ prepare_IOTstack_clone
$ cd IOTstack
$ git merge upstream/master
$ git push
$ cd ../..
$ rm -rf sync_my_fork

Notes:

  • the folder name sync_my_fork is not important. It is just making the point that synchronisation is best done in a separate clone which is thrown away afterwards.
  • the incantation depends on prepare_IOTstack_clone. If you have not yet set up the helper scripts, you should go back and do that.

Once you have executed all those commands, refresh the browser page pointing to your fork. You will find that your fork is "even" with SensorsIot/IOTstack.

The incantation might seem like a good candidate for a script of its own but it is best done one command at a time. In particular, if anything goes wrong with the merge, you don't want to blindly push. You want to resolve the conflicts first.

Prepare a commit message

It might sound like we are getting ahead of ourselves but I'm assuming you've already tested the changes you want to turn into a Pull Request and you're at the point where you want to prepare the Pull Request.

The rules for a commit message are:

  1. A summary line no longer than 72 characters.
  2. A blank line.
  3. As many detail lines as you need, each no longer than 72 characters, MarkDown format assumed.

I don't know how hard-and-fast the 72-character limit is. I just stick to it and I've never had any trouble.

Invent a name for your Pull Request

Pull Requests are implemented via Git branches. The syntax I follow for a branch name is a European-format date followed by one or more key words summarising the purpose of the change, separated by hyphens. For example:

20200926-gitea-kunde21

My PR was proposing switching the gitea container from being based on an older image (called "kapdap") to a newer image called "kunde21". There is no science to this. The name just has to be (a) unique within the scope of your fork, and (b) meaningful to you.

The two main main benefits of starting branch names with yyyymmdd dates are:

  • you are unlikely to collide with any existing branch name; and
  • your Pull Request branches sort in a sensible order.

You don't have to follow my syntax. Pick whatever works for you. You can always see what branches exist by clicking on the "Switch branches or tags" button on your fork's page on GitHub (the button usually says "Master").

Once you've chosen a suitable branch name, define it as a variable like this:

$ BRANCH="20200926-gitea-kunde21"

Clone your fork to your local computer

Incantation:

$ mkdir "$BRANCH"
$ cd "$BRANCH"
$ prepare_IOTstack_clone
$ cd IOTstack

Notes:

  • Don't try to prepare Pull Requests in a clone of IOTstack which is actually running services (ie contains a docker-compose.yml).
  • The upper-level folder does not have to be the name of your branch but using the branch name will help you keep track of what you have done during later steps. You will also see the sense of doing it this way if you ever have multiple Pull Requests on the boil at the same time.
  • It is a good idea to start every Pull Request like this. It guarantees that you have the latest version of everything and reduces the risk that your changes will conflict with something else.

Checkout your branch

If you've only just done the prepare_IOTstack_clone this next command is probably redundant, but it is still good form:

$ git pull upstream master

Then, create and switch to your branch. The branch is where you will actually make your changes.

$ git checkout -b "$BRANCH"

All this does is switch the clone of the fork on your local computer to a branch with the name of your Pull Request. At this point, there are no changes and the branch is identical to the fork (which, in turn, should be identical to SensorsIot/IOTstack).

Make your changes

Here is where you do things like add new folders and files, edit existing files, and otherwise do everything you need to do to implement the change you are proposing.

Don't be tempted to do too much in a single Pull Request. One PR = one purpose. Keep it simple. Remember, other people have to understand your proposal.

Tell Git about your changes

Check what Git thinks:

$ git status

Git reports changes in three groups (along with help text to guide your actions; removed here for the sake of clarity):

Changes to be committed:

	new file:   fred

Changes not staged for commit:

	modified:   jack

Untracked files:

	bill

Your goal is to move everything related to your Pull Request listed in the second and third groups to the first group. You do that with the add command (yes, you "add" changed files):

$ git add «path»

In the above example, you would do:

$ git add jack
$ git add bill

and then the result of a git status would be:

Changes to be committed:

	new file:   fred
	modified:   jack
	new file:   bill

Keep iterating with git status and git add until everything related to your Pull Request is in the "Changes to be committed" group.

The words related to your Pull Request are in bold as a reminder. Don't blindly add everything. Always consider whether what you are adding is directly related to your Pull Request. You don't want unrelated junk appearing on GitHub by mistake.

Commit your changes

You can check what Git is about to do like this:

$ git commit -a --dry-run

If you are set up to digitally-sign your commits, add the -S (upper-case) option to the following command. Otherwise, execute it "as is":

$ git commit -a -s

This is where you will need the commit message you prepared earlier.

It is good practice to check the log to see if there are any obvious problems:

$ git log

The log is a long document. It starts with your changes, and then all the changes that have been made by anyone before you, in reverse chronological order.

Because the log is a long document, Git invokes your chosen pager (see the core.pager setting in Set Git options).

Assuming your pager is either the standard Unix less or more:

  • press the space bar to advance a page at a time; then
  • press q to exit when you have finished your review.

If you set core.pager to a different pager, you will have to know how to navigate and exit.

Tell GitHub about your changes

Synchronise your fork (on GitHub) with the copy on your local computer. You do this by pushing the changes from your computer to GitHub:

$ git push origin "$BRANCH"

Generate the Pull Request

Go back to the browser window that is open on your fork. Either wait a few moments or refresh the page.

Eventually a button will appear inviting you to create a Pull Request. Click and follow your nose.

Congratulations. That's your first IOTstack Pull Request.

Monitoring your Pull Request

Your work is not done, however. You will need to keep an eye on your Pull Request (in the Pull Requests tab of SensorsIot/IOTstack). A discussion may be opened, or changes may be requested by other people, or you might discover something you missed, or conflicts may appear that you will need to resolve.

Updating your Pull Request

Start by going back into the directory within which you prepared your Pull Request Help! I deleted the folder!

$ BRANCH="20200926-gitea-kunde21"
$ cd "$BRANCH/IOTstack"

Make sure your local repository is in sync with GitHub:

$ git pull upstream master
$ git pull origin "$BRANCH"

Then:

Or, to express the last three steps more succinctly:

$ git status
$ git add ...
$ git commit -a -s
$ git push origin "$BRANCH"

where:

  • you iterate the status and add commands until the new changes related to your Pull Request have been added; then
  • commit your changes, remembering to add the -S option if you digitally-sign your commits; then
  • push your changes to GitHub.

There is no need to repeat the Generate the Pull Request step. GitHub knows that the changes are to become part of the Pull Request.

Recreating the Pull Request structure

It's a good idea to keep the Pull Request folder around until GitHub tells you it is safe to delete the branch (see Tidy-up). But, if you don't have the folder handy, just re-create the structure:

$ BRANCH="20200926-gitea-kunde21"
$ mkdir "$BRANCH"
$ cd "$BRANCH"
$ prepare_IOTstack_clone
$ cd IOTstack
$ git checkout "$BRANCH"

All fixed. Now you can go back to Updating your Pull Request.

Tidy-up

Eventually, if the maintainers incorporate your Pull Request, a button will appear near the bottom of the page for your Pull Request on Github, telling you that it is safe to delete the branch that formed the basis of your Pull Request. Once you click that button, the Pull Request cycle is complete.

At that point, it is safe to delete the folder on your local computer. If you have followed these instructions as written, the folder will be the name of your branch so, in this example:

$ rm -rf 20200926-gitea-kunde21

Testing Pending Pull Requests

There is always a delay between when a Pull Request is submitted and when it is merged into the master branch. The maintainers need to find time to consider each request and test it.

However, you may spot a Pull Request that you want to apply to your own instance of IOTstack ahead of the change being applied to the master branch. This is how you do it.

Applying a single Pull Request

Start by going to the page of the pull request you're interested in and figure out its ID. The URL bar is a good place to look:

https://github.com/SensorsIot/IOTstack/pull/168

The ID is 168. You can also see it on the end of the summary line:

Add exclusions for Visual Studio Code to IOTstack git ignores #168

This is what you would do if you you wanted to apply that Pull Request to your own IOTstack instance:

$ PR=168
$ cd ~/IOTstack
$ git checkout master
$ git pull origin master
$ git fetch origin pull/$PR/head:PR${PR}
$ git checkout PR${PR}

Notes on the git fetch command:

  1. The "168" is the ID of the Pull Request.

  2. The "fetch" creates a local branch on your own computer. In this example, it is "PR168"

  3. If you are working in a clone of your own fork of IOTstack (eg created via prepare_IOTstack_clone) then:

    • "origin" is your fork, while
    • "upstream" is SensorsIot/IOTstack,

    so you will need to change the command like this:

    $ git fetch upstream pull/$PR/head:PR${PR}

When you want to revert (eg when PR168 makes it into the master):

$ git checkout master
$ git branch -D PR168

Merging multiple Pull Requests

Let's say you want to apply two pending Pull Requests with the following IDs:

  • 168
  • 131

You can't follow the strategy for a single Pull Request because each branch is distinct. What you need to do is create a "holding" branch and merge the pending pull requests into that branch:

$ PRS="131 168"
$ cd ~/IOTstack
$ git checkout master
$ git pull origin master
$ for PR in $PRS ; do git fetch origin pull/$PR/head:PR${PR} ; done
$ git checkout -b merge-pull-requests 
$ for PR in $PRS ; do git merge -m "local merge branch" PR${PR} ; done

Remember: if you are working in a clone of your IOTstack fork, you will need to change "origin" to "upstream" in the git fetch commands.

It is best to apply pending Pull Requests in ascending order but, even so, you may find that you run into merge conflicts. If you hit a conflict, this command is your friend:

$ git merge --abort

Merge conflicts can sometimes be resolved by changing the order in which you apply the pending Pull Requests but othertimes you need to give Git some help by using one of the following forms:

$ git merge -X theirs -m "local merge branch" PRxxx
$ git merge -X   ours -m "local merge branch" PRxxx

The first form tells Git to prefer the incoming Pull Request (overriding what is already in the merge-pull-requests branch) while the second form tells Git to prefer what is already in the merge-pull-requests branch.

If a Pull Request which is part of your local merge-pull-requests branch gets applied to master, just delete your local branch and re-merge the pull requests which are still pending. For example, assume PR131 has become part of master and you only need PR168:

$ cd ~/IOTstack
$ git branch -D merge-pull-requests
$ git branch -D PR131
$ git checkout master
$ git pull origin master
$ git checkout -b merge-pull-requests 
$ git merge -m "local merge branch" PR168

Of course, if PR168 is the only Pull Request still pending, the last two lines of the above could be simplified to:

$ git checkout PR168
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment