Preparing a Pull Request for IOTstack
If you want to fix a bug or propose an enhancement for IOTstack, you will need to prepare a Pull Request (PR).
Please don't try to do anything in this gist inside an
~/IOTstack folder on a Raspberry Pi that is also being used to run your Docker containers. It's easy to get confused and you could accidentally break your own working IOTstack.
You can do everything on the same Raspberry Pi that is running your IOTstack but it's usually easier to work on a desktop or laptop. If you accept this advice, there's no risk of breaking your running IOTstack.
This guide barely scratches the surface of Pull Requests. There is lots of advice on GitHub and Google will find plenty of hits for any question you might have.
Table of contents
- Understand how IOTstack works
- Preparation (one-time)
- Creating a Pull Request
- Open a browser window to your fork on GitHub
- Synchronise your fork with SensorsIot/IOTstack
- Prepare a commit message
- Invent a name for your Pull Request
- Clone your fork to your local computer
- Checkout your branch
- Make your changes
- Tell Git about your changes
- Commit your changes
- Tell GitHub about your changes
- Generate the Pull Request
- Monitoring your Pull Request
- Testing Pending Pull Requests
Please don't skip this section or skim-read. It's really important.
Start by making a clean clone of IOTstack:
$ git clone https://github.com/SensorsIot/IOTstack.git CleanIOTstack
- The folder name
CleanIOTstackis not important. I'm just making the point that it should not be
- It doesn't matter what your working directory is when you start. It could be your home directory, your desktop, your documents folder. It's your choice.
Explore the result:
$ ls -A1F CleanIOTstack .bash_aliases ✅ .git/ ❎ .github/ ❎ .gitignore ❎ .native/ ✅ .templates/ ✅ .tmp/ ❎ LICENSE ❎ README.md ✅ docs/ ✅ duck/ ✅ menu.sh* ✅ mkdocs.yml ✅ scripts/ ✅
Observe what is not there:
- There is no
- There is no
- There is no
- There is no
- Stay away from anything with a
- The vast majority of Pull Requests will involved changes to the files or folders marked with
- Don't add any new top-level folders without first discussing it with a maintainer.
- Do not ever do anything that causes a
volumesfolder, or a
docker-compose.ymlfile to be added to the repository on GitHub.
When you choose a service in
menu.sh, this is (conceptually) what happens:
Services are defined in the
.templatesfolder. For example, the definition for Node-Red is in:
menu.shcopies the service's essential components from the
.templatesfolder to a folder of the same name in the
servicesdirectory. Using Node-Red as the example, the destination folder is:
~/IOTstack/serviceswhen it is needed. Choosing "Pull full service from template" in the menu repeats this copying operation, overwriting whatever was there before.
Each service has a
servicesdirectory. For Node-Red, that's:
That file is appended to
docker-compose.yml. The first time this happens,
docker-compose.ymlwith a header.
If there is a
build.shscript in the
.templatesfolder for your service, it is run. Its output is a
Dockerfilewhich winds up in the
.templatesfolder can also have a static
Dockerfile. If it does, that is copied to the
directoryfix.shscript exists, it is run. Typical operations include:
- creating placeholder files and/or folders in the
~/IOTstack/volumesdirectory (possibly also creating
- changing permissions and ownership on files and/or folders in the
- creating placeholder files and/or folders in the
menu.sh finishes its work, you are told to do this:
$ cd ~/IOTstack $ docker-compose up -d
The "up" tells
docker-compose to process
docker-compose.yml, pulling down base images as needed.
Dockerfile is defined for a service, that is run to build a custom image atop the base image.
docker-compose also auto-creates any missing file system structures defined in
volumes statements listed in
docker-composedoes not auto-create any missing file system structures for
devicesentries; those cause errors.
docker-compose instantiates the images as running containers.
Once you get your mind around this basic process, it should be self-evident that a clean checkout from GitHub is the factory for all IOTstack installations, while the contents of
docker-compose.yml represent each user's individual choices, configuration options and data. It should then be obvious why you must not use a Pull Request to:
- create the
servicesdirectory or anything inside it. You must work with the
- create the
volumesdirectory or anything inside it. You must work with the
- create the
backupsdirectory or anything inside it. This folder is the exclusive domain of backup and restore scripts.
- shoehorn stuff into containers any old which-way. You must work with either a static
build.shsystem, both of which are in
If you break these rules and your changes don't get caught by a maintainer before they get into production, you risk stuffing up everyone else's IOTstack.
If you don't already have an account on GitHub, create one. Point your browser to:
and go through the signup process.
You will also find it helpful review the information at:
The sheer quantity of information on that page may seem daunting and you can get by without doing any of it. You'll just be entering your credentials a fair bit.
Hint: It really is a good idea to set up SSH key-pairs.
Open your browser at:
Click the "Fork" button near the top, right of the page and follow your nose. All this does is make an exact copy of SensorsIot/IOTstack. The only real difference is that you have write privileges on your fork.
You only need to fork SensorsIot/IOTstack once. Your fork can be used to prepare as many Pull Requests as you like.
It is possible to delete a fork but GitHub makes it hard to do (lots of "are you sure?"). You should interpret that resistance as saying, "Keep your fork!"
Once GitHub creates your fork, click the big green "Code" button. That opens a popup menu. Click the widget next to the URL. That puts the URL of your fork on the clipboard.
Paste the URL into a text editor window and save it somewhere. You will need it later (see prepare_IOTstack_clone script).
This is the bare minimum of what is really a huge scope. I can't promise that this is all that is needed to get the job done. Google is your friend.
Copy the following text to your clipboard:
# General .DS_Store Desktop.ini .AppleDouble .LSOverride # Icon must end with two \r Icon # Thumbnails ._* Thumbs.db # Files that might appear in the root of a volume .DocumentRevisions-V100 .fseventsd .Spotlight-V100 .TemporaryItems .Trashes .VolumeIcon.icns .com.apple.timemachine.donotpresent # Directories potentially created on remote AFP share .AppleDB .AppleDesktop Network Trash Folder Temporary Items .apdisk # Compiled Python files *.pyc # Compiled C++ files *.out # Application specific files venv node_modules .sass-cache # VSCode .vscode *.code-workspace
then, using the text editor of your choice, paste it into a file with the following path name:
See which Git options (if any) are in place already:
$ git config --global --list
If you get an error like this:
fatal: unable to read config file '~/.gitconfig'
then execute these commands, substituting the email address you use to login to GitHub:
$ git config --global user.email 'user@domain' $ git config --global --list
Make sure you have at least the following in place:
user.name=Your Name user.email=user@domain core.excludesfile=~/.gitignore_global core.pager=less -r core.editor='bbedit' --wait
You can fix any omissions with these commands:
$ git config --global user.name 'Your Name' $ git config --global user.email 'user@domain' $ git config --global core.excludesfile '~/.gitignore_global' $ git config --global core.pager "less -r" $ git config --global core.editor "'bbedit' --wait"
Your Nameis associated with commits but is otherwise not significant.
user@domainshould be how you sign-in to GitHub.
- You can use any pager you like.
--waitflag on the
core.editoris appropriate for an external GUI editor like
bbedit. If you want to use a traditional in-terminal editor like
vimthen omit the
If you have created a key-pair for signing your commits then you can also add:
$ git config --global user.signingkey «fingerprint»
I use two helper scripts. I urge you to put them into your
bin folder or equivalent. Copy the text of each to your clipboard, paste into a text editor window, and save the file with the recommended name.
Don't forget to make the scripts executable, like this:
$ chmod u+x prepare_github_clone_from_fork $ chmod u+x prepare_IOTstack_clone
This script is generic in the sense that it will work for any GitHub repository, not just IOTstack. The script expects three parameters:
- the URL of your fork on GitHub
- the URL of the upstream repository on GitHub (the repository that was forked to make the first URL); and
- the name of a folder on your local computer into which the cloned copy of the fork should be placed.
#!/usr/bin/env bash if [ "$#" -ne 3 ]; then echo "Usage: $(basename "$0") forkURL upstreamURL repo" echo " (repo = eg \"IOTstack\" or \"osx-cpu-temp\"" exit -1 fi # $1=forkURL # $2=upstreamURL # $3=repo mkdir -p "$3" git clone "$1" "$3" cd "$3" git remote add -f upstream "$2" git checkout master
This script is specific to IOTstack. It takes no parameters and just calls
prepare_github_clone_from_fork with appropriate arguments.
In the earlier Fork the SensorsIot/IOTstack repository step, you saved the URL of your fork. You will need that now.
Copy the following text into a text editor:
#!/usr/bin/env bash prepare_github_clone_from_fork \ "«URL OF YOUR IOTSTACK FORK»" \ "https://github.com/SensorsIot/IOTstack.git" \ "IOTstack"
Replace the «URL OF YOUR IOTSTACK FORK» with the URL you saved before, then save the file into your
bin with the name
Always start by opening a browser window pointing to your fork on GitHub.
If you just forked SensorsIot/IOTstack then your browser window will already be open to the correct page.
Check the status line. You are looking for this:
This branch is even with SensorsIot:master.
If you see that message, go onto prepare a commit message.
However, if you see a message indicating that your fork is "behind" SensorsIot/IOTstack, then you will need to synchronise your fork first.
Any time the upstream changes, it will be "ahead" of your fork. That means your fork will be "behind" and has to be brought up-to-date.
This is the incantation:
$ mkdir sync_my_fork $ cd sync_my_fork $ prepare_IOTstack_clone $ cd IOTstack $ git merge upstream/master $ git push $ cd ../.. $ rm -rf sync_my_fork
- the folder name
sync_my_forkis not important. It is just making the point that synchronisation is best done in a separate clone which is thrown away afterwards.
- the incantation depends on
prepare_IOTstack_clone. If you have not yet set up the helper scripts, you should go back and do that.
Once you have executed all those commands, refresh the browser page pointing to your fork. You will find that your fork is "even" with SensorsIot/IOTstack.
The incantation might seem like a good candidate for a script of its own but it is best done one command at a time. In particular, if anything goes wrong with the
merge, you don't want to blindly
push. You want to resolve the conflicts first.
It might sound like we are getting ahead of ourselves but I'm assuming you've already tested the changes you want to turn into a Pull Request and you're at the point where you want to prepare the Pull Request.
The rules for a commit message are:
- A summary line no longer than 72 characters.
- A blank line.
- As many detail lines as you need, each no longer than 72 characters, MarkDown format assumed.
I don't know how hard-and-fast the 72-character limit is. I just stick to it and I've never had any trouble.
Pull Requests are implemented via Git branches. The syntax I follow for a branch name is a European-format date followed by one or more key words summarising the purpose of the change, separated by hyphens. For example:
My PR was proposing switching the
gitea container from being based on an older image (called "kapdap") to a newer image called "kunde21". There is no science to this. The name just has to be (a) unique within the scope of your fork, and (b) meaningful to you.
The two main main benefits of starting branch names with yyyymmdd dates are:
- you are unlikely to collide with any existing branch name; and
- your Pull Request branches sort in a sensible order.
You don't have to follow my syntax. Pick whatever works for you. You can always see what branches exist by clicking on the "Switch branches or tags" button on your fork's page on GitHub (the button usually says "Master").
Once you've chosen a suitable branch name, define it as a variable like this:
$ mkdir "$BRANCH" $ cd "$BRANCH" $ prepare_IOTstack_clone $ cd IOTstack
- Don't try to prepare Pull Requests in a clone of IOTstack which is actually running services (ie contains a
- The upper-level folder does not have to be the name of your branch but using the branch name will help you keep track of what you have done during later steps. You will also see the sense of doing it this way if you ever have multiple Pull Requests on the boil at the same time.
- It is a good idea to start every Pull Request like this. It guarantees that you have the latest version of everything and reduces the risk that your changes will conflict with something else.
If you've only just done the
prepare_IOTstack_clone this next command is probably redundant, but it is still good form:
$ git pull upstream master
Then, create and switch to your branch. The branch is where you will actually make your changes.
$ git checkout -b "$BRANCH"
All this does is switch the clone of the fork on your local computer to a branch with the name of your Pull Request. At this point, there are no changes and the branch is identical to the fork (which, in turn, should be identical to SensorsIot/IOTstack).
Here is where you do things like add new folders and files, edit existing files, and otherwise do everything you need to do to implement the change you are proposing.
Don't be tempted to do too much in a single Pull Request. One PR = one purpose. Keep it simple. Remember, other people have to understand your proposal.
Check what Git thinks:
$ git status
Git reports changes in three groups (along with help text to guide your actions; removed here for the sake of clarity):
Changes to be committed: new file: fred Changes not staged for commit: modified: jack Untracked files: bill
Your goal is to move everything related to your Pull Request listed in the second and third groups to the first group. You do that with the
add command (yes, you "add" changed files):
$ git add «path»
In the above example, you would do:
$ git add jack $ git add bill
and then the result of a
git status would be:
Changes to be committed: new file: fred modified: jack new file: bill
Keep iterating with
git status and
git add until everything related to your Pull Request is in the "Changes to be committed" group.
The words related to your Pull Request are in bold as a reminder. Don't blindly add everything. Always consider whether what you are adding is directly related to your Pull Request. You don't want unrelated junk appearing on GitHub by mistake.
You can check what Git is about to do like this:
$ git commit -a --dry-run
If you are set up to digitally-sign your commits, add the
-S (upper-case) option to the following command. Otherwise, execute it "as is":
$ git commit -a -s
This is where you will need the commit message you prepared earlier.
It is good practice to check the log to see if there are any obvious problems:
$ git log
The log is a long document. It starts with your changes, and then all the changes that have been made by anyone before you, in reverse chronological order.
Because the log is a long document, Git invokes your chosen pager (see the
core.pager setting in Set Git options).
Assuming your pager is either the standard Unix
- press the space bar to advance a page at a time; then
qto exit when you have finished your review.
If you set
core.pagerto a different pager, you will have to know how to navigate and exit.
Synchronise your fork (on GitHub) with the copy on your local computer. You do this by pushing the changes from your computer to GitHub:
$ git push origin "$BRANCH"
Go back to the browser window that is open on your fork. Either wait a few moments or refresh the page.
Eventually a button will appear inviting you to create a Pull Request. Click and follow your nose.
Congratulations. That's your first IOTstack Pull Request.
Your work is not done, however. You will need to keep an eye on your Pull Request (in the Pull Requests tab of SensorsIot/IOTstack). A discussion may be opened, or changes may be requested by other people, or you might discover something you missed, or conflicts may appear that you will need to resolve.
Start by going back into the directory within which you prepared your Pull Request Help! I deleted the folder!
$ BRANCH="20200926-gitea-kunde21" $ cd "$BRANCH/IOTstack"
Make sure your local repository is in sync with GitHub:
$ git pull upstream master $ git pull origin "$BRANCH"
- Prepare a (new) commit message
- Make your changes
- Tell Git about your changes
- Commit your changes
- Tell GitHub about your changes
Or, to express the last three steps more succinctly:
$ git status $ git add ... $ git commit -a -s $ git push origin "$BRANCH"
- you iterate the
addcommands until the new changes related to your Pull Request have been added; then
- commit your changes, remembering to add the
-Soption if you digitally-sign your commits; then
- push your changes to GitHub.
There is no need to repeat the Generate the Pull Request step. GitHub knows that the changes are to become part of the Pull Request.
It's a good idea to keep the Pull Request folder around until GitHub tells you it is safe to delete the branch (see Tidy-up). But, if you don't have the folder handy, just re-create the structure:
$ BRANCH="20200926-gitea-kunde21" $ mkdir "$BRANCH" $ cd "$BRANCH" $ prepare_IOTstack_clone $ cd IOTstack $ git checkout "$BRANCH"
All fixed. Now you can go back to Updating your Pull Request.
Eventually, if the maintainers incorporate your Pull Request, a button will appear near the bottom of the page for your Pull Request on Github, telling you that it is safe to delete the branch that formed the basis of your Pull Request. Once you click that button, the Pull Request cycle is complete.
At that point, it is safe to delete the folder on your local computer. If you have followed these instructions as written, the folder will be the name of your branch so, in this example:
$ rm -rf 20200926-gitea-kunde21
There is always a delay between when a Pull Request is submitted and when it is merged into the master branch. The maintainers need to find time to consider each request and test it.
However, you may spot a Pull Request that you want to apply to your own instance of IOTstack ahead of the change being applied to the master branch. This is how you do it.
Start by going to the page of the pull request you're interested in and figure out its ID. The URL bar is a good place to look:
The ID is 168. You can also see it on the end of the summary line:
Add exclusions for Visual Studio Code to IOTstack git ignores #168
This is what you would do if you you wanted to apply that Pull Request to your own IOTstack instance:
$ cd ~/IOTstack $ git checkout master $ git pull origin master $ git fetch origin pull/168/head:PR168 $ git checkout PR168
Notes on the
git fetch command:
The "168" in "pull/168/head" is the ID of the Pull Request.
The "PR168" and is just a name for a local branch on your own computer and really could be anything.
If you are working in a clone of your own fork of IOTstack (eg created via prepare_IOTstack_clone) then:
- "origin" is your fork, while
- "upstream" is SensorsIot/IOTstack,
so you will need to change the command like this:
$ git fetch upstream pull/168/head:PR168
When you want to revert (eg when PR168 makes it into the master):
$ git checkout master $ git branch -D PR168
Let's say you want to apply two pending Pull Requests with the following IDs:
You can't follow the strategy for a single Pull Request because each branch is distinct. What you need to do is create a "holding" branch and merge the pending pull requests into that branch:
$ cd ~/IOTstack $ git checkout master $ git pull origin master $ git fetch origin pull/131/head:PR131 $ git fetch origin pull/168/head:PR168 $ git checkout -b merge-pull-requests $ git merge -m "local merge branch" PR131 $ git merge -m "local merge branch" PR168
Remember: if you are working in a clone of your IOTstack fork, you will need to change "origin" to "upstream" in the
It is best to apply pending Pull Requests in ascending order but, even so, you may find that you run into merge conflicts. If you hit a conflict, this command is your friend:
$ git merge --abort
Merge conflicts can sometimes be resolved by changing the order in which you apply the pending Pull Requests but othertimes you need to give Git some help by using one of the following forms:
$ git merge -X theirs -m "local merge branch" PRxxx $ git merge -X ours -m "local merge branch" PRxxx
The first form tells Git to prefer the incoming Pull Request (overriding what is already in the
merge-pull-requests branch) while the second form tells Git to prefer what is already in the
If a Pull Request which is part of your local
merge-pull-requests branch gets applied to master, just delete your local branch and re-merge the pull requests which are still pending. For example, assume PR131 has become part of master and you only need PR168:
$ cd ~/IOTstack $ git branch -D merge-pull-requests $ git branch -D PR131 $ git checkout master $ git pull origin master $ git checkout -b merge-pull-requests $ git merge -m "local merge branch" PR168
Of course, if PR168 is the only Pull Request still pending, the last two lines of the above could be simplified to:
$ git checkout PR168