Skip to content

Instantly share code, notes, and snippets.

@tallguyjenks
Last active August 22, 2023 19:55
Show Gist options
  • Star 56 You must be signed in to star a gist
  • Fork 8 You must be signed in to fork a gist
  • Save tallguyjenks/ca3339b8b5353159f631836268e3f791 to your computer and use it in GitHub Desktop.
Save tallguyjenks/ca3339b8b5353159f631836268e3f791 to your computer and use it in GitHub Desktop.
ZettelKasten Sync Code
# To permanently cache the credentials
git config --global credential.helper store
# To ignore files that could cause issues across different workspaces
touch .gitignore
echo ".obsidian/cache
.trash/
.DS_Store" > .gitignore
# Making out local ZettelKasten into a local Git Repository
git init
git add .
git commit -m "init"
# Pushing our local repository into our remote repository on GitHub
git remote add origin https://github.com/USER/REPONAME.git
git push -u origin master
# Making a new script to automate our repo management
touch zk_sync
chmod +x zk_sync
# -e: edit your crontab file i.e. your list of cronjobs
crontab -e
# My Cron Job:
# */30 * * * * /Users/bryanjenks/.local/bin/zk_sync >/dev/null 2>&1
#!/usr/bin/env sh
# ^^^^^^^^^^^^^^^ This says find the first instance of a sh (shell)
# binary and use that shell to execute these commands.
# There is little to no complexity here and no bashisms so it
# should work just fine on most systems and instances of shells
# (bash, zsh, sh, etc.)
ZK_PATH="PATH TO YOUR VAULT"
# ^^^^^^^^^^^^^^^^^^^^^^^^^^ We are assigning the variable `ZK_PATH`
# with the (maybe) long string to our vault's location (mine is super
# long so this makes the final command look cleaner,
# it's unnecessary if you care)
cd "$ZK_PATH"
# ^^^^^^^^^^^ cd: Change Directory to your vault's location
git pull
# ^^^^^^ So if any changes occurred remotely or on another machine
# your local machine knows to pull those changes down instead of
# having to wait for a local change to run the script
CHANGES_EXIST="$(git status --porcelain | wc -l)"
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ we are assigning
# a value to the variable `CHANGES_EXIST`, the value is the output
# of `git add --porcelain` which outputs a simple list of just the
# changed files and then the output is piped into the `wc` utility
# which is "word count" but with the `-l` flag it will count lines.
# basically, it says how many total files have been modified.
# if there are no changes the output is 0
if [ "$CHANGES_EXIST" -eq 0 ]; then
exit 0
fi
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The whole if block is saying
# in plain english: if there are no changes (CHANGES_EXIST = 0)
# then exit with no error code `exit 0` if there are changes,
# then continue on with the script
git pull
# ^^^^^^ git pull: this will look at your repo and say "any changes?"
# if there are they will be brought down and applied to your local machine
# In the context of a team environment, a more robust approach is needed
# as this workflow doesnt factor in branches, merge conflicts, etc
# but if you leave your home machine, do work on the work machine,
# push to the remote repo before you return to the home machine, then
# you can just get the latest changes applied to the home machine and
# continue on like normal
git add .
# ^^^^^^^ git add. = add all current changes in the repo no
# matter the level of nested folders/files
git commit -q -m "Last Sync: $(date +"%Y-%m-%d %H:%M:%S")"
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# git commit -q -m: this says we are committing changes to
# our repo, -q says BE QUIET no output prints to terminal
# if ran manually, -m defines a message for the commit log
# the -m message is "Last Sync: $(date +"%Y-%m-%d %H:%M:%S")" this
# runs the command date with the formatting arguments for a
# date in YYYY-MM-DD HH-MM-SS format as your commit message
git push -q
# ^^^^^^^^^ git push -q: push the changes to github and
# BE QUIET about it The semicolons between commands are
# just saying run each command and then run the subsequent
# command, they're just separators
@Joshfairhead
Copy link

Joshfairhead commented Aug 9, 2021

Hmm, I followed your instructions to the best of my abilities but I'm afraid the automation isn't working as expected. I've set up git and sync'd manually without issue. I've added your script in /usr/local/bin and added it as a cronjobs. Running the script manually updates github when a new note is created, but not when new detail is added to an existing note (I get "Already up to date" as a message). If I remove the new note and try to run the script again I get the same error message.

Further to that, it also doesn't seem like the cron job is working. I've set the update to every minute for the sake of testing and nothing seems to happen.

Any idea what might be happening or going wrong?

@tallguyjenks
Copy link
Author

@Joshfairhead are you on MacOS? if so, there's a little work you need to do to make cron work normally. there should be a link on my git-sync workflow write up on medium to resolving the MacOS issue. if you're not on MacOS there might be another issue and it would be best to open a discussion on my FAQ repo

@Joshfairhead
Copy link

Joshfairhead commented Aug 12, 2021

Thanks,

I am on MacOS but unfortunately the Cron trick you mentioned doesn't seem to be the issue - cron is working as its time stamping commits - its just not pushing for some reason. Sorry to post here, I cant seem to find your FAQ repo to follow up in the appropriate place; I've searched your github profile (and am aware that we're on gist)!

In the meanwhile I've been methodically debugging. My shell is zsh and I have powerbars installed so I can see the git status of a given folder. When the power bar is yellow, it means uncommitted changes. Green means everything is committed.

  • When run manually the script successfully pushes to github provided that changes are not committed (yellow).
  • When run manually the script fails to push to github when changes are already committed (green).
  • When using cron to run the script, it fails to push to github on its own steam; however, when chron runs the script it does commit changes with the a time stamp in the specified format - checked via git log. Regardless it fails to push.

My reasoning is thus:

  • The push command in the script is failing... but I don't know why.
  • I removed the -q flag as its unfamiliar to me but the issue persists.

Can you think of a reason the scripts push command may not work when changes are committed?

@tallguyjenks
Copy link
Author

FAQ: https://github.com/BryanJenksCommunity/FAQ/discussions

The reason it might be failing is the issue with macOS and cron, you need to add the cron binary to a location or do something with giving it admin level permissions so that it can run git push. this is the same issue I had early on. MacOS doesn't favor cron so it takes some finagling to get it working right. there should be an article linked on the medium write up I made that covers this

@Joshfairhead
Copy link

Thanks again Bryan,

I've been through the article as recommended previously and Cron does have escalated status as recommended. Perhaps I need to add the git binary there as well?

How should I move this thread to the community to retain context?

@tallguyjenks
Copy link
Author

yeah go ahead and move it :)

@dd1512
Copy link

dd1512 commented Dec 27, 2021

Hi guys, I'd had the exact issue as Josh's and tried out everything in the medium post but still did not work.
Then I output the cron log to a file and it seemed to complain that it failed to access to github:
fatal: could not read Username for 'https://github.com': Device not configured (even the script worked just fine when running it manually).

I followed the instructions in https://stackoverflow.com/questions/40274484/fatal-could-not-read-username-for-https-github-com-device-not-configured and move from using https to ssh by using git remote set-url --delete origin [https link] and git remote set-url --add origin [ssh link] and it did it.

However, this was just my error. Josh could have been having another error altogether but just posting here in case somebody has the same error.

@jsc-smith
Copy link

Love the idea of this and was struggling a little with the MacOS side of things until following the mentioned instructions for file permissions. However, it is still not working 100% as expected, though strangely it is when I run the script without it running through the crontab.

When the crontab version runs (script as above) it runs all the commands except for the final gitpush, so it checks if there are any changes and then commits them correctly but refuses to push. If I run the script in isolation it will also push them up to git.

Any ideas?

@tallguyjenks
Copy link
Author

@jsc-smith thats probably because crontab doesnt have the permissions to run a push because you need to take the crontab binary and add give it permissions like i said in my article if you're having problems on macOS with cron look at this article

@jsc-smith
Copy link

@tallguyjenks appreciate you getting back to me so quickly!

Yeah I managed to get most of the crontab functionality in order by following the article you linked (full-disk access) whereby I have added cron, smbd, terminal and even crontab from usr/bin ... The script runs automatically thanks to the cron but it just doesn't seem to run the final push command ... So I'm not sure why it is just this command?

I know that there are 3rd party plugins on Obsidian to achieve the sync, but it's been an interesting and a learning experience doing and understanding how this way.

@Joshfairhead
Copy link

I resolved my push issues a little while back, it was to do with auth on Githubs side but I forget the exact details. I was manually able to push in terminal I was missing some credentials that stopped cron from doing it. I recommend playing around and configuring the Github settings, at some point it'll pop you out to a browser window to auth and from there everything works fine!

@grempe
Copy link

grempe commented May 22, 2022

For those finding this. A cleaner and supported way to do this is to use launchd on macOS instead of cron. In my case this eliminated the issues I was having with git push when operating in the cron shell environment.

The script provided above remains identical but remove the entry you added in crontab -e and instead follow the instructions here to create your own launchd service.

https://ellismin.com/2020/03/launchd-1/

https://rakhesh.com/mac/macos-launchctl-commands/

Here is mine, stored at ~/Library/LaunchAgents/us.rempe.obsidian.plist. You'll need to modify that path of course.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>us.rempe.obsidian</string>
    <key>ProgramArguments</key>
        <array>
            <string>/Users/glenn/bin/obsidian_sync.sh</string>
        </array>
    <key>StartInterval</key>
    <integer>60</integer>
  </dict>
</plist>

In this case, the script /Users/glenn/bin/obsidian_sync.sh is run as my user every 60 seconds as specified in the StartInterval.

@pvroegh
Copy link

pvroegh commented Oct 19, 2022

@tallguyjenks, thanks for this script!
For those who want this in Powershell, I've ported the script: https://gist.github.com/pvroegh/633da28e615fe0e7101dc3a206b2b31a.

@phense
Copy link

phense commented Aug 12, 2023

It is generally a better idea to rely on git than on something like Google Drive (with file-based snapshots) and Obsidian's file recovery plugin.

The major downside of using Git and Obsidian is the following: Git makes delta-based snapshots of your files. Let's say you have a big text file. You save it in git, and it will save the whole file. You change 10 lines and make a new git commit. Now it won't save the entire file again but only the changes. Keeping that in mind, in Obsidian there is some internal refactoring going on. Let's say you are working on the same markdown file, but now you include 10 pictures. You put those .png files in the same directory as the markdown file, and you do a git commit to save your changes.
After working on your file, you notice that your directory has become a mess of files and should be cleaned up. Now you have a decision to make:

  • Make a "media" subfolder and drag and drop your image files in there, within Obsidian. This allows Obsidian to change the links in your markdown file dynamically so that you don't have to do anything yourself. But git cannot do delta's on media files. It only knows that you "deleted" your 10 images from this folder, and you "added" 10 new images in the "media" subfolder. Sure, it will "delete" the old images, but they are still within your repository, together with the same "new" images in the new subfolder. They now exist twice, or how many times you move them around.
  • Instead, you open a console and manually call the git mv command on each individual image file. This will allow git to recognize that the images just moved, and will not add them a second time. But Obsidian will not realize the changed file location, so you have to manually change the image-links in your markdown file.

This is not a big problem initially, especially not the example I described. This was only to describe the issue.
Now imagine you have a main structure of your obsidian vault, and you realize that it would be a great idea to prefix the main folders. Something like this: "000 - Inbox", "100 - Daily", "200 - Projects", ... , "FFE - Meta" (obsidian related files, templates, dataview views, etc.), "FFF - Archive".

Your obsidian vault is 350 MB, most of it from images and maybe PDF documents. You rename your entire folder structure and do a git commit. You will realize that your git repository suddenly doubled in size because it deleted all the files and added all the files under a new folder structure, but kept the references to the old files inside the repository. Git will have also lost all "delta"-changes-references for the new file. Even though it may have 10 different changes to your "thesis.tex" file, for example, that is only true for the old thesis.tex file, not the new thesis.tex file.
Change something in your folder structure again, for example make a "FFF Media" folder and movie all the media files into it, and suddenly, your git repository expanded to almost 1 GB.

I have yet to find a fix around this issue.
The alternative is using obsidian synch, or the "Remotely Save" - plugin. Though, people complained about that particular plugin loosing data / deleting files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment