Synchronize history across bash sessions

sync-history.sh

# Synchronize history between bash sessions
#
# Make history from other terminals available to the current one. However,
# don't mix all histories together - make sure that *all* commands from the
# current session are on top of its history, so that pressing up arrow will
# give you most recent command from this session, not from any session.
#
# Since history is saved on each prompt, this additionally protects it from
# terminal crashes.


# keep unlimited shell history because it's very useful
export HISTFILESIZE=-1
export HISTSIZE=-1
shopt -s histappend   # don't overwrite history file after each session


# on every prompt, save new history to dedicated file and recreate full history
# by reading all files, always keeping history from current session on top.
update_history () {
  history -a ${HISTFILE}.$$
  history -c
  history -r  # load common history file
  # load histories of other sessions
  for f in `ls ${HISTFILE}.[0-9]* 2>/dev/null | grep -v "${HISTFILE}.$$\$"`; do
    history -r $f
  done
  history -r "${HISTFILE}.$$"  # load current session history
}
if [[ "$PROMPT_COMMAND" != *update_history* ]]; then
  export PROMPT_COMMAND="update_history; $PROMPT_COMMAND"
fi

# merge session history into main history file on bash exit
merge_session_history () {
  if [ -e ${HISTFILE}.$$ ]; then
    cat ${HISTFILE}.$$ >> $HISTFILE
    \rm ${HISTFILE}.$$
  fi
}
trap merge_session_history EXIT


# detect leftover files from crashed sessions and merge them back
active_shells=$(pgrep `ps -p $$ -o comm=`)
grep_pattern=`for pid in $active_shells; do echo -n "-e \.${pid}\$ "; done`
orphaned_files=`ls $HISTFILE.[0-9]* 2>/dev/null | grep -v $grep_pattern`

if [ -n "$orphaned_files" ]; then
  echo Merging orphaned history files:
  for f in $orphaned_files; do
    echo "  `basename $f`"
    cat $f >> $HISTFILE
    \rm $f
  done
  echo "done."
fi

This script sounds very interesting. But what am I supposed to do with the script? What do I have to configure how to use this?
Thanks

@dkadioglu save and source it in your shell configuration file (usually ~/.bashrc on Linux, ~/.bash_profile on Mac) - like this:

source sync-history.sh

Let me know if you need more detailed instructions.

Thank you very much, works so far. One thing though: It seems, that the deletion of entries (history -d offset) is not possible anymore. At least for me, whichever item I delete, it still appears in the history. I tried to find a solution for that, without success so far. Do you have any idea?

@dkadioglu save and source it in your shell configuration file (usually ~/.bashrc on Linux, ~/.bash_profile on Mac) - like this:

source sync-history.sh

Let me know if you need more detailed instructions.

hi there i stumbled on this, and just wanted to mention that instead of using source sync-history.sh there is a safer way to do this :

[[ -r /path/to/file.sh ]] && . /path/to/file.sh

explanation:

• [[ -r checks fi the file that follows is readable and returns True if so

• && executes only if the command before returns True (or exit status 0 = success)

• . is somewhat shorthad for source (note there is a spacebehind the '.' so '. '

result : tests is if the file exists and is readable before sourcing it.

you can combine it with:

 FILE="nohup.out"  ; [[ -r $FILE ]] && .  $FILE ; [[ ! -e   $FILE ]] && echo "! Waring $FILE not found "

wich will give you a warning if you somhow in the future accidently rename or move the file.

if you have your bash rc split up in multiple files (like i have to create some order in the chaos) you can use:

for conf in $(ls /opt/local/scripts/rc/bash/[3-9]??_*) ; do 
    printf "Loading: $(basename $conf)"
    [[ -r "${conf}" ]] && source "${conf}" 
    printf '\x1b[40G\x1b[32mDONE\x1b[0m\n'
done

wich will in this case find all the files that start with a digit from 3-9 (my dir contents look like :bash 000_bashrc.conf 100_includes.conf 201_opts.conf 221_binds.conf ... 701_exports.conf hoefkens.bash_history ... an the loop is called from within includes (wich so the 000 and 1* , 2* dont need to get loaded
and source them one by one printing a success message to stdout
starting a new tty session thus looks like:

@hoefkensj Thank you very much for the helpful tip and the improvement!

If you run this script on macOS, the pgrep statement gets confused because the process name is -bash. It's easy to fix, and the improvement works on Linux and OpenBSD too.

Change this line in the gist:

active_shells=$(pgrep `ps -p $$ -o comm=`)

to look like this:

active_shells=$(pgrep -- `ps -p $$ -o comm=`)

Would it be possible to place the temporary history files in a specific folder? I would like to mount it like a volume in a docker container together with ~/.bash_history, so i can also have access and keep history for the terminal activity from the container also

yeah , im currently writing something of my own (currently buggy but) ... extended version of this ....
note that i said buggy so :) but it might help you allong since i do use a directory to store , several files (one for each pid , boot, everythig , .... ) in /var/cache/bash/history... here is the code of what it currently is :

#!/usr/bin/env bash
# ############################################################################
# # PATH: /opt/local/config/rc/bash               AUTHOR: Hoefkens.j@gmail.com
# # FILE: 311_history.conf                                 2023-04-04 09:33:40
# ############################################################################
#
# set -o xtrace
# set -o nounset
function bash_history() {
 	function HELP() {
        echo -e "\nUsage: bash_history [option]\n"
        echo -e "Options:"
        echo -e "  install\tCreate the necessary HISTDIRectories and files for the history tracking"
        echo -e "  clean\t\tRemove the current HISTSESSION's history and reset the command history"
        echo -e "  start\t\tStart a new HISTSESSION and begin tracking commands"
        echo -e "  stop\t\tStop the current HISTSESSION and stop tracking commands"
        echo -e "  show [--all]\tDisplay the command history, use --all to show all history"
        echo -e "  active\tList active history HISTSESSIONs"
        echo -e "  orphaned\tList orphaned history HISTSESSIONs"
        echo -e "  help\t\tDisplay this help message\n"
        echo -e "Example: bash_history start\n"
	}

	function install_fifo(){
		printf '%s...' "$1" 
		sudo rm -rfv "$1" 
		sudo mkfifo "$1"  
		sudo chmod 666 "$1"
		echo "DONE"
	}
	


	function history_install() {		
		function _install(){
			printf 'Installing: %s...' "$1"
			if [[ "$2"  == 'd' ]] ; then
				mkdir -p -m777  "$HISTDIR"  || sudo mkdir -p -m777 "$HISTDIR" 
			else		
				[[ -e "$1" ]] && install -m 777 /dev/null "$1"
				[[ ! -w "$1" ]] && sudo chmod 777 $1
			fi
			echo "DONE"
		}
		_install "$HISTDIR" "d"
		_install "$HISTSYSFULL" 	
		_install "$HISTSYSBOOt"
		_install "$HISTSYSMETA"
		[[ ! -e $HISTSYSLAST ]] &&   install_fifo "$HISTSYSLAST"
		[[ ! -e $HISTSYSUNIQ ]] &&   install_fifo "$HISTSYSUNIQ"

	}
	function history_update()	{
		builtin history -a "$HISTSYSLAST"
		builtin history -a "$HISTSYSFULL"
		builtin history -a "$HISTSESSION" 
		builtin history -c	
		cat "$HISTSYSFULL" "$HISTSYSBOOt" |tac| awk '!seen[$0]++' |tac > "$HISTFILE"
		cat $HISTSYSLAST |tee -a $HISTFILE |md5sum| history_meta >> "$HISTSYSMETA"
		builtin history -r "$HISTFILE"
	}
	function history_meta() 	{
		#N                   #STAMP 		
		local dat usr hst pid tty
		pNR=$( cat "$HISTSYSFULL" |wc -l )
		NR=$((pNR+1))
		_date="$( date +%s )" 
		_user="$USER" 
		_host="$HOSTNAME" 
		_ppid="$$"
		_tty="$(tty)"
		
		printf '%s\t\t%s\t\t%s\t\t%s\t\t' "$NR" "$_ppid" "$_date" "$BOOTSTAMP" 
		printf '%s\t\t%s\t\t%s\t\t%s\t\t' "$_host" "$_user" "$_tty" "$SHELL" 
		printf '%s\t\t%s\n' 							"$PS1" "$1" 
	}
	function history_start(){
		history_cleanup 
		[[ ! -e $HISTFILE ]] && install -m 777 /dev/null "$HISTFILE" 
		[[ ! -e $HISTSESSION ]] && install -m 777 /dev/null "$HISTSESSION" 

	}
	function history_cleanup (){
		cat "$HISTFILE" $HISTSESSION >> "$HISTDIR/$$.recovered"
		[[ -e $HISTFILE ]] && sudo trash "$HISTFILE" 
		[[ -e $HISTSESSION ]] && sudo trash "$HISTSESSION"
		[[ -e $HISTFILE ]] && sudo rm -rvf "$HISTFILE" 
		[[ -e $HISTSESSION ]] && sudo rm -rvf "$HISTSESSION"
	}
	function history_stop(){
		[[ -e $HISTFILE ]] && trash "$HISTFILE" 
		[[ -e $HISTSESSION ]] && trash "$HISTSESSION"
		history_cleanup
	}
	## HELPER FUNCTION if not on system:
	
	if [[ -z $(which batcat 2>/dev/null) ]]; then 
		function batcat () {
			local _cat _bat LANG STRING COLOR
			LANG="$1"
			shift 1
			STRING="$@"
			_cat=$( which "cat" )
			_bat=$( which "bat" )
			[[ -n "$_bat" ]] && printf '%s' "$@"  | $( printf '%s --%s --%s=%s' "$_bat" "plain" "language" "$LANG" ) 
			[[ -z $_bat ]] && echo $( printf '%s' "$@" ) | $( printf '%s' "$_cat"  ) 
		};
	fi	

	export BOOTSTAMP="$(uptime -s | tr -d '\-: ')" 
	export HISTSIZE=-1 
	export HISTFILESIZE="$HISTSIZE" 
	export HISTCONTROL=''
	export PFIX="history"
	export HISTDIR="/var/cache/history/bash"

	export HISTFILE="${HISTDIR}/${PFIX}.$$"
	export HISTSESSION="${HISTHISTDIR}/${PFIX}.HISTSESSION.$$"
	export HISTSYSBOOt="${HISTDIR}/system.boot.${BOOTSTAMP}"
	export HISTSYSFULL="${HISTDIR}/system.full.${PFIX}"
	export HISTSYSMETA="${HISTDIR}/system.meta.${PFIX}"
	export HISTSYSLAST="${HISTDIR}/system.last.${PFIX}" #FIFO



	FNC=${FUNCNAME[0]}
	case "$1" in
		install) history_install  &>/dev/null;;
		help)    HELP  &>/dev/null;;
		start)   history_start  &>/dev/null;;
		stop)    history_stop  &>/dev/null;;
		update)  history_update  &>/dev/null ;;
		uniq)    shift && history_uniq "$@"  &>/dev/null ;;
		meta)    history_meta "$@"  &>/dev/null ;;
		debug)   set -o xtrace
	esac

}	

function HISTCLEANUP() {
  echo "Cleaning up HIST files..."
  bash_history stop
  sleep 0.1
}

function HISTUPDATE() {
	builtin history -a "$HISTSYSLAST"
	bash_history update 
}

function history(){
	bash_history show
	builtin history "$@"
}

trap HISTCLEANUP  EXIT
bash_history start &>/dev/null
echo $SHELL  &>/dev/null
bash_history update  &>/dev/null
[[ "${PROMPT_COMMAND}" != *"HISTUPDATE"* ]] && export PROMPT_COMMAND="HISTUPDATE ; ${PROMPT_COMMAND}"

ps: i plan to add git functionality to it and some form of ecryption, so i can sinc it between machines using github aswell...

@hoefkensj ambitious@nice, but could not make it work. In the end I just:
export HISTFILE=~/.bash/.bash_history

Is there a way to record only direct inputs? Now it seems it catches all inputs, history file is full of mc's outputs like:
cd "printf '%b' '\0057home"

Still trying to fix this to use with tmux. Now it catches all the commands sent from random apps, but later this breaks the input ending up with:
$ mc
$ ls -la
$ ls -latop
can't remove that ^ ls -la
upd: Ok, that part probably was caused by cyd01/KiTTY and fixed by changing termtype to tmux-256color

MC's internal commands has a leading space so it won't be saved in history, but it still gets saved. Fixed it by checking for leading space and exiting the function. But is there a better solution to rrecord only user inputs?

update_history () {
    lastCmd=$(fc -lnr | head -n1)
    # clean the spaces left by fc
    lastCmdReal=${lastCmd:2}
    # remove all leading spaces
    lastCmdRealClean=${lastCmd##+([[:space:]])}

    if [[ "$lastCmdReal" == " $lastCmdRealClean"  ]]; then
       return 0
    fi