Salt SSH vs Heist

salt-ssh-vs-heist.md

Feature parity between Salt SSH and Heist

Salt SSH features

• Works without a master. Unlike Heist minions, does not hold persistent ssh connection and there is nothing running on a remote system until you run a salt-ssh command
• Supports masterless orchestrations, see the ssh and roster arguments https://docs.saltproject.io/en/master/ref/states/all/salt.states.saltmod.html#salt.states.saltmod.state
• Salt-SSH minions https://docs.saltproject.io/en/latest/ref/configuration/master.html#enable-ssh-minions Unlike Heist minions, they do not hold persistent ssh connections and there is nothing running on a remote system until you run a command like salt \* state.apply. This is quite important feature for those who are against running any management agents
• Can work without internet access
• Supports master tops on masterless minions saltstack/salt#65481
• PCRE and Glob targeting
• Salt client (available directly and via salt-api) https://docs.saltproject.io/en/latest/ref/clients/index.html#sshclient
• Supports ~/.ssh/config (host aliases, users, ssh options, jumphosts like ProxyJump="user@jumphost.example.com")
• Raw shell calls
• One-off calls without a roster: salt-ssh --sudo user@host state.apply
• Supports Saltfile (it allows running salt-ssh from a single folder with all states and config files). Doesn't work in Heist https://gitlab.com/saltstack/pop/heist-salt/-/issues/70
• Default minion options + per-minion overrides in a roster (Heist has this too)
• Lots of roster options https://docs.saltproject.io/en/latest/topics/ssh/roster.html#targets-data Notable highlights:
  • Minion options
  • Sudo support
  • Password-based logins
  • ssh options (e.g. ProxyJump="user@jumphost.example.com")
  • Both plain and password-protected ssh keys
  • SSH agent forwarding
  • Pre-flight scripts and options
  • set_path option
• Pass extra modules via CLI/config: --min-extra-modules --thin-extra-modules
• Access the regular mine/event bus from salt-ssh (mine and publish wrappers) saltstack/salt#65646)
• Can run saltcheck tests over salt-ssh
• Different Python versions https://docs.saltproject.io/en/latest/topics/ssh/ssh_ext_alternatives.html (not needed in Heist, because of Onedir packages)

Salt SSH rosters

• ansible - Read in an Ansible inventory file or script.
• cache - The cache roster provides a flexible interface to the Salt Masters' minion cache to access regular minions over salt-ssh.
• cloud - Use the cloud cache on the master to derive IPv4 addresses based on minion ID.
• clustershell - This roster resolves hostname in a pdsh/clustershell style
• dir - Create a salt roster out of a flat directory of files.
• flat - Read in the roster from a flat file using the renderer system
• range - This roster resolves targets from a range server.
• scan - Scan a netmask or ipaddr for open ssh ports
• sshconfig - Parses roster entries out of Host directives from SSH config
• sshknownhosts - Parses roster entries out of Host directives from SSH known_hosts
• terraform - Dynamic roster from terraform current state (also see https://github.com/dmacvicar/terraform-provider-salt)

Heist rosters

• heist.roster.clustershell
• heist.roster.flat
• heist.roster.scan

Salt SSH shortcomings

• Doesn't support Windows destination hosts (not sure whether Windows source host is supported)
• By default wants to access /etc, /var/log, /var/cache. CLI args or a Saltfile is needed to overcome that.
• Not every Salt feature is supported (wrappers are needed)
• Issues related to incomplete state tree being transferred to remote hosts (saltstack/salt#60003)

Useful features to borrow from salt-cloud

• preflight_cmds https://docs.saltproject.io/en/latest/topics/cloud/misc.html#running-pre-flight-commands
• inline_script saltstack/salt#19843
• file_map (to upload files) https://docs.saltproject.io/en/latest/topics/cloud/misc.html#file-map-upload
• ssh_gateway_command saltstack/salt#48062

Heist shortcomings

• The biggest one is lack of masterless support (+ orchestrations)
• Hard to use without configs and rosters for one-off runs: /opt/saltstack/salt/extras-*/bin/heist salt.minion -c /dev/null --roster-dir /tmp -R /dev/null --roster-data '{"minion1": {"host": "minion1", "username": "root"}}'
• No way to define additional minion options, grains
• Slow to start
• Hard to bootstrap with extensions and other dependencies https://gitlab.com/saltstack/pop/heist-salt/-/issues/73
• Impossible to install a specific minion version https://gitlab.com/saltstack/pop/heist-salt/-/issues/71
• No way to specify custom artifact repo locations
• Hard to get remote Salt logs (possible in salt-ssh https://twitter.com/SaltTips/status/1146306964026253312)
• No bash/zsh completions
• In the past, the overall user experience and documentation was bad https://salt.tips/new-features-in-salt-silicon/#heist-minion-presence-events Many of these things have been addressed

Comments by @lkubb

heist-salt

• + I assume the bidirectional tunnel solves many architectural problems Salt-SSH currently has (requiring wrappers, empty __pillar__ on the remote, incomplete state tree for Jinja imports, very hacky to simulate peer publishing on the remote - e.g. for the x509_v2 modules, strict separation of regular from SSH minions, the other way around works now)
• + Delivers compiled core dependencies like cryptography to the remote at no cost.
• - Agree that requiring a master is a significant downstep. Salt-SSH can be used standalone, a bit like Ansible. The cognitive and actual steps to get /opt/saltstack/salt/extras-*/bin/heist salt.minion ... into a workable state also seem more complicated than for salt-ssh ....
• - Not sure about the integration depth into Salt itself. I assume to target the minions from Salt, I have to (in some way) instantiate a LocalClient from the master node (and target via the minion_type grain)? Does this require persistent connections to be established before in a separate step or are they created on-the-fly (and the remote cleaned up afterwards) like with Salt-SSH?
• - From what I can tell, heist-salt currently only works for x86_64. This is easily fixed for cpu archs that have official onedir artifacts (and provides more stable support for supported platforms), but will still lack support for those without (apart from custom onedir builds).

Salt-SSH

• + Allows to run raw SSH commands
• + I can provide custom roster modules via the fileserver, does this work for heist-salt? E.g., I'm using a custom module that leverages Salt (+ optionally Vault) to generate short-lived SSH certificates for authentication on-the-fly (https://github.com/lkubb/salt-openssh-formula)
• + Uses a plain text protocol for communication that can easily be wrapped for specialized use cases (https://github.com/QubesOS/qubes-mgmt-salt/blob/main/ssh-wrapper)
• + Works even without an active internet connection on both the master and the remote (I assume with offline_mode and cached artifacts this is valid for heist-salt as well)
• - The remote is a dumb executor, which requires many workarounds (see heist-salt first point and saltstack/salt#64542)
• - The minion/master hybridization salt-ssh does on the master is very flaky (especially re. __opts__) (e.g. saltstack/salt#36796 (comment), https://github.com/lkubb/salt-openssh-formula/blob/5dc3140b4ffbc43f7b70b3c8f32fda45687948d0/_roster/autocert_flat.py#L232-L253)