Proxmox VE sends backup in clear text when the encryption key file is not found in place

Readme.md

Proxmox VE sends backup in clear text when the encryption key file is not found in place

I setup my proxmox server to send backup to PBS.

I also wanted to ensure that the backup encryption key file was not sitting in the disk in clear text. Because proxmox does not support FDE out of the box, my plan was that the actual file will remain in an encrypted drive which I will decrypt after boot and then create a symlink to "/etc/pve/priv/storage/${storeid}.enc".

The flaw in this plan is though that if Promox triggered the backup before that then the file would be not be found and it'll fail the backup. Or would it? Seems like a sensible thing to fail the backup if enc key file is not in place. So I removed the enc file and ran the backup.

And lo and behold backup DOES NOT FAIL.

So I asked on forum, reddit and also discord but to no avail. I decided to take a look at the source code. I am no perl/rust dev but from the source code it is evident that

PROXMOX WILL SEND BACKUP IN CLEAR TEXT IF ENCRYPTION KEY FILE IS NOT AVAILABLE

Check the following snippets taken from Proxmox repos (pve-storage & pbs-client)

Methods calls propagates as do_raw_client_cmd -> pbs_open_encryption_key

pbs_open_encryption_key will simply return undef if the file is not in place. And due to this do_raw_client_cmd will set the backup mode to none by sending '--crypt-mode=none' to the pbs backup client, resulting the backup to be sent in clear text.

In my opinion this is a serious security issue.

When the user checkmarks the encryption option during the pbs adding phase in UI, the following will be added to /etc/pve/storage.cfg.

pbs: pbs-backup
        datastore xx
        server xxx
        content backup
        encryption-key md5 fingerprint of key
        fingerprint md5 fingerprint of server
        prune-backups keep-all=1
        username xx@xx

In UI it is reflected that encryption is on, but in reality even though unlikely it's very possible that if the file is missing due to some reason the encryption will be silently disabled. And this is a HUGE disconnect from what is shown to the user in the UI and user will never know backup is completely exposed.

Resolution

A proper way to handle this would be to check if the encryption-key value is avavilable in the storage config in /etc/pve/storage.cfg and fail the backup if the encryption key does not match the fingerprint AND if the encryption key file is missing. User will take note of this and know backup has failed and nothing was sent in clear text.

PBSPlugin.pm

sub pbs_encryption_key_file_name {
    my ($scfg, $storeid) = @_;

    return "/etc/pve/priv/storage/${storeid}.enc";
}

# Returns a file handle if there is an encryption key, or `undef` if there is not. Dies on error.
sub pbs_open_encryption_key {
    my ($scfg, $storeid) = @_;

    my $encryption_key_file = pbs_encryption_key_file_name($scfg, $storeid);

    my $keyfd;
    if (!open($keyfd, '<', $encryption_key_file)) {
	return undef if $! == ENOENT;
	die "failed to open encryption key: $encryption_key_file: $!\n";
    }

    return $keyfd;
}

my sub do_raw_client_cmd {
    my ($scfg, $storeid, $client_cmd, $param, %opts) = @_;

    my $use_crypto = $USE_CRYPT_PARAMS->{$client_cmd};
    my $use_master = $USE_MASTER_KEY->{$client_cmd};

    my $client_exe = '/usr/bin/proxmox-backup-client';
    die "executable not found '$client_exe'! Proxmox backup client not installed?\n"
	if ! -x $client_exe;

    my $repo = PVE::PBSClient::get_repository($scfg);

    my $userns_cmd = delete $opts{userns_cmd};

    my $cmd = [];

    push @$cmd, @$userns_cmd if defined($userns_cmd);

    push @$cmd, $client_exe, $client_cmd;

    # This must live in the top scope to not get closed before the `run_command`
    my ($keyfd, $master_fd);
    if ($use_crypto) {
	if (defined($keyfd = pbs_open_encryption_key($scfg, $storeid))) {
	    my $flags = fcntl($keyfd, F_GETFD, 0)
		// die "failed to get file descriptor flags: $!\n";
	    fcntl($keyfd, F_SETFD, $flags & ~FD_CLOEXEC)
		or die "failed to remove FD_CLOEXEC from encryption key file descriptor\n";
	    push @$cmd, '--crypt-mode=encrypt', '--keyfd='.fileno($keyfd);
	    if ($use_master && defined($master_fd = pbs_open_master_pubkey($scfg, $storeid))) {
		my $flags = fcntl($master_fd, F_GETFD, 0)
		    // die "failed to get file descriptor flags: $!\n";
		fcntl($master_fd, F_SETFD, $flags & ~FD_CLOEXEC)
		    or die "failed to remove FD_CLOEXEC from master public key file descriptor\n";
		push @$cmd, '--master-pubkey-fd='.fileno($master_fd);
	    }
	} else {
	    push @$cmd, '--crypt-mode=none';
	}
    }

    push @$cmd, @$param if defined($param);

    push @$cmd, "--repository", $repo;
    if ($client_cmd ne 'status' && defined(my $ns = $scfg->{namespace})) {
	push @$cmd, '--ns', $ns;
    }

    local $ENV{PBS_PASSWORD} = pbs_get_password($scfg, $storeid);

    local $ENV{PBS_FINGERPRINT} = $scfg->{fingerprint};

    # no ascii-art on task logs
    local $ENV{PROXMOX_OUTPUT_NO_BORDER} = 1;
    local $ENV{PROXMOX_OUTPUT_NO_HEADER} = 1;

    if (my $logfunc = $opts{logfunc}) {
	$logfunc->("run: " . join(' ', @$cmd));
    }

    run_command($cmd, %opts);
}

Does the backup show as not encrypted when it hits PBS?

Yes it shows up as encrypted. No indication that data was sent as cleartext

When the backup runs, whats args do you see for the process

Not familiar with how to check that. Got a command I can run?

during a VM is actively backing up, run the below command, it'll display all processes with "proxmox-backup-client" in it.

ps x | grep proxmox-backup-client

https://forum.proxmox.com/threads/proxmox-is-sending-backup-to-pbs-in-clear-text-when-encryption-file-is-unavailable.113611/#post-491190

Hey sorry I was going to run the command today but looks like they are already responding to the post. Hope it gets patched soon. Thank you for the help.

I saw the forum post, fingers crossed. Although they should probably have a security inbox monitored 24/7, encase of a zero day 😅