Skip to content

Instantly share code, notes, and snippets.

@pedroamador
Forked from rwenz3l/proxmox_lxc_nfs_server.md
Created October 13, 2019 18:03
Show Gist options
  • Save pedroamador/dec94897537ef025945f1b96c97fcf59 to your computer and use it in GitHub Desktop.
Save pedroamador/dec94897537ef025945f1b96c97fcf59 to your computer and use it in GitHub Desktop.
Install a NFS Server inside a LXC Container on Proxmox 5.1

Installing NFS inside LXC Container on Proxmox 5.1

Host Setup:

Create LXC Container as usual, but do not start it yet.

# Install NFS-Kernel on Host
apt install nfs-kernel-server

# Create a new AppArmor file: 
touch /etc/apparmor.d/lxc/lxc-default-with-nfsd

# Write Profile:
cat > /etc/apparmor.d/lxc/lxc-default-with-nfsd << 'EOF'
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-with-nfsd flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,
  mount fstype=nfsd,
  mount fstype=rpc_pipefs,
  mount fstype=cgroup -> /sys/fs/cgroup/**,
}
EOF

# Activate the new Profile:
apparmor_parser -r /etc/apparmor.d/lxc-containers

# Add Profile to Container:
# (in this case: id = 200)
echo 'lxc.apparmor.profile = lxc-container-default-with-nfsd' \
  >> /etc/pve/nodes/sniebel/lxc/200.conf

# As well as to it's config:
echo 'lxc.apparmor.profile = lxc-container-default-with-nfsd' \
  >> /var/lib/lxc/200/config
  
# Also add your mountpoint to the container:
# If you have a cluster setup:
echo 'mp0: /mnt/host_storage,mp=/mnt/container_storage' \
  >> /etc/pve/nodes/cluster_node/lxc/200.conf

# If you have a single node setup:
echo 'mp0: /mnt/host_storage,mp=/mnt/container_storage' \
  >> /etc/pve/lxc/200.conf

# Finall start the container:
lxc-start -n 200

Container Setup:

ssh into the container or do a simple lxc-attach -n 200 on your host (where 200 is the id).

# Install nfs
apt update
apt install nfs-kernel-server

# Edit Exports
nano /etc/exports

# or append like so (example):
echo '/mnt/container_storage 192.168.0.0/16(rw,async,insecure,no_subtree_check,all_squash,anonuid=501,anongid=100,fsid=1)' \
  >> /etc/exports

# disconnect from the container

# Restart it:

Host again:

Back on the Host restart the container:

lxc-stop -n 200
lxc-start -n 200

Because the nfs-kernel is on the host, the container cannot access it's status. service nfsd status therefore shows as 'not running' inside the container. .. this seems to be normal (?)


Further useful commands:

nfsstat # list NFS statistics
@MrColumbo
Copy link

Should this still work with Proxmox 7.1?

@pedroamador
Copy link
Author

Yes, it works on 6.x and 7.x. I’m uses this this currently

For recent versuons (6 and 7) is easy, symply edit the LXC instance configurarion and enable this this in “features” section:

fuse=1
nesting=1

This settibgs replaxe all “app armour” config of the container. The server config and mount point config is the same.

@MrColumbo
Copy link

MrColumbo commented Jan 24, 2022

Thanks for the feedback.

Somehow it does not work for me.

I just installed a fresh lxc in proxmox using the debian 11 template
I added fuse and nesting (nesting was already enabled per default)
started the container and did an
apt install nfs-kernel-server rpcbind

This is the full output

Reading package lists... Done
Building dependency tree... Done
The following additional packages will be installed:
keyutils libevent-2.1-7 libnfsidmap2 nfs-common
Suggested packages:
open-iscsi watchdog
The following NEW packages will be installed:
keyutils libevent-2.1-7 libnfsidmap2 nfs-common nfs-kernel-server rpcbind
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 682 kB of archives.
After this operation, 2028 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ftp.debian.org/debian bullseye/main amd64 rpcbind amd64 1.2.5-9 [51.4 kB]
Get:2 http://ftp.debian.org/debian bullseye/main amd64 keyutils amd64 1.6.1-2 [52.8 kB]
Get:3 http://ftp.debian.org/debian bullseye/main amd64 libevent-2.1-7 amd64 2.1.12-stable-1 [188 kB]
Get:4 http://ftp.debian.org/debian bullseye/main amd64 libnfsidmap2 amd64 0.25-6 [32.6 kB]
Get:5 http://ftp.debian.org/debian bullseye/main amd64 nfs-common amd64 1:1.3.4-6 [232 kB]
Get:6 http://ftp.debian.org/debian bullseye/main amd64 nfs-kernel-server amd64 1:1.3.4-6 [125 kB]
Fetched 682 kB in 0s (1476 kB/s)
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
Selecting previously unselected package rpcbind.
(Reading database ... 20146 files and directories currently installed.)
Preparing to unpack .../0-rpcbind_1.2.5-9_amd64.deb ...
Unpacking rpcbind (1.2.5-9) ...
Selecting previously unselected package keyutils.
Preparing to unpack .../1-keyutils_1.6.1-2_amd64.deb ...
Unpacking keyutils (1.6.1-2) ...
Selecting previously unselected package libevent-2.1-7:amd64.
Preparing to unpack .../2-libevent-2.1-7_2.1.12-stable-1_amd64.deb ...
Unpacking libevent-2.1-7:amd64 (2.1.12-stable-1) ...
Selecting previously unselected package libnfsidmap2:amd64.
Preparing to unpack .../3-libnfsidmap2_0.25-6_amd64.deb ...
Unpacking libnfsidmap2:amd64 (0.25-6) ...
Selecting previously unselected package nfs-common.
Preparing to unpack .../4-nfs-common_1%3a1.3.4-6_amd64.deb ...
Unpacking nfs-common (1:1.3.4-6) ...
Selecting previously unselected package nfs-kernel-server.
Preparing to unpack .../5-nfs-kernel-server_1%3a1.3.4-6_amd64.deb ...
Unpacking nfs-kernel-server (1:1.3.4-6) ...
Setting up rpcbind (1.2.5-9) ...
Created symlink /etc/systemd/system/multi-user.target.wants/rpcbind.service → /lib/systemd/system/rpcbind.service.
Created symlink /etc/systemd/system/sockets.target.wants/rpcbind.socket → /lib/systemd/system/rpcbind.socket.
Setting up libevent-2.1-7:amd64 (2.1.12-stable-1) ...
Setting up keyutils (1.6.1-2) ...
Setting up libnfsidmap2:amd64 (0.25-6) ...
Setting up nfs-common (1:1.3.4-6) ...
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

Creating config file /etc/idmapd.conf with new version
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
Adding system user statd' (UID 108) ... Adding new user statd' (UID 108) with group nogroup' ... Not creating home directory /var/lib/nfs'.
Created symlink /etc/systemd/system/multi-user.target.wants/nfs-client.target → /lib/systemd/system/nfs-client.target.
Created symlink /etc/systemd/system/remote-fs.target.wants/nfs-client.target → /lib/systemd/system/nfs-client.target.
nfs-utils.service is a disabled or a static unit, not starting it.
Setting up nfs-kernel-server (1:1.3.4-6) ...
Created symlink /etc/systemd/system/multi-user.target.wants/nfs-server.service → /lib/systemd/system/nfs-server.service.
A dependency job for nfs-server.service failed. See 'journalctl -xe' for details.
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

Creating config file /etc/exports with new version
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

Creating config file /etc/default/nfs-kernel-server with new version
A dependency job for nfs-server.service failed. See 'journalctl -xe' for details.
invoke-rc.d: initscript nfs-kernel-server, action "start" failed.
● nfs-server.service - NFS server and services
Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
Active: inactive (dead)

Jan 24 17:17:54 collector systemd[1]: Dependency failed for NFS server and services.
Jan 24 17:17:54 collector systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.
Jan 24 17:17:54 collector systemd[1]: Dependency failed for NFS server and services.
Jan 24 17:17:54 collector systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.
Failed to start nfs-kernel-server, ignoring.

...easy to reproduce I hope ;)

What am I missing?

@pedroamador
Copy link
Author

I have the NFS over LXC running in a pve 7 server, based on a new created lxc instance

Here's my config:

# cat 10020.conf 
arch: amd64
cores: 4
features: fuse=1,nesting=1
hostname: file
memory: 2048
nameserver: 10.0.0.1
net0: name=eth0,bridge=vmbr1,firewall=1,gw=10.0.0.1,hwaddr=******,ip=10.0.0.20/24,type=veth
ostype: ubuntu
rootfs: data:subvol-10020-disk-0,size=100G
searchdomain: local
swap: 2048

You can't use "unprivileged container" for this, so uncheck the option (or do a "vzdump" and "pct restore -unprivileged 0"). The rest of the config is the same, installing "nfs-kernel-server" on the LXC instance and... it's all :)

In the LXC instance:

root@file:~# lsmod|grep nfs
nfsd                  405504  13
auth_rpcgss            94208  2 nfsd,rpcsec_gss_krb5
nfs_acl                16384  1 nfsd
lockd                 102400  1 nfsd
grace                  16384  2 nfsd,lockd
sunrpc                393216  20 nfsd,auth_rpcgss,lockd,rpcsec_gss_krb5,nfs_acl
root@file:~# cat /etc/exports 
# /etc/exports: the access control list for filesystems which may be exported
#		to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
/mnt/container_storage 10.0.0.0/24(rw,async,insecure,no_subtree_check,no_root_squash)
root@file:~# rpcinfo | egrep "service|nfs"
   program version netid     address                service    owner
    100003    3    tcp       0.0.0.0.8.1            nfs        superuser
    100003    4    tcp       0.0.0.0.8.1            nfs        superuser
    100003    3    udp       0.0.0.0.8.1            nfs        superuser
    100003    3    tcp6      ::.8.1                 nfs        superuser
    100003    4    tcp6      ::.8.1                 nfs        superuser
    100003    3    udp6      ::.8.1                 nfs        superuser

In a KVM instance client with this directory mounted:

root@client:~# mount -t nfs file:/mnt/container_storage /mnt/container_storage/
root@client:~# df -h /mnt/container_storage/
Filesystem                        Size  Used Avail Use% Mounted on
file:/mnt/container_storage  100G  831M  100G   1% /mnt/container_storage

@MrColumbo
Copy link

thanks a lot - i thought with the options it will work on an unprivileged container as well but I was obviously wrong. Now it works for me as well.

@pedroamador
Copy link
Author

Great :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment