configuration.nix

{ config, pkgs, ... }:

{
  imports = [ ./default.nix /root/nixcfg/core.nix ];
  services = {
    example = {
      enable = true;
    };
  };
  environment.systemPackages = with pkgs; [ wget tcpdump ltrace gdb ];
  networking.hostName = "nixos.example.com";
  networking.firewall.allowedTCPPorts = [ 80 ];
  virtualisation = {
    memorySize = 512;
    graphics = false;
    qemu.networkingOptions = [ "-net nic,macaddr=52:54:00:12:34:01" "-net vde,sock=/run/vde.ctl" ];
  };
}

irc.log

clever, irc.freenode.net BOT [13:31] 
nekroze: http://pastebin.com/pCNqL5yN

[13:32] 
with this i just nix-build vm.nix -A master -o master

[13:32] 
to build the entire vm and store it in a master symlink

[13:33] 
hmmm, let me paste more files into a gist

[13:35] 
nekroze: https://gist.github.com/cleverca22/5da6f037df2c7ea76211

[13:35] 
configuration.nix is one of the servers, and i added some custom network config to stick all of the VM's into a dedicated LAN


nekroze, irc.freenode.net BOT [13:35] 
clever: Ok, so that first one you can change master to either of those others down the bottom liek router, that all have their own config. And that just builds the vm's themselves right not runs them?


clever, irc.freenode.net BOT [13:35] 
line 15 makes that server headless

[13:36] 
routercfg.nix acts as a router, bridging that private LAN to the real LAN, and it has chromium so i can browse the website on the dummy server


nekroze, irc.freenode.net BOT [13:36] 
clever: But would these be persistent?


clever, irc.freenode.net BOT [13:36] 
and vm.nix ties everything together

[13:36] 
all of these store the rootfs in the current directory when you run the bootup script

[13:37] 
as a qemu qcow file

[13:37] 
ive got an old one left-over in ~ from when i last used it, -rw-r--r-- 1 root root 148M Nov  7 16:28 router.qcow2

[13:37] 
yeah, all the nix expression does is build the vm

[13:38] 
but if you import that into configuration.nix, and create a systemd unit that ran ${machine1}/bin/run-amd-nixos-vm, it would run on bootup


nekroze, irc.freenode.net BOT [13:38] 
clever: So could I put a systemd.units option in for each vm that executes "cd /vm-place && nix-build vm.nix -A router -o router" for the router vm?


clever, irc.freenode.net BOT [13:39] 
no need to do nix-build from the systemd unit


nekroze, irc.freenode.net BOT [13:39] 
clever: So vm.nix can be imported into the host configuration.nix and used there?


clever, irc.freenode.net BOT [13:39] 
just refer to the machine directly, let machine1 = (import vm.nix).master;

[13:39] 
and then "cd /vm-storage; ${machine1}/bin/run-amd-nixos-vm" from a systemd unit


nekroze, irc.freenode.net BOT [13:40] 
clever: Oh, wow. that is epic.


clever, irc.freenode.net BOT [13:40] 
now the unit depends on the vm, and when you 'nixos-rebuild', it will build all of the VMs and the host


nekroze, irc.freenode.net BOT [13:40] 
would it also detect changes just to the vm's configurations?


clever, irc.freenode.net BOT [13:40] 
yeah, next time you nixos-rebuild, it will rebuild the vm's

[13:40] 
and automaticaly tell systemd to restart the guests


nekroze, irc.freenode.net BOT [13:41] 
Ok, i need to read all of this over until I truely understand this.


clever, irc.freenode.net BOT [13:41] 
systemd will likely SIGTERM qemu, and ive got no idea what will happen then

[13:41] 
hopefully, that will register as an ACPI power button, and the guest will shutdown


nekroze, irc.freenode.net BOT [13:41] 
clever: I think kvm vm's are standard linux processes so it should be ok

[13:42] 
clever: Thank you so much as always for your assistance mate.


maurer, irc.freenode.net BOT [13:48] 
Is there a way to get a parameterized stdenv? clever mentioned clangStdenv, but I'd also like to be able to flip to musl, or if I make a build for it, icc

[13:50] 
Or is the best answer for me to just build several custom stdenvs?


clever, irc.freenode.net BOT [13:51] 
maurer: sounds like you just need to read the source for clangStdenv and make your own modified one


maurer, irc.freenode.net BOT [13:51] 
clever: Yeah, that's less parameterized than I was hoping sadly

[13:51] 
clever: since I'm not just trying to build a specific configuration, I'm trying to build many different varieties of a binary to make sure my analysis isn't specific to a compiler or libc

[13:51] 
(or arch, but I figured I'd leave the cross-compilation questions to later)


clever, irc.freenode.net BOT [13:52] 
maurer: sounds like you want to write a function that will create a new stdenv based on its parameters


maurer, irc.freenode.net BOT [13:52] 
clever: Yeah, I was hoping there was one

[13:52] 
I couldn't find it


nekroze, irc.freenode.net BOT [14:12] 
And in each of those config like sections you are importing the qemu-vm.nix which seems odd.


clever, irc.freenode.net BOT [14:12] 
ah, some of that file is now unused, let me edit it


nekroze, irc.freenode.net BOT [14:13] 
does the part in vm.nix that specifies networking.extraHosts does that get copied to all vms? or just to the host?


clever, irc.freenode.net BOT [14:13] 
nekroze: https://gist.github.com/cleverca22/5da6f037df2c7ea76211

[14:13] 
that goes into every guest


nekroze, irc.freenode.net BOT [14:13] 
clever: Ohhh, I see.


clever, irc.freenode.net BOT [14:14] 
it also sets the password of root on every guest, to root

[14:15] 
so i can easily manage the testing servers


nekroze, irc.freenode.net BOT [14:15] 
clever: So by default virtualization is set to all defaults there but can be overriden by guests


clever, irc.freenode.net BOT [14:15] 
yeah


nekroze, irc.freenode.net BOT [14:15] 
cool

roger.qiu [14:15] 
If you make a change to the vm configuration, would this ever affect whether those VMs are still compatible with the qcow2 file you persisted prior?


clever, irc.freenode.net BOT [14:16] 
at one point i had graphics=false in vm.nix to make it all headless

[14:16] 
but you cant set it to both false and true, so when i made it true on the router, i had to remove it from vm.nix

[14:16] 
CMCDragonkai: yeah


nekroze, irc.freenode.net BOT [14:16] 
clever: Do they just pop up as windows? I have never run non headless kvm vm's


clever, irc.freenode.net BOT [14:16] 
nekroze: yeah, it just opens a normal x11 window

[14:17] 
CMCDragonkai: the qemu stuff this is using will use 9plan to mount the host /nix direclty into the guest at runtime

[14:17] 
so the guest has full read-only access to the store

[14:17] 
and by just changing the kernel cmdline when it boots up, it can change which nixos image it boots as a guest

roger.qiu [14:18] 
So I'm a bit confused. If it rebuilds the VM from a config change, doesn't that mean any state mutation would have been lost after rebuilding?


nekroze, irc.freenode.net BOT [14:18] 
Thats why I have been banging my head against the wall trying to figure out how to use it. Thanks again clever!


clever, irc.freenode.net BOT [14:18] 
CMCDragonkai: the qcow file is only the / partition, but it always uses /nix from the host

[14:19] 
so any changes made on the host will take effect next time you boot the vm


nekroze, irc.freenode.net BOT [14:19] 
CMCDragonkai: But with nixos you can have kind of stateless servers that have all state on a 9plan share or nfs share


clever, irc.freenode.net BOT [14:19] 
it saves all state to / which is a qcow image

[14:20] 
i think its a copy-on-write image with /dev/zero as the base image


nekroze, irc.freenode.net BOT [14:20] 
I would think it might make nix-collect-garbage a bit dangerous though.

roger.qiu [14:20] 
Oh, so thr config changes only change stuff in the /nix inside the guest.


clever, irc.freenode.net BOT [14:20] 
nekroze: the build-vm thing stores a symlink of the entire nixos beside the bin script for running it, so it wont GC

roger.qiu [14:20] 
And possibly other bootup parameters? So that way new services can start inside the guest.


clever, irc.freenode.net BOT [14:20] 
nekroze: your safe as long as you dont delete the result symlink while its running


nekroze, irc.freenode.net BOT [14:20] 
clever: Oh so it does follow the guests as dependencies! thats cool.


clever, irc.freenode.net BOT [14:21] 
CMCDragonkai: yeah

[14:21] 
CMCDragonkai: next time you boot the guest, it will use a new derivation for /etc, which will then update all systemd units

[14:21] 
which is almost exactly how a baremetal nixos system updates things


nekroze, irc.freenode.net BOT [14:21] 
Would there be any way to do a switch from inside the guest instead of rebooting? At least for simple package updates.


clever, irc.freenode.net BOT [14:22] 
run the './bin/switch-to-configuration test' that is within nixos output

[14:22] 
that will update /run/current-system and reload systemd units that changed


nekroze, irc.freenode.net BOT [14:23] 
clever: this just keeps getting cooler


clever, irc.freenode.net BOT [14:23] 
i have also done some more crazy things

roger.qiu [14:23] 
But this only works for linux guests

[14:23] 
Right?

[14:23] 
No unikernels... etc


clever, irc.freenode.net BOT [14:23] 
yeah, the switch test one only works on a nixos guest


nekroze, irc.freenode.net BOT [14:23] 
CMCDragonkai: just for nix guests


clever, irc.freenode.net BOT [14:24] 
i ran './bin/switch-to-configuration boot' on a gentoo system

[14:24] 
that overwrote the gentoo bootloader with the nixos bootloader, and configured the system to boot nixos

[14:27] 
as for why i ran switch boot on something not nixos, it was a netbook, you cant just boot the install CD when you dont have a cd-drive

roger.qiu [14:29] 
Install USB?

[14:29] 
By the way, why 9p?


clever, irc.freenode.net BOT [14:30] 
i could have done usb, but it was more fun to try something new

[14:31] 
i think 9p is the only thing qemu+linux support, for mounting the host to the guest with full symlink/hardlink/uid support


nekroze, irc.freenode.net BOT [14:32] 
CMCDragonkai: Its supposed to be very fast as well.


nekroze, irc.freenode.net BOT [14:43] 
clever: One final question about that vm.nix stuff if I may. You say that I could use it in a systemd service definition with "cd /vm-storage; ${machine1}/bin/run-amd-nixos-vm". Does that amd part of that run command matter or does it just specify a 64bit vm? Are there other options?


clever, irc.freenode.net BOT [14:48] 
nekroze: thats the hostname for the guest

[14:48] 
nekroze: the build-vm stuff will name it after the networking.hostname set inside the guest


nekroze, irc.freenode.net BOT [14:51] 
clever: so how can I predict the name. which part of /bin/run-amd-nixos-vm changes based on the hostname?


clever, irc.freenode.net BOT [14:52] 
$out/bin/run-${vmName}-vm

[14:52] 
https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix#L477


nekroze, irc.freenode.net BOT [14:54] 
clever: Ah so for your example it might be /bin/run-example.nixos.org-vm

[14:54] 
yeah?


clever, irc.freenode.net BOT [14:55] 
nekroze: yeah

[14:55] 
some of the guides i saw just do ./result/bin/run-*-vm


nekroze, irc.freenode.net BOT [14:56] 
clever: Yeah I was just thinking that

[14:56] 
clever: Where did you find guides for this stuff nothing I could find was on topic


clever, irc.freenode.net BOT [14:56] 
dont remember exactly


nekroze, irc.freenode.net BOT [14:56] 
clever: Once again, cheers for all the help mate.

[14:57] 
Have a good one all


clever, irc.freenode.net BOT [14:57] 
yep

roger.qiu [15:04] 
This is a great feature for again local computer configuration!

routercfg.nix

{ config, pkgs, ... }:

{
  imports = [ ./default.nix /root/nixcfg/core.nix ];
  services = {
    dhcpd = {
      enable = true;
      interfaces = [ "eth0" ];
      machines = [
        { hostName = "nix1"; ethernetAddress = "52:54:00:12:34:01"; ipAddress = "192.168.3.11"; }
        { hostName = "nix2"; ethernetAddress = "52:54:00:12:34:02"; ipAddress = "192.168.3.12"; }
      ];
      extraConfig = ''
        subnet 192.168.3.0 netmask 255.255.255.0 {
          option subnet-mask 255.255.255.0;
          option broadcast-address 192.168.3.255;
          option routers 192.168.3.1;
          option domain-name-servers 192.168.2.61;
          range 192.168.3.100 192.168.3.200;
        }
      '';
    };
    xserver = {
      enable = true;
      displayManager.slim = {
        enable = true;
        autoLogin = true;
        defaultUser = "clever";
      };
      desktopManager.xfce = {
        enable = true;
      };
    };
  };
  environment.systemPackages = with pkgs; [ chromium wget nmap ];
  networking = {
    hostName = "router";
    interfaces.eth0 = {
      ipAddress = "192.168.3.1";
      prefixLength = 24;
    };
#    nat = {
#      enable = true;
#      externalInterface = "eth0";
#      internalIPs = [ "192.168.3.0/24" ];
#      internalInterfaces = [ "eth1" ];
#    };
  };
  virtualisation = {
    memorySize = 1024;
    qemu.networkingOptions = [
      "-net nic,vlan=0,macaddr=52:54:00:12:34:00"
      "-net vde,vlan=0,sock=/run/vde.ctl"
    ];
  };
}

vm.nix

{ system ? builtins.currentSystem }:

let
  loadcfg = cfgfile: { config, pkgs, ...}: {
    imports = [ <nixos/modules/virtualisation/qemu-vm.nix> cfgfile ];
    config = {
      networking.extraHosts = ''
        192.168.3.11 nixos.example.com
        192.168.3.12 nixos2.example.com
      '';
      virtualisation = {
      };
      users.extraUsers.root.password = "root";
    };
  };
  mkcfg = cfgfile:
    import <nixos/lib/eval-config.nix> {
      inherit system;
      modules = [ (loadcfg cfgfile) ];
    };
in {
  router = (mkcfg ./routercfg.nix).config.system.build.vm;
  master = (mkcfg ./configuration.nix).config.system.build.vm;
  slave = (mkcfg ./configuration2.nix).config.system.build.vm;
  tox = (mkcfg ./tox.nix).config.system.build.vm;
}