nixops-notes.md

Notes from a NixOps beginner

by Matan Shenhav (ixxie - m.shenhav@gmail.com)

2020/07/01 - while work towards NixOps 2.0 is being committed to master

First impressions

I first looked into NixOps a few years ago, and only very briefly. Trying to get back into it again now, there are some logistical obstacles.

Figuring out the status of the project

If I didn't chat with Graham on #nixops I wouldn't have know about ongoing work to upgrade the plugin system. Not sure there is a smooth way to Nix users to keep track of the latest news, but if there is I'm missing out on it.

Finding the documentation

1. Docs disappeared from nixos.org navigation;
2. Docs linked in the github README are outdated;
3. Eventually, gchristiansen linked me the updated docs.

nixops --help

1. The CLI's help is quite a good overview of the workflow;
2. Since I did get through this workflow a few years ago, it was easy to get back to it with this as reference;
3. But would be nice to have that reflected in the new docs.

Trying to create a Digital Ocean deployment with NixOps

Finding an example of using NixOps with the Digital Ocean plugin

1. Surprisingly hard;
2. Got the answer from typetetris on #nixops: a discourse link;
3. The snippet in the thread was a bit hard to follow;
4. The link to tomberek's example of a NixOps DO deployment was very illuminating;

Trying to create a simple Digital Ocean deployment

From here, I made a simplified version of tomberek's deployment:

{
  resources.sshKeyPairs.ssh-key = {};

  test1 = { config, pkgs, lib, ... }: {
    #imports = [ ./xserver.nix ];
    deployment = {
        targetEnv = "digitalOcean";
        digitalOcean = {
            name = "test1";
            region = "fra1";
            size = "1gb";
            authToken = builtins.readFile ./secrets/DO_token.txt;
        };
    };

    # System packages installed
    environment.systemPackages = with pkgs; [
      openssh openssl vim tree gitAndTools.git 
    ];

    # key-based access only
    services.openssh = {
      enable = true;
      challengeResponseAuthentication = false;
      passwordAuthentication = false;
    };

    users.users.ixxie = {
      isNormalUser = true;
      group = "wheel";
      openssh.authorizedKeys.keys = lib.splitString "\\n" (builtins.getEnv "authkeys") ++ [
        "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOm2JiPs6geaZ+coOju+kpUIbaJkLOnydTGcPc+K4V5ksqkqDW2i2fPjZdV3U8Eihv+wUmyYkj5SU+Q75JYy1/0oKwWQi2SX9EqrSsK/JOryex8FmqwhKwm7+afrryILCOJyhhNGeKOm04stxY50UDSrCmOSpyX15PZnMPB6BRuWdiWi3jvGwja2+lFwtKlIJuYooBFCAE7R7buqHgduhvtoLWTh8sLRiKDo9vP7s63qyXmvCx7tY06lSD3V65rRBd6SjA8mqHQZN9RL0RgJry65HVMIE2BapniLeUJi2L32hvttstvkj2PMA0Obm+bxlimKSSXZkTRPoxC/p3tWy7 ixxie@meso"
      ];
    };
    users.mutableUsers = false;
    
  };
}

For some reason, still unknown to me, this failed with the following error:

[ixxie@meso:~/mesoconfig]$ nixops deploy
test1..> creating droplet ...Traceback (most recent call last):
  File "/nix/store/waqv6scxpz6kqiq4vsnq32nbw9y07fny-nixops-1.7/bin/..nixops-wrapped-wrapped", line 991, in <module>
    args.op()
  File "/nix/store/waqv6scxpz6kqiq4vsnq32nbw9y07fny-nixops-1.7/bin/..nixops-wrapped-wrapped", line 412, in op_deploy
    max_concurrent_activate=args.max_concurrent_activate)
  File "/nix/store/waqv6scxpz6kqiq4vsnq32nbw9y07fny-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1063, in deploy
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/nix/store/waqv6scxpz6kqiq4vsnq32nbw9y07fny-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1052, in run_with_notify
    f()
  File "/nix/store/waqv6scxpz6kqiq4vsnq32nbw9y07fny-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 1063, in <lambda>
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/nix/store/waqv6scxpz6kqiq4vsnq32nbw9y07fny-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 996, in _deploy
    nixops.parallel.run_tasks(nr_workers=-1, tasks=self.active_resources.itervalues(), worker_fun=worker)
  File "/nix/store/waqv6scxpz6kqiq4vsnq32nbw9y07fny-nixops-1.7/lib/python2.7/site-packages/nixops/parallel.py", line 44, in thread_fun
    result_queue.put((worker_fun(t), None, t.name))
  File "/nix/store/waqv6scxpz6kqiq4vsnq32nbw9y07fny-nixops-1.7/lib/python2.7/site-packages/nixops/deployment.py", line 969, in worker
    r.create(self.definitions[r.name], check=check, allow_reboot=allow_reboot, allow_recreate=allow_recreate)
  File "/nix/store/waqv6scxpz6kqiq4vsnq32nbw9y07fny-nixops-1.7/lib/python2.7/site-packages/nixops/backends/digital_ocean.py", line 151, in create
    droplet.create()
  File "/nix/store/c8janp4fmr2yj2rx4lv4hsnypxgw8d6a-python2.7-python-digitalocean-1.15.0/lib/python2.7/site-packages/digitalocean/Droplet.py", line 545, in create
    self.name)
  File "/nix/store/c8janp4fmr2yj2rx4lv4hsnypxgw8d6a-python2.7-python-digitalocean-1.15.0/lib/python2.7/site-packages/digitalocean/Droplet.py", line 511, in __get_ssh_keys_id_or_fingerprint
    results = key.load_by_pub_key(ssh_key)
  File "/nix/store/c8janp4fmr2yj2rx4lv4hsnypxgw8d6a-python2.7-python-digitalocean-1.15.0/lib/python2.7/site-packages/digitalocean/SSHKey.py", line 51, in load_by_pub_key
    data = self.get_data("account/keys/")
  File "/nix/store/c8janp4fmr2yj2rx4lv4hsnypxgw8d6a-python2.7-python-digitalocean-1.15.0/lib/python2.7/site-packages/digitalocean/baseapi.py", line 193, in get_data
    req = self.__perform_request(url, type, params)
  File "/nix/store/c8janp4fmr2yj2rx4lv4hsnypxgw8d6a-python2.7-python-digitalocean-1.15.0/lib/python2.7/site-packages/digitalocean/baseapi.py", line 90, in __perform_request
    raise TokenError("No token provided. Please use a valid token")
digitalocean.TokenError: No token provided. Please use a valid token

As far as I know the token is valid.

Finding the Digital Ocean plugin

My next step was to try and figure out if the DO plugin docs could help me debug this.

1. Googling for the DO plugin wasn't easy;
2. I got the answer from gchristiansen on #nixops;
3. gchristiansen pointed out this plugin should be updated to 2.0 format.

Upgrading the Digital Ocean Plugin to 2.0 Format

gchristiansen shared the authoring instructions from the new docs. These are a pretty good start!

You can see this work in the PR for the upgrade.

Packaging with Poetry and Poety2Nix

From setup.py to pyproject.toml

1. Some kind of @version@ templating was used in the setup.py and it was unclear whether / how to port into pyproject.toml;
2. It seemed to be built by release.nix and I thought I should keep it to maintain continuity in the plugin;
3. gchristiansen clarified that release.nix needs to be deleted and the version hardcoded; I infered from release.nix that plugin versioning should follow corresponding NixOps version;
4. Since the version which master will become is implicit, I had to ask gchristiansen what it will be; he answered 2.0.
5. There are other lines in setup.py for which it is not obvious whether actions needs to be take: packages, packages_data, entrypoints and py_modules to name a few;
6. I shared a gist with gchristiansen and he said it looked okay, so I proceeded with next steps assuming this will get tested later anyway.

Important Notes

This section seems a step in itself, so consider renaming it for clarity.

Updating resource subclasses

1. The example in the authoring page refers to the ResourceDefinition and ResourceOptions classes, but the plugin uses the MachineDefinition and MachineOptionss classes; I had to check this was not a renaming, but it was easy to find Machine is a type of Resource.
2. MachineState was not mentioned in the authoring page so I didn't really know what to do with it. It thought maybe it mapped to MachineOptions but gchristiansen clarified that MachineDefinition is modified to use a new MachineOptions which MachineState is kept as-is.
3. When modifying MachineDefinition I didn't realize the get_type class method should be kept, since no other methods were shown in the authoring page and I have no reference point (having never developed any NixOps plugin before).
4. The authoring page mentioned XML format, but the Digital Ocean plugin didn't use XML as the basis of its subclasses. Rather, it seemed to leveral nixops.utils functions. This was a bit confusing, but I figured there may be some step in the evolution not covered in the authoring page.

Onwards with Poetry

Creating plugin.py

The plugin I am working on already has a plugin.py in the Python source code directory. This now has the same name as the repo itself, and I am unsure whether to create one at the repo root, or do nothing. I tried both of these, but nixops list-plugins is not showing the nixops-digitalocean plugin as expected.

gschristian clarified this issue is due to the dash in the module name, not permitted by Python. I followed the authoring page instructions to align the module name with the plugin name, but opted to rename the module to follow the plugin name which resulted in the illegal name.

Fiddling with Poetry

At this point I found that I am missing some dependencies, and had to to start familiarizing myself with Poetry. Now, I have previously done some work with Python, but never touched Poetry. Consider adding a link to an intro to Poetry as well as its manual in the NixOps manual and specifically the authoring section, because many people building these plugins have even less knowledge of Python than I do.

And then...

+-------------------+
| Installed Plugins |
+-------------------+
|    digitalocean   |
+-------------------+
(nixops-digitalocean-8Il532fe-py3.7) 

🎉

Getting the plugin into working shape

Even before upgrading to the new architecture, the plugin was broken (see above). The error now appears in the following form, when calling nixops deploy:

Traceback (most recent call last):
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/bin/nixops", line 33, in <module>
    sys.exit(load_entry_point('nixops', 'console_scripts', 'nixops')())
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/__main__.py", line 710, in main
    args.op(args)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/script_defs.py", line 638, in op_deploy
    max_concurrent_activate=args.max_concurrent_activate,
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1431, in deploy
    self.run_with_notify("deploy", lambda: self._deploy(**kwargs))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1420, in run_with_notify
    f()
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1431, in <lambda>
    self.run_with_notify("deploy", lambda: self._deploy(**kwargs))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1220, in _deploy
    self.evaluate_active(include, exclude, kill_obsolete)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1161, in evaluate_active
    self.evaluate()
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 520, in evaluate
    defn = _create_definition(name, cfg, cfg["targetEnv"])
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1749, in _create_definition
    return cls(name, nixops.resources.ResourceEval(config))
  File "/home/ixxie/sparklet/repos/nixops-digitalocean/nixops_digitalocean/backends/digital_ocean.py", line 59, in __init__
    self.auth_token = config.auth_token
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/util.py", line 113, in __getattr__
    return self[key]
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/util.py", line 101, in __getitem__
    return self._dict[key]
KeyError: 'auth_token'

At this point, I am trying to comprehend the Python classes provided by NixOps and used in the nixops-digitalocean plugin: MachineOptions, MachineDefinition and MachineState. It seems most of the complex logic of the plugin lives in the DigitalOceanState class extending MachineState. I find myself wishing for better documentation of the classes critical for writing plugins; even a simple overview of these would go a long way.

I am also having trouble understanding what a typical plugin look's like; the guide tells me about some core files I must create - default.nix, overrides.nix, shell.nix, plugin.py, etc - but what other files are typically there? I notice most plugins have a backends/ directory where all the magic happens. How is this organized? How should I name things? I would love to see a link in the docs to a plugin that is up to date and following best practices, and a short explanation of the structure of a plugin. Talking to gchristiansen, it seems the Packet plugin is a good candidate for a reference implementation, or at least a good example to link in the docs.

Renaming a resource

In order to make the code more readable, I decided (perhaps foolishly early) to rename the backend ambigously called DigitalOcean to be called Droplet; this conforms to DOs own naming of that resource, and to other plugins' convention of using the cloud provider's resource names as the backend name.

After renaming, my test deployment broke:

[ixxie@meso:~/sparklet/repos/sparklet-infra]$ nixops deploy
Traceback (most recent call last):
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/bin/nixops", line 33, in <module>
    sys.exit(load_entry_point('nixops', 'console_scripts', 'nixops')())
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/__main__.py", line 710, in main
    args.op(args)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/script_defs.py", line 613, in op_deploy
    with deployment(args) as depl:
  File "/nix/store/ihy2vly61ndky6qlv1q4dfdiv28vszkh-python3-3.7.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/script_defs.py", line 40, in deployment
    depl = open_deployment(sf, args)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/script_defs.py", line 118, in open_deployment
    depl = sf.open_deployment(uuid=args.deployment)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/statefile.py", line 194, in open_deployment
    deployment = self._find_deployment(uuid=uuid)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/statefile.py", line 188, in _find_deployment
    return nixops.deployment.Deployment(self, res[0][0], sys.stderr)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 112, in __init__
    r = _create_state(self, type, name, id)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1766, in _create_state
    raise nixops.deployment.UnknownBackend("unknown resource type ‘{0}’".format(type))
nixops.deployment.UnknownBackend: unknown resource type ‘digitalOcean’
(nixops-digitalocean-8Il532fe-py3.7) 

And nothing I did could fix it. I can't even delete the original deployment:

[ixxie@meso:~/sparklet/repos/sparklet-infra]$ nixops delete -d 8cc516ee-bad4-11ea-9400-0242f1ef645a
Traceback (most recent call last):
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/bin/nixops", line 33, in <module>
    sys.exit(load_entry_point('nixops', 'console_scripts', 'nixops')())
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/__main__.py", line 710, in main
    args.op(args)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/script_defs.py", line 211, in op_delete
    with one_or_all(args) as depls:
  File "/nix/store/ihy2vly61ndky6qlv1q4dfdiv28vszkh-python3-3.7.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/script_defs.py", line 90, in one_or_all
    yield [open_deployment(sf, args)]
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/script_defs.py", line 118, in open_deployment
    depl = sf.open_deployment(uuid=args.deployment)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/statefile.py", line 194, in open_deployment
    deployment = self._find_deployment(uuid=uuid)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/statefile.py", line 188, in _find_deployment
    return nixops.deployment.Deployment(self, res[0][0], sys.stderr)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 112, in __init__
    r = _create_state(self, type, name, id)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1766, in _create_state
    raise nixops.deployment.UnknownBackend("unknown resource type ‘{0}’".format(type))
nixops.deployment.UnknownBackend: unknown resource type ‘digitalOcean’
(nixops-digitalocean-8Il532fe-py3.7) 

Fiddling with stuff

Now I tried to fiddle with some things to get the deployment working:

• config: MachineOptions - advocated in the authoring page - seemed wrong so I used the derived class, setting config: DropletOptions
• Peeking into the GCE plugin's GCE definition I found properties declared using self.config instead of config so I tried to do this as well.

Something seemed to work, because now I ended up with a type error unrelated to the earlier auth_key key error.

Property type error

The error in question appeared as follows:

[ixxie@meso:~/sparklet/repos/nixops-digitalocean]$ nixops deploy -d sparklet
Traceback (most recent call last):
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/bin/nixops", line 33, in <module>
    sys.exit(load_entry_point('nixops', 'console_scripts', 'nixops')())
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/__main__.py", line 710, in main
    args.op(args)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/script_defs.py", line 638, in op_deploy
    max_concurrent_activate=args.max_concurrent_activate,
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1431, in deploy
    self.run_with_notify("deploy", lambda: self._deploy(**kwargs))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1420, in run_with_notify
    f()
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1431, in <lambda>
    self.run_with_notify("deploy", lambda: self._deploy(**kwargs))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1220, in _deploy
    self.evaluate_active(include, exclude, kill_obsolete)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1161, in evaluate_active
    self.evaluate()
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 520, in evaluate
    defn = _create_definition(name, cfg, cfg["targetEnv"])
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1749, in _create_definition
    return cls(name, nixops.resources.ResourceEval(config))
  File "/home/ixxie/sparklet/repos/nixops-digitalocean/nixops_digitalocean/backends/droplet.py", line 57, in __init__
    super().__init__(name, config)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/backends/__init__.py", line 53, in __init__
    super().__init__(name, config)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/resources/__init__.py", line 50, in __init__
    self.config = config_type(**config)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/util.py", line 167, in __init__
    setattr(self, key, _transform_value(key, value))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/util.py", line 155, in _transform_value
    typeguard.check_type(key, value, ann)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/lib/python3.7/site-packages/typeguard/__init__.py", line 622, in check_type
    format(argname, qualified_name(expected_type), qualified_name(value)))
TypeError: type of region must be str; got NoneType instead
(nixops-digitalocean-8Il532fe-py3.7) 

In an attempt to improve my understanding of the codebase and trace the issue, I started typehinting the plugin and checking it with mypy - see below. However, gchristiansen pointed out the source of this issue was an incorrect structuring of the DropletOptions: it should be nested, much like the Packet plugin does here.

Type checking with mypy

As mentioned in the last section, at this point I realize mypy would be helpful, but I never used it before, so I have to start coming to grips with it. It seems the number one challenge of using it now is figuring out the types NixOps uses.

Looking into the source code, I see some types are explicitly defined; for example, MachineDefinitionType is explicitly defined in the backends module. Should I be creating a derivative DropletDefinitionType or simply use this in my type hints?

Again, mypy is another new technology for me; having links to a gentle introduction would be helpful. In particular, it's not exactly obvious how one goes about typing a Python codebase.

I repeatedly faced errors like TypeError: type of region must be str; got NoneType instead, and solved them by wrapping all types Optional. I am not sure this is the right thing to do, and I feel like maybe I am missing something about how the mypy types hook into Nix types.

In the end, I managed to resolve most issues, but was left with the following:

nixops_digitalocean/backends/droplet.py:150: error: Item "ResourceState[ResourceDefinition]" of "Optional[ResourceState[ResourceDefinition]]" has no attribute "public_key"
nixops_digitalocean/backends/droplet.py:150: error: Item "None" of "Optional[ResourceState[ResourceDefinition]]" has no attribute "public_key"
nixops_digitalocean/backends/droplet.py:157: error: Item "ResourceState[ResourceDefinition]" of "Optional[ResourceState[ResourceDefinition]]" has no attribute "private_key"
nixops_digitalocean/backends/droplet.py:157: error: Item "None" of "Optional[ResourceState[ResourceDefinition]]" has no attribute "private_key"
nixops_digitalocean/backends/droplet.py:203: error: "ResourceState[ResourceDefinition]" has no attribute "public_key"
Found 5 errors in 1 file (checked 9 source files)
(nixops-digitalocean-8Il532fe-py3.7) 

Naming conventions

I found myself wondering about whether there are conventions on how to name the resources and targetEnv in the plugin. I'm wondering whether all assets in the plugin should be prefixed with an abbreviation of the cloud provider - e.g. doDroplet, doVolume, doFloatingIP, etc.

Mystery issubclass issue

Having (hopefully) resolved most other issues with the plugin, I now ran into the following - rather mysterious - error:

[ixxie@meso:~/sparklet/repos/nixops-digitalocean]$ nixops deploy -d sparklet
Traceback (most recent call last):
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/bin/nixops", line 33, in <module>
    sys.exit(load_entry_point('nixops', 'console_scripts', 'nixops')())
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/__main__.py", line 710, in main
    args.op(args)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/script_defs.py", line 638, in op_deploy
    max_concurrent_activate=args.max_concurrent_activate,
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1431, in deploy
    self.run_with_notify("deploy", lambda: self._deploy(**kwargs))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1420, in run_with_notify
    f()
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1431, in <lambda>
    self.run_with_notify("deploy", lambda: self._deploy(**kwargs))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1220, in _deploy
    self.evaluate_active(include, exclude, kill_obsolete)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1161, in evaluate_active
    self.evaluate()
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 527, in evaluate
    name, config["resources"][res_type][name], res_type
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1749, in _create_definition
    return cls(name, nixops.resources.ResourceEval(config))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/resources/ssh_keypair.py", line 21, in __init__
    super().__init__(name, config)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/resources/__init__.py", line 45, in __init__
    if not issubclass(config_type, ResourceOptions):
TypeError: issubclass() arg 1 must be a class
(nixops-digitalocean-8Il532fe-py3.7) 

I happened to discover from the #nixops channel logs that DigitialKiwi had encountered the same issue. Its rather unclear whether this issue stems from the plugin or NixOps itself.

At this point I got some help from adisbladis in #nixops, who recommended I try leveraging the --pdb flag of the nixops CLI (hitherto unfamiliar to me) to figure out what value config_type has. In this shell I discovered the following:

(Pdb) config_type
'ResourceOptions'
(Pdb) type(config_type)
<class 'str'>

Meeting with the NixOps team

In meeting with adisbladis and gchristiansen, gchristiansen intorduced me to the architecture of NixOps and clarified some gaps in my knowledge about the way it worked.

adisbladis also traced the issubclass issue to an unhandeled edgecase in NixOps. Other issues with the plugin were uncovered and fixed by adisbladis.

Cleaning up the PR

I now proceeded to create a new draft PR where I split the previous WIP commit into distinct atomic commits, and rebase adisbladis' commits unto this new branch. Before pushing, I decided to run a rest to make sure everything is kosher, but ran into the following:

[ixxie@meso:~/sparklet/repos/nixops-digitalocean]$ nixops --pdb deploy -d sparklet
error: attribute 'network' missing, at /home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nix/eval-machine-info.nix:51:26
error: evaluation of the deployment specification failed
(nixops-digitalocean-8Il532fe-py3.7) 

adisbladis helped me figure out that I was missing a network.description = "something"; in my deployment.nix. He agreed that since this is not really used anywhere, this requirement should probably be removed.

At this point, the deployment seems to work! 🎉

Anomaly in the deployment

One oddity did pop up; even thought the deployment seems to be perfectly in order, an error appeared:

...
...
test1..> starting the following units: audit.service, kmod-static-nodes.service, network-link-ens3.service, network-local-commands.service, network-setup.service, nix-daemon.socket, nscd.service, systemd-journal-catalog-update.service, systemd-modules-load.service, systemd-sysctl.service, systemd-timesyncd.service, systemd-tmpfiles-clean.timer, systemd-tmpfiles-setup-dev.service, systemd-udev-trigger.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket, systemd-update-done.service
test1..> the following new units were started: resolvconf.service, sys-devices-virtual-misc-tun.device, sys-module-tun.device, systemd-binfmt.service, systemd-coredump.socket
test1..> error: Traceback (most recent call last):
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1000, in worker
    res
Exception: unable to activate new configuration (exit code 4)

Traceback (most recent call last):
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/bin/nixops", line 33, in <module>
    sys.exit(load_entry_point('nixops', 'console_scripts', 'nixops')())
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/__main__.py", line 710, in main
    args.op(args)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/script_defs.py", line 637, in op_deploy
    max_concurrent_activate=args.max_concurrent_activate,
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1458, in deploy
    self.run_with_notify("deploy", lambda: self._deploy(**kwargs))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1447, in run_with_notify
    f()
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1458, in <lambda>
    self.run_with_notify("deploy", lambda: self._deploy(**kwargs))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1405, in _deploy
    max_concurrent_activate=max_concurrent_activate,
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1046, in activate_configs
    ", ".join(["‘{0}’".format(x) for x in failed]),
Exception: activation of 1 of 1 machines failed (namely on ‘test1’)
(nixops-digitalocean-8Il532fe-py3.7) 

I managed to reproduce this error.