DavidS/00_README.md

## 00_README.md

      
    Raw
  

              00_README.md
            
          
    Draft for new type and provider API
Hi *,
The type and provider API has been the bane of my existence since I started writing native resources. Now, finally, we'll do something about it. I'm currently working on designing a nicer API for types and providers. My primary goals are to provide a smooth and simple ruby developer experience for both scripters and coders. Secondary goals were to eliminate server side code, and make puppet 4 data types available. Currently this is completely aspirational (i.e. no real code has been written), but early private feedback was encouraging.
To showcase my vision, this gist has the apt_key type and provider ported over to my proposal. The second example there is a more long-term teaser on what would become possible with such an API.
The new API, like the existing, has two parts: the implementation that interacts with the actual resources, a.k.a. the provider, and information about what the implementation is all about. Due to the different usage patterns of the two parts, they need to be passed to puppet in two different calls:
The Puppet::SimpleResource.implement() call receives the current_state = get() and set(current_state, target_state, noop) methods. get returns a list of discovered resources, while set takes the target state and enforces those goals on the subject. There is only a single (ruby) object throughout an agent run, that can easily do caching and what ever else is required for a good functioning of the provider. The state descriptions passed around are simple lists of key/value hashes describing resources. This will allow the implementation wide latitude in how to organise itself for simplicity and efficiency.
The Puppet::SimpleResource.define() call provides a data-only description of the Type. This is all that is needed on the server side to compile a manifest. Thanks to puppet 4 data type checking, this will already be much more strict (with less effort) than possible with the current APIs, while providing more automatically readable documentation about the meaning of the attributes.
Details in no particular order:


All of this should fit on any unmodified puppet4 installation. It is completely additive and optional. Currently.


The Type definition

It is data-only.
Refers to puppet data types.
No code runs on the server.
This information can be re-used in all tooling around displaying/working with types (e.g. puppet-strings, console, ENC, etc.).
autorelations are restricted to unmodified attribute values and constant values.
No more validate or munge! For the edge cases not covered by data types, runtime checking can happen in the implementation on the agent. There it can use local system state (e.g. different mysql versions have different max table length constraints), and it will only fail the part of the resource tree, that is dependent on this error. There is already ample precedent for runtime validation, as most remote resources do not try to replicate the validation their target is already doing anyways.
It maps 1:1 to the capabilities of PCore, and is similar to the libral interface description (see libral#1). This ensures future interoperability between the different parts of the ecosystem.
Related types can share common attributes by sharing/merging the attribute hashes.
defaults, read_only, and similar data about attributes in the definition are mostly aesthetic at the current point in time, but will make for better documentation, and allow more intelligence built on top of this later.


The implementation are two simple functions current_state = get(), and set(current_state, target_state, noop).

get on its own is already useful for many things, like puppet resource.
set receives the current state from get. While this is necessary for proper operation, there is a certain race condition there, if the system state changes between the calls. This is no different than what current implementations face, and they are well-equipped to deal with this.
set is called with a list of resources, and can do batching if it is beneficial. This is not yet supported by the agent.
the current_state and target_state values are lists of simple data structures built up of primitives like strings, numbers, hashes and arrays. They match the schema defined in the type.
Calling r.set(r.get, r.get) would ensure the current state. This should run without any changes, proving the idempotency of the implementation.
The ruby instance hosting the get and set functions is only alive for the duration of an agent transaction. An implementation can provide a initialize method to read credentials from the system, and setup other things as required. The single instance is used for all instances of the resource.
There is no direct dependency on puppet core libraries in the implementation.

While implementations can use utility functions, they are completely optional.
The dependencies on the logger, commands, and similar utilities can be supplied by a small utility library (TBD).


Having a well-defined small API makes remoting, stacking, proxying, batching, interactive use, and other shenanigans possible, which will make for a interesting time ahead.


The logging of updates to the transaction is only a sketch. See the usage of logger throughout the example. I've tried different styles for fit.

the logger is the primary way of reporting back information to the log, and the report.
results can be streamed for immediate feedback
block-based constructs allow detailed logging with little code ("Started X", "X: Doing Something", "X: Success|Failure", with one or two calls, and only one reference to X)


Obviously this is not sufficient to cover everything existing types and providers are able to do. For the first iteration we are choosing simplicity over functionality.

Generating more resource instances for the catalog during compilation (e.g. file#recurse or concat) becomes impossible with a pure data-driven Type. There is still space in the API to add server-side code.
Some resources (e.g. file, ssh_authorized_keys, concat) cannot or should not be prefetched. While it might not be convenient, a provider could always return nothing on the get() and do a more customized enforce motion in the set().
With current puppet versions, only "native" data types will be supported, as type aliases do not get pluginsynced. Yet.
With current puppet versions, puppet resource can't load the data types, and therefore will not be able to take full advantage of this. Yet.


There is some "convenient" infrastructure (e.g. parsedfile) that needs porting over to this model.


Testing becomes possible on a completely new level. The test library can know how data is transformed outside the API, and - using the shape of the type - start generating test cases, and checking the actions of the implementation. This will require developer help to isolate the implementation from real systems, but it should go a long way towards reducing the tedium in writing tests.


What do you think about this?
Cheers, David

  
## 10_apt_key_type.rb
Puppet::SimpleResource.define(
    name: 'apt_key',
    docs: <<-EOS,
      This type provides Puppet with the capabilities to manage GPG keys needed
      by apt to perform package validation. Apt has it's own GPG keyring that can
      be manipulated through the `apt-key` command.

      apt_key { '6F6B15509CF8E59E6E469F327F438280EF8D349F':
        source => 'http://apt.puppetlabs.com/pubkey.gpg'
      }

      **Autorequires**:
      If Puppet is given the location of a key file which looks like an absolute
      path this type will autorequire that file.
    EOS
    attributes:   {
        ensure:      {
            type: 'Enum[present, absent]',
            docs: 'Whether this apt key should be present or absent on the target system.'
        },
        id:          {
            type:    'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]',
            docs:    'The ID of the key you want to manage.',
            namevar: true,
        },
        content:     {
            type: 'Optional[String]',
            docs: 'The content of, or string representing, a GPG key.',
        },
        source:      {
            type: 'Variant[Stdlib::Absolutepath, Pattern[/\A(https?|ftp):\/\//]]',
            docs: 'Location of a GPG key file, /path/to/file, ftp://, http:// or https://',
        },
        server:      {
            type:    'Pattern[/\A((hkp|http|https):\/\/)?([a-z\d])([a-z\d-]{0,61}\.)+[a-z\d]+(:\d{2,5})?$/]',
            docs:    'The key server to fetch the key from based on the ID. It can either be a domain name or url.',
            default: 'keyserver.ubuntu.com'
        },
        options:     {
            type: 'Optional[String]',
            docs: 'Additional options to pass to apt-key\'s --keyserver-options.',
        },
        fingerprint: {
            type:      'String',
            docs:      'The 40-digit hexadecimal fingerprint of the specified GPG key.',
            read_only: true,
        },
        long:        {
            type:      'String',
            docs:      'The 16-digit hexadecimal id of the specified GPG key.',
            read_only: true,
        },
        short:       {
            type:      'String',
            docs:      'The 8-digit hexadecimal id of the specified GPG key.',
            read_only: true,
        },
        expired:     {
            type:      'Boolean',
            docs:      'Indicates if the key has expired.',
            read_only: true,
        },
        expiry:      {
            type:      'String',
            docs:      'The date the key will expire, or nil if it has no expiry date, in ISO format.',
            read_only: true,
        },
        size:        {
            type:      'String',
            docs:      'The key size, usually a multiple of 1024.',
            read_only: true,
        },
        type:        {
            type:      'String',
            docs:      'The key type, one of: rsa, dsa, ecc, ecdsa.',
            read_only: true,
        },
        created:     {
            type:      'String',
            docs:      'Date the key was created, in ISO format.',
            read_only: true,
        },
    },
    autorequires: {
        file:    '$source', # will evaluate to the value of the `source` attribute
        package: 'apt',
    },
)

## 20_apt_key_provider.rb

Puppet::SimpleResource.implement('apt_key') do
  commands apt_key: 'apt-key'
  commands gpg: '/usr/bin/gpg'

  def get
    cli_args   = %w(adv --list-keys --with-colons --fingerprint --fixed-list-mode)
    key_output = apt_key(cli_args).encode('UTF-8', 'binary', :invalid => :replace, :undef => :replace, :replace => '')
    pub_line   = nil
    fpr_line   = nil

    key_output.split("\n").collect do |line|
      if line.start_with?('pub')
        pub_line = line
      elsif line.start_with?('fpr')
        fpr_line = line
      end

      next unless (pub_line and fpr_line)

      result   = key_line_to_hash(pub_line, fpr_line)

      # reset everything
      pub_line = nil
      fpr_line = nil

      result
    end.compact!
  end

  def self.key_line_to_hash(pub_line, fpr_line)
    pub_split = pub_line.split(':')
    fpr_split = fpr_line.split(':')

    # set key type based on types defined in /usr/share/doc/gnupg/DETAILS.gz
    key_type  = case pub_split[3]
                  when '1'
                    :rsa
                  when '17'
                    :dsa
                  when '18'
                    :ecc
                  when '19'
                    :ecdsa
                  else
                    :unrecognized
                end

    fingerprint = fpr_split.last
    expiry      = pub_split[6].empty? ? nil : Time.at(pub_split[6].to_i)

    {
        name:        fingerprint,
        ensure:      'present',
        fingerprint: fingerprint,
        long:        fingerprint[-16..-1], # last 16 characters of fingerprint
        short:       fingerprint[-8..-1], # last 8 characters of fingerprint
        size:        pub_split[2],
        type:        key_type,
        created:     Time.at(pub_split[5].to_i),
        expiry:      expiry,
        expired:     !!(expiry && Time.now >= expiry),
    }
  end

  def set(current_state, target_state, noop = false)
    existing_keys = Hash[current_state.collect { |k| [k[:name], k] }]
    target_state.each do |key|
      logger.warning(key[:name], 'The id should be a full fingerprint (40 characters) to avoid collision attacks, see the README for details.') if key[:name].length < 40
      if key[:source] and key[:content]
        logger.fail(key[:name], 'The properties content and source are mutually exclusive')
        next
      end

      current = existing_keys[k[:name]]
      if current && key[:ensure].to_s == 'absent'
        logger.deleting(key[:name]) do
          begin
            apt_key('del', key[:short], noop: noop)
            r = execute(["#{command(:apt_key)} list | grep '/#{resource.provider.short}\s'"], :failonfail => false)
          end while r.exitstatus == 0
        end
      elsif current && key[:ensure].to_s == 'present'
        # No updating implemented
        # update(key, noop: noop)
      elsif !current && key[:ensure].to_s == 'present'
        create(key, noop: noop)
      end
    end
  end

  def create(key, noop = false)
    logger.creating(key[:name]) do |logger|
      if key[:source].nil? and key[:content].nil?
        # Breaking up the command like this is needed because it blows up
        # if --recv-keys isn't the last argument.
        args = ['adv', '--keyserver', key[:server]]
        if key[:options]
          args.push('--keyserver-options', key[:options])
        end
        args.push('--recv-keys', key[:id])
        apt_key(*args, noop: noop)
      elsif key[:content]
        temp_key_file(key[:content], logger) do |key_file|
          apt_key('add', key_file, noop: noop)
        end
      elsif key[:source]
        key_file = source_to_file(key[:source])
        apt_key('add', key_file.path, noop: noop)
        # In case we really screwed up, better safe than sorry.
      else
        logger.fail("an unexpected condition occurred while trying to add the key: #{key[:id]} (content: #{key[:content].inspect}, source: #{key[:source].inspect})")
      end
    end
  end

  # This method writes out the specified contents to a temporary file and
  # confirms that the fingerprint from the file, matches the long key that is in the manifest
  def temp_key_file(key, logger)
    file = Tempfile.new('apt_key')
    begin
      file.write key[:content]
      file.close
      if name.size == 40
        if File.executable? command(:gpg)
          extracted_key = execute(["#{command(:gpg)} --with-fingerprint --with-colons #{file.path} | awk -F: '/^fpr:/ { print $10 }'"], :failonfail => false)
          extracted_key = extracted_key.chomp

          unless extracted_key.match(/^#{name}$/)
            logger.fail("The id in your manifest #{key[:name]} and the fingerprint from content/source do not match. Please check there is not an error in the id or check the content/source is legitimate.")
          end
        else
          logger.warning('/usr/bin/gpg cannot be found for verification of the id.')
        end
      end
      yield file.path
    ensure
      file.close
      file.unlink
    end
  end
end

## 30_api_proxy_type.rb
Puppet::SimpleResource.define(
    name:       'iis_application_pool',
    docs:       'Manage an IIS application pool through a powershell proxy.',
    attributes: {
        ensure:                {
            type: 'Enum[present, absent]',
            docs: 'Whether this ApplicationPool should be present or absent on the target system.'
        },
        name:                  {
            type:    'String',
            docs:    'The name of the ApplicationPool.',
            namevar: true,
        },
        state:                 {
            type:    'Enum[running, stopped]',
            docs:    'The state of the ApplicationPool.',
            default: 'running',
        },
        managedpipelinemode:   {
            type: 'String',
            docs: 'The managedPipelineMode of the ApplicationPool.',
        },
        managedruntimeversion: {
            type: 'String',
            docs: 'The managedRuntimeVersion of the ApplicationPool.',
        },
    }
)

## 40_api_proxy_provider.rb
Puppet::SimpleResource.implement('iis_application_pool') do
  # hiding all the nasty bits
  require 'puppet/provider/iis_powershell'
  include Puppet::Provider::IIS_PowerShell

  def get
    # call out to PowerShell to talk to the API
    result = run('fetch_application_pools.ps1', logger)

    # returns an array of hashes with data according to the schema above
    JSON.parse(result)
  end

  def set(current_state, target_state, noop = false)
    # call out to PowerShell to talk to the API
    result = run('enforce_application_pools.ps1', JSON.generate(current_state), JSON.generate(target_state), logger, noop)

    # returns an array of hashes with status data from the changes
    JSON.parse(result)
  end

end
	Puppet::SimpleResource.define(
	name: 'apt_key',
	docs: <<-EOS,
	This type provides Puppet with the capabilities to manage GPG keys needed
	by apt to perform package validation. Apt has it's own GPG keyring that can
	be manipulated through the `apt-key` command.

	apt_key { '6F6B15509CF8E59E6E469F327F438280EF8D349F':
	source => 'http://apt.puppetlabs.com/pubkey.gpg'
	}

	Autorequires:
	If Puppet is given the location of a key file which looks like an absolute
	path this type will autorequire that file.
	EOS
	attributes: {
	ensure: {
	type: 'Enum[present, absent]',
	docs: 'Whether this apt key should be present or absent on the target system.'
	},
	id: {
	type: 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]',
	docs: 'The ID of the key you want to manage.',
	namevar: true,
	},
	content: {
	type: 'Optional[String]',
	docs: 'The content of, or string representing, a GPG key.',
	},
	source: {
	type: 'Variant[Stdlib::Absolutepath, Pattern[/\A(https?\|ftp):\/\//]]',
	docs: 'Location of a GPG key file, /path/to/file, ftp://, http:// or https://',
	},
	server: {
	type: 'Pattern[/\A((hkp\|http\|https):\/\/)?([a-z\d])([a-z\d-]{0,61}\.)+[a-z\d]+(:\d{2,5})?$/]',
	docs: 'The key server to fetch the key from based on the ID. It can either be a domain name or url.',
	default: 'keyserver.ubuntu.com'
	},
	options: {
	type: 'Optional[String]',
	docs: 'Additional options to pass to apt-key\'s --keyserver-options.',
	},
	fingerprint: {
	type: 'String',
	docs: 'The 40-digit hexadecimal fingerprint of the specified GPG key.',
	read_only: true,
	},
	long: {
	type: 'String',
	docs: 'The 16-digit hexadecimal id of the specified GPG key.',
	read_only: true,
	},
	short: {
	type: 'String',
	docs: 'The 8-digit hexadecimal id of the specified GPG key.',
	read_only: true,
	},
	expired: {
	type: 'Boolean',
	docs: 'Indicates if the key has expired.',
	read_only: true,
	},
	expiry: {
	type: 'String',
	docs: 'The date the key will expire, or nil if it has no expiry date, in ISO format.',
	read_only: true,
	},
	size: {
	type: 'String',
	docs: 'The key size, usually a multiple of 1024.',
	read_only: true,
	},
	type: {
	type: 'String',
	docs: 'The key type, one of: rsa, dsa, ecc, ecdsa.',
	read_only: true,
	},
	created: {
	type: 'String',
	docs: 'Date the key was created, in ISO format.',
	read_only: true,
	},
	},
	autorequires: {
	file: '$source', # will evaluate to the value of the `source` attribute
	package: 'apt',
	},
	)

	Puppet::SimpleResource.implement('apt_key') do
	commands apt_key: 'apt-key'
	commands gpg: '/usr/bin/gpg'

	def get
	cli_args = %w(adv --list-keys --with-colons --fingerprint --fixed-list-mode)
	key_output = apt_key(cli_args).encode('UTF-8', 'binary', :invalid => :replace, :undef => :replace, :replace => '')
	pub_line = nil
	fpr_line = nil

	key_output.split("\n").collect do \|line\|
	if line.start_with?('pub')
	pub_line = line
	elsif line.start_with?('fpr')
	fpr_line = line
	end

	next unless (pub_line and fpr_line)

	result = key_line_to_hash(pub_line, fpr_line)

	# reset everything
	pub_line = nil
	fpr_line = nil

	result
	end.compact!
	end

	def self.key_line_to_hash(pub_line, fpr_line)
	pub_split = pub_line.split(':')
	fpr_split = fpr_line.split(':')

	# set key type based on types defined in /usr/share/doc/gnupg/DETAILS.gz
	key_type = case pub_split[3]
	when '1'
	:rsa
	when '17'
	:dsa
	when '18'
	:ecc
	when '19'
	:ecdsa
	else
	:unrecognized
	end

	fingerprint = fpr_split.last
	expiry = pub_split[6].empty? ? nil : Time.at(pub_split[6].to_i)

	{
	name: fingerprint,
	ensure: 'present',
	fingerprint: fingerprint,
	long: fingerprint[-16..-1], # last 16 characters of fingerprint
	short: fingerprint[-8..-1], # last 8 characters of fingerprint
	size: pub_split[2],
	type: key_type,
	created: Time.at(pub_split[5].to_i),
	expiry: expiry,
	expired: !!(expiry && Time.now >= expiry),
	}
	end

	def set(current_state, target_state, noop = false)
	existing_keys = Hash[current_state.collect { \|k\| [k[:name], k] }]
	target_state.each do \|key\|
	logger.warning(key[:name], 'The id should be a full fingerprint (40 characters) to avoid collision attacks, see the README for details.') if key[:name].length < 40
	if key[:source] and key[:content]
	logger.fail(key[:name], 'The properties content and source are mutually exclusive')
	next
	end

	current = existing_keys[k[:name]]
	if current && key[:ensure].to_s == 'absent'
	logger.deleting(key[:name]) do
	begin
	apt_key('del', key[:short], noop: noop)
	r = execute(["#{command(:apt_key)} list \| grep '/#{resource.provider.short}\s'"], :failonfail => false)
	end while r.exitstatus == 0
	end
	elsif current && key[:ensure].to_s == 'present'
	# No updating implemented
	# update(key, noop: noop)
	elsif !current && key[:ensure].to_s == 'present'
	create(key, noop: noop)
	end
	end
	end

	def create(key, noop = false)
	logger.creating(key[:name]) do \|logger\|
	if key[:source].nil? and key[:content].nil?
	# Breaking up the command like this is needed because it blows up
	# if --recv-keys isn't the last argument.
	args = ['adv', '--keyserver', key[:server]]
	if key[:options]
	args.push('--keyserver-options', key[:options])
	end
	args.push('--recv-keys', key[:id])
	apt_key(*args, noop: noop)
	elsif key[:content]
	temp_key_file(key[:content], logger) do \|key_file\|
	apt_key('add', key_file, noop: noop)
	end
	elsif key[:source]
	key_file = source_to_file(key[:source])
	apt_key('add', key_file.path, noop: noop)
	# In case we really screwed up, better safe than sorry.
	else
	logger.fail("an unexpected condition occurred while trying to add the key: #{key[:id]} (content: #{key[:content].inspect}, source: #{key[:source].inspect})")
	end
	end
	end

	# This method writes out the specified contents to a temporary file and
	# confirms that the fingerprint from the file, matches the long key that is in the manifest
	def temp_key_file(key, logger)
	file = Tempfile.new('apt_key')
	begin
	file.write key[:content]
	file.close
	if name.size == 40
	if File.executable? command(:gpg)
	extracted_key = execute(["#{command(:gpg)} --with-fingerprint --with-colons #{file.path} \| awk -F: '/^fpr:/ { print $10 }'"], :failonfail => false)
	extracted_key = extracted_key.chomp

	unless extracted_key.match(/^#{name}$/)
	logger.fail("The id in your manifest #{key[:name]} and the fingerprint from content/source do not match. Please check there is not an error in the id or check the content/source is legitimate.")
	end
	else
	logger.warning('/usr/bin/gpg cannot be found for verification of the id.')
	end
	end
	yield file.path
	ensure
	file.close
	file.unlink
	end
	end
	end
	Puppet::SimpleResource.define(
	name: 'iis_application_pool',
	docs: 'Manage an IIS application pool through a powershell proxy.',
	attributes: {
	ensure: {
	type: 'Enum[present, absent]',
	docs: 'Whether this ApplicationPool should be present or absent on the target system.'
	},
	name: {
	type: 'String',
	docs: 'The name of the ApplicationPool.',
	namevar: true,
	},
	state: {
	type: 'Enum[running, stopped]',
	docs: 'The state of the ApplicationPool.',
	default: 'running',
	},
	managedpipelinemode: {
	type: 'String',
	docs: 'The managedPipelineMode of the ApplicationPool.',
	},
	managedruntimeversion: {
	type: 'String',
	docs: 'The managedRuntimeVersion of the ApplicationPool.',
	},
	}
	)
	Puppet::SimpleResource.implement('iis_application_pool') do
	# hiding all the nasty bits
	require 'puppet/provider/iis_powershell'
	include Puppet::Provider::IIS_PowerShell

	def get
	# call out to PowerShell to talk to the API
	result = run('fetch_application_pools.ps1', logger)

	# returns an array of hashes with data according to the schema above
	JSON.parse(result)
	end

	def set(current_state, target_state, noop = false)
	# call out to PowerShell to talk to the API
	result = run('enforce_application_pools.ps1', JSON.generate(current_state), JSON.generate(target_state), logger, noop)

	# returns an array of hashes with status data from the changes
	JSON.parse(result)
	end

	end