Collecting duplicate resources in puppet

I could probably write a long design article explaining why identical duplicate resources should be allowed [1] in puppet. If puppet is going to survive in the long-term, they will have to build in this feature. In the short-term, I will have to hack around deficiency. As luck would have it, Mr. Bode has already written part one of the hack: ensure_resource.

Why?

Suppose you have a given infrastructure with N vaguely identical nodes. N could equal 2 for a dual primary or active-passive cluster, or N could be greater than 2 for a more elaborate N-ary cluster. It is sufficient to say, that each of those N nodes might export an identical puppet resource which one (or many) clients might need to collect, to operate correctly. It’s important that each node export this, so that there is no single point of failure if one or more of the cluster nodes goes missing.

How?

As I mentioned, ensure_resources is a good enough hack to start. Here’s how you take an existing resource, and make it duplicate friendly. Take for example, the bulk of my dhcp::subnet resource:

define dhcp::subnet(
      $subnet,
      # [...]
      $range = [],
      $allow_duplicates = false
) {
      if $allow_duplicates { # a non empty string is also a true
            # allow the user to specify a specific split string to use...
            $c = type($allow_duplicates) ? {
                  'string' => "${allow_duplicates}",
                  default => '#',
            }
            if "${c}" == '' {
                  fail('Split character(s) cannot be empty!')
            }

            # split into $realname-$uid where $realname can contain split chars
            $realname = inline_template("<%= name.rindex('${c}').nil?? name : name.slice(0, name.rindex('${c}')) %>")
            $uid = inline_template("<%= name.rindex('${c}').nil?? '' : name.slice(name.rindex('${c}')+'${c}'.length, name.length-name.rindex('${c}')-'${c}'.length) %>")

            $params = { # this must use all the args as listed above...
                  'subnet' => $subnet,
                  # [...]
                  'range' => $range,
                  # NOTE: don't include the allow_duplicates flag...
            }

            ensure_resource('dhcp::subnet', "${realname}", $params)
      } else { # body of the actual resource...

            # BUG: lol: https://projects.puppetlabs.com/issues/15813
            $valid_range = type($range) ? {
                  'array' => $range,
                  default => [$range],
            }

            # the templating part of the module... 
            frag { "/etc/dhcp/subnets.d/${name}.subnet.frag":
                  content => template('dhcp/subnet.frag.erb'),
            }
      }
}

As you can see, I added an $allow_duplicates parameter to my resource. If it is set to true, then when the resource is defined, it parses out a trailing #comment from the $namevar. This can guarantee uniqueness for the $name (if they happen to be on the same node) but more importantly, it can guarantee uniqueness on a collector, where you will otherwise be unable to workaround the $name collision.

This is how you use this on one of the exporting nodes:

@@dhcp::subnet { "dmz#${hostname}":
    subnet => ...,
      range => [...],
      allow_duplicates => '#',
}

and on the collector:

Dhcp::Subnet <<| tag == 'dhcp' and title != "${dhcp_zone}" |>> {
}

There are a few things to notice:

  1. The $allow_duplicates argument can be set to true (a boolean), or to any string. If you pick a string, then that will be used to “split” out the end comment. It’s smart enough to split with a reverse index search so that your name can contain the #’s if need be. By default it looks for a single #, but you could replace this with ‘XXX123HACK‘ if that was the only unique string match you can manage. Make sure not to use the string value of ‘true‘.
  2. On my collector I like to filter by title. This is the $namevar. Sadly, this doesn’t support any fancier matching like in_array or startswith. I consider this a puppet deficiency. Hopefully someone will fix this to allow general puppet code here.
  3. Adding this to each resource is kind of annoying. It’s obviously a hack, but it’s the right thing to do for the time being IMHO.

Hope you had fun with this.

Happy hacking,

James

PS: [1] One side note, in the general case for custom resources, I actually think that by default duplicate parameters should be required, but that a resource could provide an optional function such as is_matched which would take as input the two parameter hash trees, and decide if they’re “functionally equivalent”. This would let an individual resource decide if it matters that you specified thing=>yes in one and thing=>true in the other. Functionally it matters that duplicate resources don’t have conflicting effects. I’m sure this would be particularly bug prone, and probably cause thrashing in some cases, which is why, by default the parameters should all match. </babble>

Overriding attributes of collected exported resources

This post is about a particularly elegant (and crucial) feature in puppet exported resources: attribute overriding. If you’re not already familiar with exported resources, you should start there, as they are the killer feature that makes configuration management with puppet awesome. (I haven’t found any explicit docs about this feature either, so feel free to comment if you know where they’re hidden.)

Setup: I’ve got a virtual machine which exports a resource to N different nodes. I’d like to define the resource with just one exported (@@) definition on my virtual machine.

Problem: One (or more) of the attributes needs to be changed based on which node it gets collected on. To make things more complicated, I’m using the same class definition on each of those N nodes to collect the resource. I don’t want to have to write N separate node definitions:

@@some::resource { 'the_name':
    foo => 'bar',
    #abc => 'different_on_each_node',
    tag => 'magic',
}

Solution: It turns out that for exported (or virtual) resources, you can specify attributes that get set upon collection. Naturally they can depend on a variable such as $name, which is unique to where they get collected:

Some::Resource <<| tag == 'magic' |>> {
    abc => "node-${name}",    # override!
}

Bonus: You can obviously use other variables throughout including in the collection (tag == ‘magic’) area, on both the source and the destination. Instead of a simple equality like I’ve used, you can actually specify a more complex expression, including other variables such as title (the $name).

Hope this takes your puppet coding to another level,

Happy hacking,

James