Collecting duplicate resources in puppet

I could probably write a long design article explaining why identical duplicate resources should be allowed [1] in puppet. If puppet is going to survive in the long-term, they will have to build in this feature. In the short-term, I will have to hack around deficiency. As luck would have it, Mr. Bode has already written part one of the hack: ensure_resource.

Why?

Suppose you have a given infrastructure with N vaguely identical nodes. N could equal 2 for a dual primary or active-passive cluster, or N could be greater than 2 for a more elaborate N-ary cluster. It is sufficient to say, that each of those N nodes might export an identical puppet resource which one (or many) clients might need to collect, to operate correctly. It’s important that each node export this, so that there is no single point of failure if one or more of the cluster nodes goes missing.

How?

As I mentioned, ensure_resources is a good enough hack to start. Here’s how you take an existing resource, and make it duplicate friendly. Take for example, the bulk of my dhcp::subnet resource:

define dhcp::subnet(
      $subnet,
      # [...]
      $range = [],
      $allow_duplicates = false
) {
      if $allow_duplicates { # a non empty string is also a true
            # allow the user to specify a specific split string to use...
            $c = type($allow_duplicates) ? {
                  'string' => "${allow_duplicates}",
                  default => '#',
            }
            if "${c}" == '' {
                  fail('Split character(s) cannot be empty!')
            }

            # split into $realname-$uid where $realname can contain split chars
            $realname = inline_template("<%= name.rindex('${c}').nil?? name : name.slice(0, name.rindex('${c}')) %>")
            $uid = inline_template("<%= name.rindex('${c}').nil?? '' : name.slice(name.rindex('${c}')+'${c}'.length, name.length-name.rindex('${c}')-'${c}'.length) %>")

            $params = { # this must use all the args as listed above...
                  'subnet' => $subnet,
                  # [...]
                  'range' => $range,
                  # NOTE: don't include the allow_duplicates flag...
            }

            ensure_resource('dhcp::subnet', "${realname}", $params)
      } else { # body of the actual resource...

            # BUG: lol: https://projects.puppetlabs.com/issues/15813
            $valid_range = type($range) ? {
                  'array' => $range,
                  default => [$range],
            }

            # the templating part of the module... 
            frag { "/etc/dhcp/subnets.d/${name}.subnet.frag":
                  content => template('dhcp/subnet.frag.erb'),
            }
      }
}

As you can see, I added an $allow_duplicates parameter to my resource. If it is set to true, then when the resource is defined, it parses out a trailing #comment from the $namevar. This can guarantee uniqueness for the $name (if they happen to be on the same node) but more importantly, it can guarantee uniqueness on a collector, where you will otherwise be unable to workaround the $name collision.

This is how you use this on one of the exporting nodes:

@@dhcp::subnet { "dmz#${hostname}":
    subnet => ...,
      range => [...],
      allow_duplicates => '#',
}

and on the collector:

Dhcp::Subnet <<| tag == 'dhcp' and title != "${dhcp_zone}" |>> {
}

There are a few things to notice:

1. The $allow_duplicates argument can be set to true (a boolean), or to any string. If you pick a string, then that will be used to “split” out the end comment. It’s smart enough to split with a reverse index search so that your name can contain the #’s if need be. By default it looks for a single #, but you could replace this with ‘XXX123HACK‘ if that was the only unique string match you can manage. Make sure not to use the string value of ‘true‘.
2. On my collector I like to filter by title. This is the $namevar. Sadly, this doesn’t support any fancier matching like in_array or startswith. I consider this a puppet deficiency. Hopefully someone will fix this to allow general puppet code here.
3. Adding this to each resource is kind of annoying. It’s obviously a hack, but it’s the right thing to do for the time being IMHO.

Hope you had fun with this.

Happy hacking,

James

PS: [1] One side note, in the general case for custom resources, I actually think that by default duplicate parameters should be required, but that a resource could provide an optional function such as is_matched which would take as input the two parameter hash trees, and decide if they’re “functionally equivalent”. This would let an individual resource decide if it matters that you specified thing=>yes in one and thing=>true in the other. Functionally it matters that duplicate resources don’t have conflicting effects. I’m sure this would be particularly bug prone, and probably cause thrashing in some cases, which is why, by default the parameters should all match. </babble>