Iteration in Puppet

People often ask how to do iteration in Puppet. Most Puppet users have a background in imperative programming, and are already very familiar with for loops. Puppet is sometimes confusing at first, because it is actually (or technically, contains) a declarative, domain-specific language. In general, DSL’s aren’t always Turing complete, nor do they need to support loops, but this doesn’t mean you can’t iterate.

Until recently, Puppet didn’t have an explicit looping construct, and it is quite possible to build complex modules without using this new construct. There are even some who believe that the language shouldn’t even contain this feature. I’ll abstain from that debate for now, but instead, I would like to show you some iteration techniques that you can use to get your desired result.

Recursion

puppets-all-the-way-downMany people forget that recursion is a form of iteration. Even more don’t realize that you can do recursion in Puppet:

#!/usr/bin/puppet apply

define recursion(
    $count
) {
    # do something here...
    notify { "count-${count}":
    }
    $minus1 = inline_template('<%= count.to_i - 1 %>')
    if "${minus1}" == '0' {
        notify { 'done counting!':
        }
    } else {
        # recurse
        recursion { "count-${minus1}":
            count => $minus1,
        }
    }
}

# kick it off...
recursion { 'start':
    count => 4,
}

If you really want to Push Puppet, even more advanced recursion is possible. In general, I haven’t found this technique very useful for module design, but it’s worth mentioning as a form of iteration. If you do find a legitimate use of this technique, please let me know!

Type iteration

We’re used to seeing simple type declarations such as:

user { 'james':
    ensure => present,
    comment => 'james is awesome!',
}

In fact, the namevar can actually accept a list:

$users = ['kermit', 'beaker', 'statler', 'waldorf', 'tom']
user { $users:
    ensure => present,
    comment => 'who gave these muppets user accounts?',
}

Which will cause Puppet to effectively iterate across the elements in $users. This is the most important type of iteration in Puppet. Please get familiar with it.

This technique can be used with any type. It can even be used to express a many-to-one dependency relationship:

# where $bricks is a list of gluster::brick names
Gluster::Brick[$bricks] -> Gluster::Volume[$name]    # volume requires bricks

Suppose you’d like to use type iteration, but you’d also like to know the index of each element. This can be useful to avoid duplicate sub-types, or to provide a unique index:

define some_module::process_array(
    $foo,
    $array    # pass in the original $name
) {
    #notice(inline_template('NAME: <%= name.inspect %>'))

    # do something here...

    # build a unique name...
    $length = inline_template('<%= array.length %>')
    $ulength = inline_template('<%= array.uniq.length %>')
    if ( "${length}" != '0' ) and ( "${length}" != "${ulength}" ) {
        fail('Array must not have duplicates.')
    }
    # if array had duplicates, this wouldn't be a unique index
    $index = inline_template('<%= array.index(name) %>')

    # iterate, knowing your index
    some::type { "${foo}:${index}":
        foo => 'hello',
        index => "${index}",
    }
}

# a list
$some_array = ['a', 'b', 'c']    # must not have duplicates

# using the type requires that you pass in $some_array twice!
some_module::process_array { $some_array:    # array
    foo => 'bar',
    array => $some_array,    # same array as above
}

While this example might seem contrived, it is actually a modified excerpt from a module that I wrote.

create_resources

This is a similar technique for when you want to specify different arguments for each type:

$defaults = {
    ensure => present,
    comment => 'a muppet',
}
$data = {
    'kermit' => {
        comment => 'the frog',
    },
    'beaker' => {
        comment => 'keep him away from your glassware',
    },
    'fozzie' => {
        home => '/home/fozzie',
    },
    'tom' => {
        comment => 'the swedish chef',
    }
}
create_resources('user', $data, $defaults)

This creates each user resource with its own arguments. If an argument isn’t given in the $data, it is taken from the $defaults hash. A similar example, and the official documentation is found here.

Template iteration

You might want to iterate to perform a simple computation, or to modify an array in some way. For static value computations, you can often use a template. Remember that the template will get executed at compile time on the Puppet Master, so code accordingly. Here are a few contrived examples:

# filter out all the integers less than zero
$array_in = [-4,3,-8,-2,1,4,-2,1,5,-1,-7,9,-3,2,6,-8,5,3,5,-6,8,9,7,-5,9,3,-3]
$array_out = split(inline_template('<%= array_in.delete_if {|x| x < 0 }.join(",") %>'), ',')
notice($array_out)

We can also use the ruby map:

# build out a greeting string
$names = ['animal', 'gonzo', 'rowlf']
# NOTE: you can also use a regular multi-line template for readability
$message = inline_template('<% if names == [] %>Hello... Anyone there?<% else %><%= names.map {|x| "Hello "+x.capitalize}.join(", ") %>.<% end %>')
notice($message)

Use your imagination! Remember that you can also write a custom function if necessary, but first check that there isn’t already a built-in function, or a stdlib function that suits your needs.

Advanced template iteration

When you really need to get fancy, it’s often time to call in a custom function. Custom functions require that you split them off into separate files, and away from the module logic, instead of keeping the functions inline and accessible as lambdas. The downside to using these “inline_template” lambdas instead, is that they can quickly turn into parlous one-liners.

# transform the $data hash
$data = {
    'waldorf' => {
        'heckles' => 'absolutely',
        'comment' => 'a critic',
    },
    'statler' => {
        'heckles' => 'all the time',
        'comment' => 'another critic!',
    },
}
# rename and filter on the 'heckles' key
$yaml = inline_template('<%= data.inject({}) {|h, (x,y)| h[x] = {"applauds" => y.fetch("heckles", "yes")}; h}.to_yaml %>')
$output = parseyaml($yaml) # parseyaml is in the puppetlabs-stdlib
notice($output)

As with simple template iteration, the key problem is transferring the data in and out of the template. In the simple case, arrays can be joined and split as long as there is a reserved character that won’t be used in the data. For the advanced template iteration, we rely on the YAML transformation functions.

Some reminders

If you properly understand the functionality that your module is trying to model/manage, you can usually break it up into separate classes and defined types, such that re-use via type iteration can fulfill your needs. Usually you’ll end up with a more properly designed module.

Test using the same version of Ruby that will run your module. Newer versions of Ruby have some incompatible changes, and new features, with respect to older versions of Ruby.

Remember that templates and functions run on the Puppet Master, but facts and types run on the client (agent).

The Puppet language is mostly declarative. Because this might be an unfamiliar paradigm, try not to look for all the imperative features that you’re used to. Having a programming background can help, because there’s certainly programming mixed in, whether you’re writing custom functions, or erb templates.

Future parser

For completeness, I should mention that the future parser now supports native iteration. If you need it, it probably means that you’re writing a fairly advanced module, and you’re comfortable manual diving. If you have a legitimate use case that isn’t possible with the existing constructs, and isn’t only a readability improvement, please let me know.

Conclusion

I hope you enjoyed this article. The next time someone asks you how to iterate in Puppet, feel free to link them this way.

Happy hacking,

James

Advertisements

16 thoughts on “Iteration in Puppet

  1. Very good overview of Iteration!

    Regarding the future parser: I agree with your implication that while it is there, you probably don’t need it for iteration.

    IMO, the future parser is a mis-step. My observation is that Puppet conditionals and loops tend to be parse order dependent, which goes counter to the stated goals of Puppet. IMO, good design eliminates conditionals wherever possible. Conditionals tend to obfuscate code and are rarely implemented in a streamlined way.

    In my own code, I tend to use only 2 constructs: case statements for setting default values, and simple 1 level if/then statements for error checking and resources that should be applied conditionally. I tend to prefer the stdlib validate_ functions to if/then for input validation.

    I recently wrote a module that created thousands of unique resources for testing purposes using only defined types, arrays, and the create_resources function.

    • Very good overview of Iteration!

      Thanks!

      Regarding the future parser: I agree with your implication that while it is there, you probably don’t need it for iteration.

      IMO, the future parser is a mis-step. My observation is that Puppet conditionals and loops tend to be parse order dependent, which goes counter to the stated goals of Puppet. IMO, good design eliminates conditionals wherever possible. Conditionals tend to obfuscate code and are rarely implemented in a streamlined way.

      I do believe that Puppet should be completely parse order independent. I’m not sure that this conflicts with conditionals though. There are some examples using conditionals that would have a circular dependency on each other, and should result in a parse error though.

      In my own code, I tend to use only 2 constructs: case statements for setting default values, and simple 1 level if/then statements for error checking and resources that should be applied conditionally. I tend to prefer the stdlib validate_ functions to if/then for input validation.

      I recently wrote a module that created thousands of unique resources for testing purposes using only defined types, arrays, and the create_resources function.

      Cool. Would be great to see the code.

    • I also created a module that creates thousands of unique resources yesterday using the same tools as you. However, after going through the experience, I have to disagree that native iteration is a bad thing to have. Consider this simple example.

      define resource_foo(
      $foo
      ) {
      notice(“My name is ${name} and my foo is ${foo})
      }

      The user will call this resource as such.

      resource_bar{‘foo’:
      count => ‘100’,
      foo_array => [‘a’,’b’,’c’],
      }

      The requirement would be that resource_bar would create $count resource_foo resources where resource_foo’s $name would be “name1” up to “name${count}” and $foo would take the value of foo_array in a round-robin fashion (first resource_foo would have $foo=’a’, 2nd one would be $foo=’b’, etc and cycle up to $count).

      While I managed to do this by calling a create_resources from resource_bar that took as arguments the result of some inline_template Ruby manipulations, using a loop would made the code infinetely cleaner and readable in my opinion.

      • Hey,

        I don’t say that you shouldn’t use the future iteration, but I try to show many ways to accomplish similar results.

        Since your code is contrived (as opposed to code from a complete module) it’s tough for me to argue that there is a better way. There might not be! What I can say, is that I haven’t had a need to require the future iteration feature. If you have a complete module that does, I’d love to see it! Maybe I can offer a more elegant approach, and maybe I won’t be able to.

        Thanks for sharing,

        Cheers

  2. BTW James:

    The Puppet stdlib introduces the range() function. You might want to take a look at it.

    This example creates /tmp/test{0..10} with content ‘foobarbaz’ using range() and defined types:

    define iterfile (
    $prefix,
    $ensure = undef,
    $suffix = $title,
    $source = undef,
    $content = undef,
    ) {
    file { “${prefix}${suffix}”:
    ensure => $ensure,
    source => $source,
    content => $content,
    }
    }

    $iter = range(‘0′, ’10’)

    iterfile { $iter:
    ensure => ‘present’,
    prefix => ‘/tmp/test’,
    content => ‘foobarbaz’,
    }

    • Hi Chris,

      I tried to allude to this possibility in my “Some reminders” section. I’ve called this “type iteration”, but it might have a more accepted name elsewhere. Thank you for posting an example! I haven’t ever used the range function, but I’ll keep it in mind for future hacks! Have you ever needed to use the range function in a module? If so, which one?

      Cheers

      • Most of my iteration is the kind that’s solved with create_resources. A lot of my code ends up being:

        $type = hiera_hash(key, undef)
        create_resources(type, $type)

        I can’t think of anywhere I’d use range() outside of writing a test.

    • The following is code is equivalent to what I just post, and is much more flexible.

      $iter = prefix(range(‘0′, ’10’), ‘/tmp/test’)

      file { $iter:
      ensure => ‘present’,
      content => ‘foobarbaz’,
      }

  3. Pingback: Looping and puppet | Keep learning...

  4. This is a great post. Thank you.

    I have a use case where iteration is required and I’m trying to find the best way to do it. I’d rather not turn on future parser if I don’t need to but there may not be an elegant way of doing it otherwise. We are running Puppet 3.7.3.

    My use case is installing n number of Tomcat instances per server for a given environment. The actual number is defined in Hiera so the environment administrators can easily adjust it. This number can be upwards of 24 for bare metal servers.

    Puppet will need to install each instance, configure the ports, properties, directories and instance name all based on the instance number. The application is identical across all instances. I don’t see a good way of doing this without a for loop. Any suggestions?

    • I think it’s somewhat straight forward using one of the techniques in this post, but highly independent of what your code and module looks like. If you point me to some of it, I can have a look– alternatively, if you’re at #configmgmt camp at ghent right now, come find me and I’ll be happy to look at it with you.

      Cheers!

      • We’re at the beginning stages of this module so there is not code just yet. I was trying to do some research first to see what my options were instead of diving right into future parser. The template iteration may work for us. I’ll let you know how it goes.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s