Advanced recursion and memoization in Puppet

As a follow-up to my original article on recursion in Puppet, and in my attempt to Push Puppet (to its limit), I’ll now attempt some more advanced recursion techniques in Puppet.

In my original recursion example, the type does recurse, but the callee cannot return any value to the caller because it is a type, and not strictly a function. This limitation immediately limits the usefulness of this technique, but I’ll try to press on! Let’s try to write a Fibonacci series function in native Puppet.

For those who aren’t familiar, the Fibonacci series function is a canonical computer science recursion example. It is very easy to write as pseudo-code or to implement in python:

def F(n):
	if n == 0: return 0
	elif n == 1: return 1
	else: return F(n-1)+F(n-2)

You can download this function here. Let’s run that script to get a table of the first few values in the Fibonacci series:

$ ./fibonacci.py
F(0) == 0
F(1) == 1
F(2) == 1
F(3) == 2
F(4) == 3
F(5) == 5
F(6) == 8
F(7) == 13
F(8) == 21
F(9) == 34
F(10) == 55
F(11) == 89
F(12) == 144
F(13) == 233
...

Now, I’ll introduce my Puppet implementation:

#!/usr/bin/puppet apply
class fibdir {
	# memoization directory
	file { '/tmp/fibonacci/':
		ensure => directory,
	}
}

define fibonacci(
	$n,
	$intermediate = false # used for pretty printing
) {
	include fibdir

	$vardir = '/tmp'
	$fibdir = "${vardir}/fibonacci"

	if "${n}" == '0' {
		# 0
		exec { "${name}: F(0)":
			command => "/bin/echo 0 > ${fibdir}/0",
			creates => "${fibdir}/0",
			require => File["${fibdir}/"],
		}

	} elsif "${n}" == '1' {
		# 1
		exec { "${name}: F(1)":
			command => "/bin/echo 1 > ${fibdir}/1",
			creates => "${fibdir}/1",
			require => File["${fibdir}/"],
		}

	} else {

		$minus1 = inline_template('<%= n.to_i - 1 %>')
		fibonacci { "${name}: F(${n}-1)":
			n => $minus1,
			intermediate => true,
		}

		$minus2 = inline_template('<%= n.to_i - 2 %>')
		fibonacci { "${name}: F(${n}-2)":
			n => $minus2,
			intermediate => true,
		}

		# who can figure out what the problem with this is ?
		$fn1 = inline_template('<%= f1=fibdir+"/"+minus1; (File.exist?(f1) ? File.open(f1, "r").read.to_i : -1) %>')
		$fn2 = inline_template('<%= f2=fibdir+"/"+minus2; (File.exist?(f2) ? File.open(f2, "r").read.to_i : -1) %>')

		if (("${fn1}" == '-1') or ("${fn2}" == '-1')) {
			$fn = '-1'
		} else {
			$fn = inline_template('<%= fn1.to_i+fn2.to_i %>')
		}

		if "${fn}" != '-1' { # did the lookup work ?
			# store fibonacci number in 'table' (memoization)
			exec { "${name}: F(${n})":
				command => "/bin/echo ${fn} > ${fibdir}/${n}",
				creates => "${fibdir}/${n}",
				require => [
					File["${fibdir}/"],
					Fibonacci["${name}: F(${n}-1)"],
					Fibonacci["${name}: F(${n}-2)"],
				],
			}

			if ! $intermediate {
				# display...
				notify { "F(${n})":
					message => "F(${n}) == ${fn}",
					require => Exec["${name}: F(${n})"],
				}
			}
		}
	}
}

# kick it off...
fibonacci { 'start':
	n => 8,
}

It is available for download. Try to read through the code yourself first. As you’ll see, if called with n == 0, or n == 1, the function creates a file with this value and exits. This is the secret to how the function (the type) passes values around. It first stores them in files, and then loads them in through templates.

Each time this runs, Puppet will complete the next step in the execution. To make this successive execution automatic, I’ve written a small bash wrapper to do this, but you can run it manually too. If you do use my wrapper, use it with the fibonacci.pp file provided in git.

The computer scientist might notice that as a side effect, we are actually memoizing. This means that if we run this type again with a larger input value, the previously completed intermediate step values are used as a starting point for the subsequent computations. Cool!

The Puppet wizard might notice that I cheated slightly. Take a minute to try to see where…

[IMAGE OF TIME PASSING]

(IMAGE OF TIME PASSING)

Have you figured it out? The problem with the current implementation is that it will only work when run locally as a standalone Puppet program. The reason, is that exec types run on the client, and the templates run on the server. This type requires that both of those elements run on the same machine so that the save/load memoization can work correctly. Since this code runs on the same machine, this isn’t a problem! This split execution model is one of the features that can confuse new Puppet users.

To adapt our function (technically a type) to work in any environment, we need to do some more hacking! We can continue to use our exec type for saving, but a fact needs to be used to load in the necessary values:

require 'facter'

fibdir = '/tmp/fibonacci/'
valid_fibdir = fibdir.gsub(/\/$/, '')+'/' # ensure trailing slash

results = {} # create list of values

if File.directory?(valid_fibdir)
	Dir.glob(valid_fibdir+'*').each do |f|
		b = File.basename(f)
		i = b.to_i # invalid returns 0
		if b == '0' or i != 0
			v = File.open(f, 'r').read.strip.to_i # read into int
			results[i] = v
		end
	end
end

results.keys.each do |x|
	Facter.add('pushing_fibonacci_'+x.to_s) do
		setcode {
			results[x]
		}
	end
end

The templates from the first version of this type need to be replaced with fact variable lookups:

# these are 'fact' lookups
$fn1 = getvar("pushing_fibonacci_${minus1}")
$fn2 = getvar("pushing_fibonacci_${minus2}")

You can use git to download all of this code as a module.

I hope you enjoyed this. Please let me know! Comment below, or send me a message.

Happy hacking,

James

P.S.: While I think this is fun, I wrote this hack to demonstrate some techniques, and to set the stage for future hacks, future techniques, and future Puppet examples. If you’re using this as a good way to actually compute values in the Fibonacci series, you’re insane!

P.P.S.: The word is actually memoization, not memorization, despite the similarities between the two words, and the two concepts.

P.P.P.S: Gluster users get extra points if they can figure out how this will lead to a feature for Puppet-Gluster. It’s a bit tricky to see if you’re not following my git commits.

Documentation for Puppet-Gluster

Ironically, one of the reasons that I started writing Puppet code, was so that I could spend more time designing and building, and less time writing documentation. I suppose I’m a victim of my success, because Puppet-Gluster has grown large enough to warrant its own documentation.

So I gave in, and put together some documentation.

It’s available as markdown, and, also as a pdf. As before, there is an examples/ directory which you might want to use as additional reference.

Please send me feedback! I want to hear about any parts you found confusing, spelling and/or grammar fixes, and any other information that you think I should add.

The documentation is also mirrored at the gluster forge.

Happy hacking,

James

 

Iteration in Puppet

People often ask how to do iteration in Puppet. Most Puppet users have a background in imperative programming, and are already very familiar with for loops. Puppet is sometimes confusing at first, because it is actually (or technically, contains) a declarative, domain-specific language. In general, DSL’s aren’t always Turing complete, nor do they need to support loops, but this doesn’t mean you can’t iterate.

Until recently, Puppet didn’t have an explicit looping construct, and it is quite possible to build complex modules without using this new construct. There are even some who believe that the language shouldn’t even contain this feature. I’ll abstain from that debate for now, but instead, I would like to show you some iteration techniques that you can use to get your desired result.

Recursion

puppets-all-the-way-downMany people forget that recursion is a form of iteration. Even more don’t realize that you can do recursion in Puppet:

#!/usr/bin/puppet apply

define recursion(
    $count
) {
    # do something here...
    notify { "count-${count}":
    }
    $minus1 = inline_template('<%= count.to_i - 1 %>')
    if "${minus1}" == '0' {
        notify { 'done counting!':
        }
    } else {
        # recurse
        recursion { "count-${minus1}":
            count => $minus1,
        }
    }
}

# kick it off...
recursion { 'start':
    count => 4,
}

If you really want to Push Puppet, even more advanced recursion is possible. In general, I haven’t found this technique very useful for module design, but it’s worth mentioning as a form of iteration. If you do find a legitimate use of this technique, please let me know!

Type iteration

We’re used to seeing simple type declarations such as:

user { 'james':
    ensure => present,
    comment => 'james is awesome!',
}

In fact, the namevar can actually accept a list:

$users = ['kermit', 'beaker', 'statler', 'waldorf', 'tom']
user { $users:
    ensure => present,
    comment => 'who gave these muppets user accounts?',
}

Which will cause Puppet to effectively iterate across the elements in $users. This is the most important type of iteration in Puppet. Please get familiar with it.

This technique can be used with any type. It can even be used to express a many-to-one dependency relationship:

# where $bricks is a list of gluster::brick names
Gluster::Brick[$bricks] -> Gluster::Volume[$name]    # volume requires bricks

Suppose you’d like to use type iteration, but you’d also like to know the index of each element. This can be useful to avoid duplicate sub-types, or to provide a unique index:

define some_module::process_array(
    $foo,
    $array    # pass in the original $name
) {
    #notice(inline_template('NAME: <%= name.inspect %>'))

    # do something here...

    # build a unique name...
    $length = inline_template('<%= array.length %>')
    $ulength = inline_template('<%= array.uniq.length %>')
    if ( "${length}" != '0' ) and ( "${length}" != "${ulength}" ) {
        fail('Array must not have duplicates.')
    }
    # if array had duplicates, this wouldn't be a unique index
    $index = inline_template('<%= array.index(name) %>')

    # iterate, knowing your index
    some::type { "${foo}:${index}":
        foo => 'hello',
        index => "${index}",
    }
}

# a list
$some_array = ['a', 'b', 'c']    # must not have duplicates

# using the type requires that you pass in $some_array twice!
some_module::process_array { $some_array:    # array
    foo => 'bar',
    array => $some_array,    # same array as above
}

While this example might seem contrived, it is actually a modified excerpt from a module that I wrote.

create_resources

This is a similar technique for when you want to specify different arguments for each type:

$defaults = {
    ensure => present,
    comment => 'a muppet',
}
$data = {
    'kermit' => {
        comment => 'the frog',
    },
    'beaker' => {
        comment => 'keep him away from your glassware',
    },
    'fozzie' => {
        home => '/home/fozzie',
    },
    'tom' => {
        comment => 'the swedish chef',
    }
}
create_resources('user', $data, $defaults)

This creates each user resource with its own arguments. If an argument isn’t given in the $data, it is taken from the $defaults hash. A similar example, and the official documentation is found here.

Template iteration

You might want to iterate to perform a simple computation, or to modify an array in some way. For static value computations, you can often use a template. Remember that the template will get executed at compile time on the Puppet Master, so code accordingly. Here are a few contrived examples:

# filter out all the integers less than zero
$array_in = [-4,3,-8,-2,1,4,-2,1,5,-1,-7,9,-3,2,6,-8,5,3,5,-6,8,9,7,-5,9,3,-3]
$array_out = split(inline_template('<%= array_in.delete_if {|x| x < 0 }.join(",") %>'), ',')
notice($array_out)

We can also use the ruby map:

# build out a greeting string
$names = ['animal', 'gonzo', 'rowlf']
# NOTE: you can also use a regular multi-line template for readability
$message = inline_template('<% if names == [] %>Hello... Anyone there?<% else %><%= names.map {|x| "Hello "+x.capitalize}.join(", ") %>.<% end %>')
notice($message)

Use your imagination! Remember that you can also write a custom function if necessary, but first check that there isn’t already a built-in function, or a stdlib function that suits your needs.

Advanced template iteration

When you really need to get fancy, it’s often time to call in a custom function. Custom functions require that you split them off into separate files, and away from the module logic, instead of keeping the functions inline and accessible as lambdas. The downside to using these “inline_template” lambdas instead, is that they can quickly turn into parlous one-liners.

# transform the $data hash
$data = {
    'waldorf' => {
        'heckles' => 'absolutely',
        'comment' => 'a critic',
    },
    'statler' => {
        'heckles' => 'all the time',
        'comment' => 'another critic!',
    },
}
# rename and filter on the 'heckles' key
$yaml = inline_template('<%= data.inject({}) {|h, (x,y)| h[x] = {"applauds" => y.fetch("heckles", "yes")}; h}.to_yaml %>')
$output = parseyaml($yaml) # parseyaml is in the puppetlabs-stdlib
notice($output)

As with simple template iteration, the key problem is transferring the data in and out of the template. In the simple case, arrays can be joined and split as long as there is a reserved character that won’t be used in the data. For the advanced template iteration, we rely on the YAML transformation functions.

Some reminders

If you properly understand the functionality that your module is trying to model/manage, you can usually break it up into separate classes and defined types, such that re-use via type iteration can fulfill your needs. Usually you’ll end up with a more properly designed module.

Test using the same version of Ruby that will run your module. Newer versions of Ruby have some incompatible changes, and new features, with respect to older versions of Ruby.

Remember that templates and functions run on the Puppet Master, but facts and types run on the client (agent).

The Puppet language is mostly declarative. Because this might be an unfamiliar paradigm, try not to look for all the imperative features that you’re used to. Having a programming background can help, because there’s certainly programming mixed in, whether you’re writing custom functions, or erb templates.

Future parser

For completeness, I should mention that the future parser now supports native iteration. If you need it, it probably means that you’re writing a fairly advanced module, and you’re comfortable manual diving. If you have a legitimate use case that isn’t possible with the existing constructs, and isn’t only a readability improvement, please let me know.

Conclusion

I hope you enjoyed this article. The next time someone asks you how to iterate in Puppet, feel free to link them this way.

Happy hacking,

James

Pushing Puppet at Puppet Camp DC, LISA 2013

Hi there,

I hope you enjoyed my “Pushing Puppet (to its limit)” talk and demos from Puppet Camp D.C., LISA 2013. As requested, I’ve posted the code and slides.

Here is the code:

https://github.com/purpleidea/puppet-pushing

This module will require three modules as dependencies. The dependencies are:

Each example doesn’t require all the dependencies, so if you’re only interested in the FSM, you only need that module.

Here are the slides:

https://github.com/purpleidea/puppet-pushing/blob/master/talks/pushing-puppet.pdf

Here is the bug fix to fix my third Exec[‘again’] demo:

https://github.com/purpleidea/puppet-common/commit/df3d004044f013415bb6001a2defd64b587d3b85

It’s my fault that I added the fancy –delta support, but forgot to test the simpler, version again. Woops.

I’ve previously written about some of this puppet material. Read through these articles for more background and details:

I haven’t yet written articles about all the techniques used during my talk. I’ll try to write future articles about these topics if you’re interested.

If anyone has some photos from the talk, I’d love for you to send me a copy.

Special thanks to Kara, Dawn and Puppet Labs for asking me to present.

If you’d like to invite me to teach, talk or consult, I’d love to come visit your {$SCHOOL, $WORK, $CITY, etc}. Contact me! I’ll be around in D.C. till Friday if you’d like to meet up and hack on some of the code or examples that I’ve published.

If you’re interested in looking at some of the “real work” modules that I’ve written, have a look through my github repositories. Puppet-Gluster and Puppet-IPA, are two that you might find most interesting.

There are a few that I haven’t yet published, so if you’re looking for a fancy module to do X, let me know and I might be a few commits away from something helpful that I haven’t made public yet.

I hope you enjoyed hacking on puppet with me, and please don’t be shy — leave me a comment about my talk, and ask questions if you have any.

Happy Hacking,

James

 

Gluster Community Day, LISA 2013, Monday

I’m here at LISA 2013 at the Gluster Community Day. I’ve been asked by Joe Brockmeier to give a little recap about what’s been going on. So here it is!

Wesley Duffee-Braun started off with a nice overview talk about GlusterFS. The great thing about his talk was that he gave a live demo, running on virtual machines, on his laptop. If you’re a new GlusterFS user, this is good exposure to help you get started. He also has the nicest slides I’ve ever seen. Someone needs to send me the .odp template!

Eco Willson gave the next talk about geo-replication, and discussed a few other GlusterFS topics too. I particularly enjoyed when he talked to us about the upcoming changes. I understood that this would include the ability to have multiple geo-replication daemons across each distributed part of each volume, and, HA between the N brick replicas. This way, your workload is split across the volume, and if a replica is down, one of the other copies can take over and continue the sync.

During lunch, I got to meet Jeff Darcy and talk a bit about storage and feature ideas on my wish-list. He knew my twitter/IRC handle, so he instantly gets 200 bonus points. You should probably checkout his blog if you haven’t already.

After lunch I gave my talk about puppet-gluster, and successfully gave two live demos. I’m really due for a blog post about some of the new features that I’ve added. I’ll try to put together a screen cast in the future. If you’re really keen on trying out some of the new features, I’m happy to share a screen session on your hosts and walk you through it. When run fully automatically, it takes between five and ten minutes to deploy a featureful GlusterFS!

I won’t be able to hack on this project for free, forever. If you’re able to donate test hardware, VM time, or sponsor a feature, it would be appreciated. I’m particularly interested in building a GlusterFS microserver cluster. If you’re interested in this too, let me know.

Wesley came back for round two to demo the GlusterForge (hi John Mark!) and Glusterinfo, which I hadn’t ever used. Nobody knew why certain projects are still “incubating”, when they’re mostly likely cooked through by now.

At this point, we had some left over time for questions and discussion. Jeff, Eco, and I formed a loosely organized “panel” to answer questions. Joe took a photo, and now owes me a copy, since I’ve never really been on a “panel” before. Jeff knows his GlusterFS internals very well. I should have come prepared with some hard questions!

To whoever was looking for Ben England’s RHS performance talk, it’s available here.

Overall, it was a very nice little event, and I hope the attendees enjoyed it. If I forgot anything, please feel free to add it into the comments. I’d also love to hear about what you enjoyed or didn’t enjoy about my talk. Let me know!

Happy Hacking,

James

Speaking at LISA 2013 about Puppet and GlusterFS

I’m speaking at LISA 2013, the “Large Installation System Administration” conference. This conference runs all week in Washington. I’ll be giving two talks during the week, and attending at least one BOF.

My first talk is on Monday during the Gluster Community Day. I’ll be speaking about puppet-gluster, and giving a live demo. I’ll be showing some new features too. If you’d like to talk more about puppet-gluster, or want to attend the talk, give me a shout, or sign up at the above Gluster Community Day link.

On Tuesday, I’ll be giving a talk during Puppet Camp DC. My talk will be about “Pushing Puppet (to the limit)”. I’ve prepared lots of fun puppet hacks, and live demos, so I expect you’ll thoroughly enjoy this.

I’ve got lots of technical hacks, and great code that I’ve published lately, but that I haven’t blogged about yet. If you follow along with my git commits, you’ll be able to figure out most of it, but I’ve got a bunch of articles coming anyways.

Thanks to John Mark and RedHat for sponsoring my trip, Dawn Foster and Kara Sowles for organizing my Puppet talk and Hilary Hartman at USENIX for helping me register.

Happy Hacking,

James

PS: If you’re at LISA 2013, and you want to buy me dinner and get some light Puppet or Gluster consulting, contact me! You can also give me a shout if you just want to talk tech. If anyone wants to meet up and hack on things, please let me know too.

I'm speaking at LISA 2013