puppet rsnapshot module

Today I am releasing a puppet module for rsnapshot. The nice feature of this module, is that it lets you configure multiple different instances of rsnapshot, so that they could all run in parallel. Rsnapshot doesn’t support this directly, so this puppet module does the heavy lifting of separating out and managing each instance.

https://github.com/purpleidea/puppet-rsnapshot

The examples/ directory should give you a hint on how to use it. For everything else, have a look at the code, or feel free to leave me a comment. I hope you find it useful!

Happy hacking,

James

 

Overriding attributes of collected exported resources

This post is about a particularly elegant (and crucial) feature in puppet exported resources: attribute overriding. If you’re not already familiar with exported resources, you should start there, as they are the killer feature that makes configuration management with puppet awesome. (I haven’t found any explicit docs about this feature either, so feel free to comment if you know where they’re hidden.)

Setup: I’ve got a virtual machine which exports a resource to N different nodes. I’d like to define the resource with just one exported (@@) definition on my virtual machine.

Problem: One (or more) of the attributes needs to be changed based on which node it gets collected on. To make things more complicated, I’m using the same class definition on each of those N nodes to collect the resource. I don’t want to have to write N separate node definitions:

@@some::resource { 'the_name':
    foo => 'bar',
    #abc => 'different_on_each_node',
    tag => 'magic',
}

Solution: It turns out that for exported (or virtual) resources, you can specify attributes that get set upon collection. Naturally they can depend on a variable such as $name, which is unique to where they get collected:

Some::Resource <<| tag == 'magic' |>> {
    abc => "node-${name}",    # override!
}

Bonus: You can obviously use other variables throughout including in the collection (tag == ‘magic’) area, on both the source and the destination. Instead of a simple equality like I’ve used, you can actually specify a more complex expression, including other variables such as title (the $name).

Hope this takes your puppet coding to another level,

Happy hacking,

James

 

Knowing when to release and deploy your code (…and a mini script)

Knowing when to release and deploy your code can turn into a complicated discussion. In general, In general, I tend to support releasing early and often, for some value of $early and $often. I’ve decided to keep this simple and introduce you to one metric that I use…

I think that I am fairly diligent in adding plenty of comments to my source code. I might even sometimes add too many. I create plenty of XXX, FIXME, or TODO tagged comments as reminders of things to work on.

To me, XXX represents an important problem that should get looked at or fixed; FIXME, reminds me that I should definitely look into something, and finally, TODO gives me homework or things to pursue when I’m in need of a new project.

I try to resolve most if not all XXX tagged comments before making a 0.1 release, FIXME’s to consider something very stable, and a lack of TODO’s mean something is completely done for now.

To count all these, I wrote a little tool that greps through the top-level directories in my ~/code/ folder, and displays the results in a table. Feel free to give it a try, and use it for your own projects.

While I don’t see this as a particularly game changing utility, it scratches my itch, and helps me keep up my bash skills. The code is available here. Let me know if you have any improvements, or if the source isn’t enough documentation for you.

Happy hacking,

James

adding range support to python’s http server to kickstart with anaconda

I’ve been working on automatic installs using kickstart and puppet. I’m using a modified python httpserver because it’s lightweight, and easy to integrate into my existing python code base. The server was churning away perfectly until anaconda started downloading the full rpm’s for installation. What was going wrong?

Traceback (most recent call last):
[...]
error: [Errno 32] Broken pipe
BorkedError: See TTBOJ for explanation and discussion

As it turns out, anaconda first downloads the headers, and then later requests the full rpm with an http range request. This second range request which begins at byte 1384, causes the “simple” httpserver to bork, because it doesn’t support this more elaborate feature.

After a bit of searching, I found rangehttpserver and was very grateful that I wouldn’t have to write this feature myself. This work by smgoller was based on the similar httpserver by xyne. Both of these people have been very responsive and kind in giving me special permission to the relevant portions of their code that I needed under the GPLv2/3+. Thanks to these two and their contribution to Free Software this let’s us all see further, instead of having to reinvent previously solved problems.

This derivative work is only one part of a larger software release that I have coming shortly, but I wanted to put this out here early to thank these guys and to make you all aware of the range issue and solution.

Thank you again and,
Happy Hacking,

James

Automatic hiera lookups in puppet 3.x

Dear readers,

I’ve started the slow migration of code from puppet 2.6 all the way to 3.x+. There were a few things I wasn’t clear on, so hopefully this will help to discuss these and make your migration easier!

I used hiera in 2.6, and I actually like it a lot so far. I was concerned that automatic lookups would pull in values that I wasn’t expecting. This is not the case or a worry. Let’s dive in and let the code speak:

# create a class in a module or site.pp for testing...
class foo(
        $a = 'apple',
        $b = 'banana'
) {
        notify { 'foo':
                message => "a is: ${a}, b is: ${b}",
        }
}

and

# define it using :: as a prefix because we want to search in the
# top level, module namespace. optional if we only have one foo.
class { '::foo':
}

and

# /etc/puppet/hiera.yaml
:backends:
        - yaml

:hierarchy:
        - globals
        - whatever
        - youlike

:yaml:
        :datadir: /etc/puppet/hieradata/

and

# /etc/puppet/hieradata/whatever.yaml (because of - whatever above)
---
foo::a: 'somevalue' # this many colons is actually valid syntax
dude: 'sweet'

will produce:

[...]
Notice: a is somevalue, b is: banana
Notice: Finished catalog run in 3.14159265359 seconds

This is the automatic lookup. You probably have zero risk of collision with earlier data in your hiera yaml files, because these lookups use keys that match the classname::paramname pattern. If you had used :: (double colons) in your keys before, then you’re insane, and you should check for any collisions! The downside to this is that my whatever.yaml looks awkward with all those colons, but I got over that very quickly.

The full lookup order is first:

# directly specified values first (of course)
class { '::foo':
        a => 'this value is used first if set.',
}

and then:

# values matching an appropriate yaml key:
---
foo::a: 'this value is used next if found.'

and finally:

class foo(
        $a = 'this parameter default value is used last.'
        $b = 'b is still for banana...'
) {
        # do stuff...
}

all as detailed in: http://projects.puppetlabs.com/issues/11608. Finding this link and setting me down the path to knowledge was all thanks to eric0 in #puppet. Thanks Eric!

Make sure to reload your puppetmaster after you make any changes to /etc/puppet/hiera.yaml, and as always:

Happy hacking,

James

SElinux causes pain when using puppet 2.x with hiera

So hiera wasn’t working when used through my puppetmaster. It worked perfectly when I was running my scripts manually with: puppet apply site.pp but the moment I switched over to regular puppetmasterd usage, everything went dead.

I realized a while back, that I could always expect some hiera failures from time to time. Whether this is hiera’s fault or not is irrelevant, the relevant part is that I quickly added:

$test = hiera('test', '')    # expects: 'Hiera is working!'
if "${test}" == '' {
    fail('The hiera function is not working as expected!')
}

to my site.pp and now I don’t risk breaking/changing a node because some data is missing. Naturally you’ll need to add a little message in your globals.yaml or similar. If you’re really cautious, you could change the above to an inequality and have your code “expect” a magic message.

To continue with the story, blkperl from #puppet recommended that I try running puppetmasterd manually to watch for any messages it might print. To my surprise, things started working! This just shows you how helpful it is to have a second set of eyes to help you on your way. So why did this work ?

To make a long story short, after following a few dead ends, it hit me, and so I looked in /var/log/audit/audit.log:

type=AVC msg=audit(1358404083.789:55416): avc:  denied  { getattr } for  pid=13830 comm="puppetmasterd" path="/etc/puppet/hiera.yaml" dev=dm-0 ino=18613624 scontext=unconfined_u:system_r:puppetmaster_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file

to find that selinux was once again causing me pain. When I had started puppetmasterd manually, I was root, which allowed me to bypass selinux rules. Sadly, I like selinux, but since I’m not nearly clever enough to want to learn how to fix this the right way, it just got disabled on another one of my machines.

Running:

restorecon -v /etc/puppet/hiera.yaml

fixed the bad selinux context I had on that file.

Hope this saves you some time and,

Happy hacking,

James

Dynamically including classes in puppet

As you might already know, I like pushing the boundaries of what puppet is able to do. Today, I realized that I needed to include a class by variable name. A simple way to do this is possible with:

$foo = 'world'
class { "hello::${foo}::params":
}

I then realized that you could also do this with the standard include keyword:

$foo = 'world'
include "hello::${foo}::params"

If you’re really insane enough to need to have your include depend on a variable, then this should suit your needs. The advantage of the second form is that if that statement appears more than once, you won’t cause a “duplicate definition” error.

Recently, I’ve been playing around with hiera. For completeness, I will mention one more form. It turns out that the hiera puppet functions offer a hiera_include function. As with all the hiera functions, the second argument lets you specify a default:

$foo = 'world'
hiera_include('...', "hello::${foo}::params")

which finishes off the trilogy. Hope this was useful, now start getting creative and

Happy hacking,

James