recursion in puppet (for no particular reason)

Posted on by
3

I’m working on some fancy puppet “code”, and I realized recursion could be very useful. I decided to try out a little hack to see if I could get it to work. I’ll jump right into the code:

#!/usr/bin/puppet

define recursion(
    $count
) {
    notify { "count-${count}":
    }
    $minus1 = inline_template('<%= count.to_i - 1 %>')
    if "${minus1}" == '0' {
        notify { 'done counting!':
        }
    } else {
        # recurse
        recursion { "count-${minus1}":
            count => $minus1,
        }
    }
}

# kick it off...
recursion { 'start':
    count => 4,
}

In theory, this should now work because of local variable scopes. Let’s see if we’ll blow up the puppet stack or not…

[james@computer tmp]$ ./rec.pp 
warning: Implicit invocation of 'puppet apply' by passing files (or flags) directly
to 'puppet' is deprecated, and will be removed in the 2.8 series.  Please
invoke 'puppet apply' directly in the future.

notice: count-4
notice: /Stage[main]//Recursion[start]/Notify[count-4]/message: defined 'message' as 'count-4'
notice: count-2
notice: /Stage[main]//Recursion[start]/Recursion[count-3]/Recursion[count-2]/Notify[count-2]/message: defined 'message' as 'count-2'
notice: count-3
notice: /Stage[main]//Recursion[start]/Recursion[count-3]/Notify[count-3]/message: defined 'message' as 'count-3'
notice: count-1
notice: /Stage[main]//Recursion[start]/Recursion[count-3]/Recursion[count-2]/Recursion[count-1]/Notify[count-1]/message: defined 'message' as 'count-1'
notice: done counting!
notice: /Stage[main]//Recursion[start]/Recursion[count-3]/Recursion[count-2]/Recursion[count-1]/Notify[done counting!]/message: defined 'message' as 'done counting!'
notice: Finished catalog run in 0.16 seconds
[james@computer tmp]$

…and amazingly this seems to work! Hopefully this will be useful for some upcoming trickery I have planned, and if not, it was a fun hack.

I decided to see if it could handle larger values, and for my simple tests, it seemed to do okay:

notice: /Stage[main]//Recursion[start]/Recursion[count-99]/Recursion[count-98]/Recursion[count-97]/Recursion[count-96]/Recursion[count-95]/Recursion[count-94]/Recursion[count-93]/Recursion[count-92]/Recursion[count-91]/Recursion[count-90]/Recursion[count-89]/Recursion[count-88]/Recursion[count-87]/Recursion[count-86]/Recursion[count-85]/Recursion[count-84]/Recursion[count-83]/Recursion[count-82]/Recursion[count-81]/Recursion[count-80]/Recursion[count-79]/Recursion[count-78]/Recursion[count-77]/Recursion[count-76]/Recursion[count-75]/Recursion[count-74]/Recursion[count-73]/Recursion[count-72]/Recursion[count-71]/Recursion[count-70]/Recursion[count-69]/Recursion[count-68]/Recursion[count-67]/Recursion[count-66]/Recursion[count-65]/Recursion[count-64]/Recursion[count-63]/Recursion[count-62]/Recursion[count-61]/Recursion[count-60]/Recursion[count-59]/Recursion[count-58]/Recursion[count-57]/Recursion[count-56]/Recursion[count-55]/Recursion[count-54]/Recursion[count-53]/Recursion[count-52]/Recursion[count-51]/Recursion[count-50]/Recursion[count-49]/Recursion[count-48]/Recursion[count-47]/Recursion[count-46]/Recursion[count-45]/Recursion[count-44]/Recursion[count-43]/Recursion[count-42]/Recursion[count-41]/Recursion[count-40]/Recursion[count-39]/Recursion[count-38]/Recursion[count-37]/Recursion[count-36]/Recursion[count-35]/Recursion[count-34]/Recursion[count-33]/Recursion[count-32]/Recursion[count-31]/Recursion[count-30]/Recursion[count-29]/Recursion[count-28]/Recursion[count-27]/Recursion[count-26]/Recursion[count-25]/Recursion[count-24]/Recursion[count-23]/Recursion[count-22]/Recursion[count-21]/Recursion[count-20]/Recursion[count-19]/Recursion[count-18]/Recursion[count-17]/Recursion[count-16]/Recursion[count-15]/Recursion[count-14]/Recursion[count-13]/Recursion[count-12]/Recursion[count-11]/Recursion[count-10]/Recursion[count-9]/Recursion[count-8]/Recursion[count-7]/Recursion[count-6]/Recursion[count-5]/Recursion[count-4]/Recursion[count-3]/Recursion[count-2]/Recursion[count-1]/Notify[done counting!]/message: defined 'message' as 'done counting!'
notice: Finished catalog run in 1.16 seconds

Running this with a count value of 1000 took 132.19 sec according to puppet, but much longer for the process to actually clean up and finish. This made my fan speed up, but at least it didn’t segfault.

Hopefully I’ll have something more useful to show you next time, but until then, keep on imagining and,

Happy hacking!

James

EDIT: A follow up is now available.

continuous display of log files (better tail -f)

Posted on by
7

All good sysadmins know about using tail -f to follow a log file. I use this all the time to follow /var/log/messages and my gluster logs in particular. Maybe everyone already knows this, but it deserves a PSA: after a certain amount of time (~days) it seems that new messages don’t appear!

What happens by default is that tail -f follows the file descriptor, not the file name, so when your log files get rotated, the file descriptor still points to the (now renamed) file which no longer gets updates any more.

The solution is to get tail to follow the file name you’re interested in:

tail --follow=name /var/log/messages

EDIT: Fortunately there is a shorter way of running this too, you can use:

tail -F

on any up to date version of tail to get the same result. This adds in –retry to the –folllow=name argument.

Happy hacking!

James

 

setting timed events in puppet

Posted on by
3

I’ve tried to push puppet to its limits, and so far I’ve succeeded. When you hit the kind of bug that forces you to hack around it, you know you are close. In any case, this isn’t about that embarrassing bug, it’s about how to set delayed actions in puppet.

Enter puppet-runonce, a module that I’ve just finished writing. It starts off with the realization that you can exec an action which also writes to a file. If it sees this file, then it knows that it has already completed, and shouldn’t run itself again. The relevant parts are here:

define runonce::exec(
    $command = '/bin/true',
    $notify = undef,
    $repeat_on_failure = true
) {
    include runonce::exec::base

    $date = "/bin/date >> /var/lib/puppet/tmp/runonce/exec/${name}"
    $valid_command = $repeat_on_failure ? {
        false => "${date} && ${command}",
        default => "${command} && ${date}",
    }

    exec { "runonce-exec-${name}":
        command => "${valid_command}",
        creates => "/var/lib/puppet/tmp/runonce/exec/${name}",    # run once
        notify => $notify,
        # TODO: add any other parameters here that users wants such as cwd and environment...
        require => File['/var/lib/puppet/tmp/runonce/exec/'],
    }
}

This depends on having an isolated namespace per module. I need this in many of my modules, and I have chosen: “/var/lib/puppet/tmp/$modulename“. I’ve added the extra feature that this object can repeatedly run until the $command succeeds or it can run once, and ignore the exit status.

Building a timer is slightly trickier, but follows from the first concept. First create a runonce object which when used, creates a file with a timestamp of “now”. Next, create a new exec object which periodically checks the time, and once we’re past a certain delta, exec the desired command. That looks something like this:

# when this is first run by puppet, a "timestamp" matching the system clock is
# saved. every time puppet runs (usually every 30 minutes) it compares the
# timestamp to the current time, and if this difference exceeds that of the
# set delta, then the requested command is executed.
define runonce::timer(
    $command = '/bin/true',
    $delta = 3600,                # seconds to wait...
    $notify = undef,
    $repeat_on_failure = true
) {
    include runonce::timer::base

    # start the timer...
    exec { "/bin/date > /var/lib/puppet/tmp/runonce/start/${name}":
        creates => "/var/lib/puppet/tmp/runonce/start/${name}",    # run once
        notify => Exec["runonce-timer-${name}"],
        require => File['/var/lib/puppet/tmp/runonce/start/'],
        alias => "runonce-start-${name}",
    }

    $date = "/bin/date >> /var/lib/puppet/tmp/runonce/timer/${name}"
    $valid_command = $repeat_on_failure ? {
        false => "${date} && ${command}",
        default => "${command} && ${date}",
    }

    # end the timer and run command (or vice-versa)
    exec { "runonce-timer-${name}":
        command => "${valid_command}",
        creates => "/var/lib/puppet/tmp/runonce/timer/${name}",    # run once
        # NOTE: run if the difference between the current date and the
        # saved date (both converted to sec) is greater than the delta
        onlyif => "/usr/bin/test -e /var/lib/puppet/tmp/runonce/start/${name} && /usr/bin/test \$(( `/bin/date +%s` - `/usr/bin/head -n 1 /var/lib/puppet/tmp/runonce/start/${name} | /bin/date --file=- +%s` )) -gt ${delta}",
        notify => $notify,
        require => [
            File['/var/lib/puppet/tmp/runonce/timer/'],
            Exec["runonce-start-${name}"],
        ],
        # TODO: add any other parameters here that users wants such as cwd and environment...
    }
}

The real “magic” is in the power of bash, and its individual elegant pieces. The `date` command makes it easy to import a previous stored value with –file, and a bit of conversion glue and mathematics gives us:

/usr/bin/test -e ${startdatefile} && /usr/bin/test $(( `/bin/date +%s` - `/usr/bin/head -n 1 ${startdatefile} | /bin/date --file=- +%s` )) -gt ${deltaseconds}

It’s a big mouthful to digest on one line, however it’s probably write only code anyways, and isn’t really that complicated anyhow. One downside is that this is only evaluated every time puppet runs, so in other words it has the approximate granularity of 30 minutes. If you’re using this for anything precise, then you’re insane!

Speaking of sanity, why would anyone want such a thing? My use case is simple: I’m writing a fancy puppet-drbd module, to help me auto-deploy clusters. I always have to manually turn up the initial sync rate to get my cluster happy, but this should be reverted for normal use. The solution is to set an initial sync rate with runonce::exec, and revert it 24 hours later with runonce::timer!

Both this module and my drbd module will be released in the near future. All of this code is AGPLv3+ so please share and enjoy with those freedoms.

Happy hacking,
James

preventing duplicate parameter values in puppet types

Posted on by
Reply

I am writing a keepalived module for puppet. It will naturally be called: “puppet-keepalived”, and I will be releasing the code in the near future! In any case, if you’re familiar with VRRP, you’ll know that each managed link (eg: resource or VIP) has a common routerid and password which are shared among all members in the group. It is important that these parameters are unique across the type definitions on a single node.

Here is an example of two different instance definitions in puppet:

keepalived::vrrp { 'VI_NET':
    state => ...,
    routerid => 42, # must be unique
    password => 'somelongpassword...',
}

keepalived::vrrp { 'VI_DMZ':
    state => ...,
    routerid => 43, # must be unique
    password => 'somedifferentlongpassword...',
}

Here puppet guarantees that the $name variable is unique. Let’s extend this magic with a trick to make sure that routerid and password are too. Here is an excerpt from the relevant puppet definition:

define keepalived::vrrp(
    $state,
    ...
    $routerid,
    $password
) {
    ...
    file { "/etc/keepalived/${instance}.vrrp":
        content => template('keepalived/keepalived.vrrp.erb'),
        ...
        ensure => present,
        # NOTE: add unnecessary alias names so that if one of those
        # variables appears more than once, an error will be raised.
        alias => ["password-${password}", "routerid-${routerid}"],
    }
    ...
}

As you can see, multiple alias names are specified with an array, and since this file definition is used for each keepalived::vrrp instance, you’ll most assuredly cause a “duplicate alias” issue if there is a duplicate routerid or password used!

This trick will also probably work across define types too. To ensure a common key, just create an object like:

file { '/root/the_unique_key':
    alias => ["token1-${token1}", "token2-${token2}", "token3-${token3}"],
}

The token prefix will guarantee that you don’t accidentally cause a collision between dissimilar parameter values, unless that’s what you want. I’ve used a file in this scenario, but you can use whatever object you like. Because of this reason, it would make sense to create a noop() type if you’re really serious about this. Maybe puppet labs can add a built-in type upstream.

This is the type of thing that’s important to do if you want to write puppet code that acts less like a templating hack and more like a library :)

Happy hacking!

James

changing *that* keyboard shortcut right there (in gnome)

Posted on by
Reply

I love my keyboard shortcuts, and I sometimes I want to change them. If you’re ever in a gnome application and wanted to change *that one right there*, you can now live-edit them!

In a terminal, first enable this feature:

gsettings set org.gnome.desktop.interface can-change-accels true

Next, hover over the menu item shortcut that you want to change. Enter the shortcut you want. It should update immediately! I like to disable this live-editing, so that I don’t accidentally change any shortcuts. To do so run:

gsettings set org.gnome.desktop.interface can-change-accels false

Too bad firefox doesn’t support this. This is one more reason why native GTK apps make your entire experience blend together (consistent) and more magic!

Happy hacking!

James

PS: If you’re curious, I used this to change the gnome-terminal and gedit cycle tab left/right actions to instead respond to the thinkpad back/forward keys which are conveniently located right above the left and right arrow keys respectively.

Source: http://library.gnome.org/users/evolution/3.3/change-keyboard-shortcuts.html.en

Multifile mode for text editors

Posted on by
Reply

Dear internets,

I’m a sysadmin/architect, which means I spend a good amount of time coding in puppet and other languages. Puppet is a great tool, however the puppet community is a bit anal strict about their style policies. I can respect this because I understand how important uniformity is when many developers are sharing code.

(On a side note, I absolutely can’t stand using spaces for indentation, but that’s another story. Please, just tell people to use tabs – preferably with a width of eight spaces.)

The puppet community has a habit of splitting up each class, subclass and function (“define”) into a separate file. When I’m hacking on a new module, I usually have everything together inside one big init.pp file until it becomes big enough that I need to split it up.

What I’d like to do, is have separate files loaded into my text editor linearly, so that as I scroll up or down with my mousewheel or keyboard, gedit or vim smoothly transitions me from one file to another. It would have per file line numbering (as usual), and a soft visual break when you transitioned from one file to the next. It would feel like a single file!

Interface wise, gedit could allow you to group tabs into this single “multifile” mode, which lets the native tab semantics (drag to reorder, open, close, see name) reorder, add, remove and view file name, respectively. Clicking on that tab would smooth scroll you right to where that file starts. I’m not sure what the grouping/ungrouping mechanism should look like, because this will depend on what is possible/sane with GTK+.

Gedit devs, can you make this happen? I will be a happy programmer/sysadmin.

Happy hacking,

James