Finite state machines in puppet

In my attempt to push puppet to its limits, (for no particular reason), to develop more powerful puppet modules, to build in a distributed lock manager, and to be more dynamic, I’m now attempting to build a Finite State Machine (FSM) in puppet.

Is this a real finite state machine, and why would you do this?

Computer science professionals might not approve of the purity level, but they will hopefully appreciate the hack value. I’ve done this to illustrate a state transition technique that will be necessary in a module that I am writing.

Can we have an example?

Sure! I’ve decided to model thermodynamic phase transitions. Here’s what we’re building:

Phase_change_-_en.svg

How does it work?

Start off with a given define that accepts an argument. It could have one argument, or many, and be of whichever type you like, such as an integer, or even a more complicated list type. To keep the example simple, let’s work with a single argument named $input.

define fsm::transition(
        $input = ''
) {
        # TODO: add amazing code here...
}

The FSM runs as follows: On first execution, the $input value is saved to a local file by means of a puppet exec type. A corresponding fact exists to read from that file and create a unique variable for the fsm::transition type. Let’s call that variable $last. This is the special part!

# ruby fact to pull in the data from the state file
found = {}
Dir.glob(transition_dir+'*').each do |d|
    n = File.basename(d)    # should be the fsm::transition name
    if n.length > 0 and regexp.match(n)
        f = d.gsub(/\/$/, '')+'/state'    # full file path
        if File.exists?(f)
            # TODO: future versions should unpickle (but with yaml)
            v = File.open(f, 'r').read.strip    # read into str
            if v.length > 0 and regexp.match(v)
                found[n] = v
            end
        end
    end
end

found.keys.each do |x|
    Facter.add('fsm_transition_'+x) do
        #confine :operatingsystem => %w{CentOS, RedHat, Fedora}
        setcode {
            found[x]
        }
    end
end

On subsequent runs, the process gets more interesting: The $input value and the $last value are used to decide what to run. They can be different because the user might have changed the $input value. Logic trees then decide what actions you’d like to perform. This lets us compare the previous state to the new desired state, and as a result, be more intelligent about which actions need to run for a successful state transition. This is the FSM part.

# logic tree modeling phase transitions
# https://en.wikipedia.org/wiki/Phase_transition
$transition = "${valid_last}" ? {
        'solid' => "${valid_input}" ? {
               'solid' => true,
               'liquid' => 'melting',
               'gas' => 'sublimation',
               'plasma' => false,
               default => '',
        },
        'liquid' => "${valid_input}" ? {
               'solid' => 'freezing',
               'liquid' => true,
               'gas' => 'vaporization',
               'plasma' => false,
               default => '',
        },
        'gas' => "${valid_input}" ? {
               'solid' => 'deposition',
               'liquid' => 'condensation',
               'gas' => true,
               'plasma' => 'ionization',
               default => '',
        },
        'plasma' => "${valid_input}" ? {
               'solid' => false,
               'liquid' => false,
               'gas' => 'recombination',
               'plasma' => true,
               default => '',
        },
        default => '',
}

Once the state transition actions have completed successfully, the exec must store the $input value in the local file for future use as the unique $last fact for the next puppet run. If there are errors during state transition execution, you may choose to not store the updated value (to cause a re-run) and/or to add an error condition fact that the subsequent puppet run will have to read in and handle accordingly. This is the important part.

$f = "${vardir}/transition/${name}/state"
$diff = "/usr/bin/test '${valid_input}' != '${valid_last}'"

# TODO: future versions should pickle (but with yaml)
exec { "/bin/echo '${valid_input}' > '${f}'":
        logoutput => on_failure,
        onlyif => "/usr/bin/test ! -e '${f}' || ${diff}",
        require => File["${vardir}/"],
        alias => "fsm-transition-${name}",
}

Can we take this further?

It might be beneficial to remember the path we took through our graph. To do this, on each transition we append the new state to a file on our local puppet client. The corresponding fact, is similar to the $last fact, except it maintains a list of values instead of just one. There is a max length variable that can be used to avoid storing unlimited old states.

Does this have a practical use?

Yes, absolutely! I realized that something like this could be useful for puppet-gluster. Stay tuned for more patches.

Hopefully you enjoyed this. By following the above guidelines, you should now have some extra tricks for building state transitions into your puppet modules. Let me know if you found this hack awesome and unique.

I’ve posted the full example module here.

Happy Hacking,

James

 

Installing missing GNOME games

I just realized that my Fedora 19 installation didn’t have any of the GNOME games installed by default any more. I guess there’s no love for nibbles. Here’s a quick one-liner to get them all back:

$ sudo yum search game | grep gnome | awk '{print $1}' | xargs sudo yum install -y
Loaded plugins: etckeeper, langpacks, refresh-packagekit
Package gnome-nibbles-3.8.0-2.fc19.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package gnome-chess.x86_64 0:3.8.3-2.fc19 will be installed
--> Processing Dependency: gnuchess for package: gnome-chess-3.8.3-2.fc19.x86_64
---> Package gnome-hearts.x86_64 0:0.3-12.fc19 will be installed
---> Package gnome-klotski.x86_64 0:3.8.2-2.fc19 will be installed
---> Package gnome-mahjongg.x86_64 0:3.8.0-3.fc19 will be installed
---> Package gnome-mines.x86_64 0:3.8.1-3.fc19 will be installed
---> Package gnome-robots.x86_64 0:3.8.1-3.fc19 will be installed
---> Package gnome-sudoku.noarch 1:3.8.1-3.fc19 will be installed
---> Package gnome-tetravex.x86_64 0:3.8.1-2.fc19 will be installed
--> Running transaction check
---> Package gnuchess.x86_64 0:6.0.3-1.fc19 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package               Arch          Version                Repository     Size
================================================================================
Installing:
 gnome-chess           x86_64        3.8.3-2.fc19           fedora        2.6 M
 gnome-hearts          x86_64        0.3-12.fc19            fedora        365 k
 gnome-klotski         x86_64        3.8.2-2.fc19           fedora        1.2 M
 gnome-mahjongg        x86_64        3.8.0-3.fc19           fedora        3.8 M
 gnome-mines           x86_64        3.8.1-3.fc19           fedora        2.7 M
 gnome-robots          x86_64        3.8.1-3.fc19           fedora        1.4 M
 gnome-sudoku          noarch        1:3.8.1-3.fc19         fedora        2.5 M
 gnome-tetravex        x86_64        3.8.1-2.fc19           fedora        1.7 M
Installing for dependencies:
 gnuchess              x86_64        6.0.3-1.fc19           fedora        2.6 M

Transaction Summary
================================================================================
Install  8 Packages (+1 Dependent package)

Total download size: 19 M
Installed size: 54 M
Downloading packages:
(1/9): gnome-chess-3.8.3-2.fc19.x86_64.rpm                 | 2.6 MB   00:04     
(2/9): gnome-mines-3.8.1-3.fc19.x86_64.rpm                 | 2.7 MB   00:05     
(3/9): gnome-klotski-3.8.2-2.fc19.x86_64.rpm               | 1.2 MB   00:05     
(4/9): gnome-robots-3.8.1-3.fc19.x86_64.rpm                | 1.4 MB   00:01     
(5/9): gnome-tetravex-3.8.1-2.fc19.x86_64.rpm                                                                                          | 1.7 MB  00:00:03     
(6/9): gnome-hearts-0.3-12.fc19.x86_64.rpm                                                                                             | 365 kB  00:00:10     
(7/9): gnuchess-6.0.3-1.fc19.x86_64.rpm                                                                                                | 2.6 MB  00:00:04     
(8/9): gnome-mahjongg-3.8.0-3.fc19.x86_64.rpm                                                                                          | 3.8 MB  00:00:11     
(9/9): gnome-sudoku-3.8.1-3.fc19.noarch.rpm                                                                                            | 2.5 MB  00:00:06     
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                         1.5 MB/s |  19 MB     00:12     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
etckeeper: pre transaction commit
  Installing : gnuchess-6.0.3-1.fc19.x86_64                                                                                                               1/9 
  Installing : gnome-chess-3.8.3-2.fc19.x86_64                                                                                                            2/9 
  Installing : gnome-klotski-3.8.2-2.fc19.x86_64                                                                                                          3/9 
  Installing : gnome-tetravex-3.8.1-2.fc19.x86_64                                                                                                         4/9 
  Installing : gnome-mines-3.8.1-3.fc19.x86_64                                                                                                            5/9 
  Installing : 1:gnome-sudoku-3.8.1-3.fc19.noarch                                                                                                         6/9 
  Installing : gnome-hearts-0.3-12.fc19.x86_64                                                                                                            7/9 
  Installing : gnome-mahjongg-3.8.0-3.fc19.x86_64                                                                                                         8/9 
  Installing : gnome-robots-3.8.1-3.fc19.x86_64                                                                                                           9/9 
etckeeper: post transaction commit
  Verifying  : gnome-chess-3.8.3-2.fc19.x86_64                                                                                                            1/9 
  Verifying  : gnuchess-6.0.3-1.fc19.x86_64                                                                                                               2/9 
  Verifying  : gnome-robots-3.8.1-3.fc19.x86_64                                                                                                           3/9 
  Verifying  : gnome-mahjongg-3.8.0-3.fc19.x86_64                                                                                                         4/9 
  Verifying  : gnome-hearts-0.3-12.fc19.x86_64                                                                                                            5/9 
  Verifying  : 1:gnome-sudoku-3.8.1-3.fc19.noarch                                                                                                         6/9 
  Verifying  : gnome-mines-3.8.1-3.fc19.x86_64                                                                                                            7/9 
  Verifying  : gnome-tetravex-3.8.1-2.fc19.x86_64                                                                                                         8/9 
  Verifying  : gnome-klotski-3.8.2-2.fc19.x86_64                                                                                                          9/9 

Installed:
  gnome-chess.x86_64 0:3.8.3-2.fc19    gnome-hearts.x86_64 0:0.3-12.fc19     gnome-klotski.x86_64 0:3.8.2-2.fc19    gnome-mahjongg.x86_64 0:3.8.0-3.fc19   
  gnome-mines.x86_64 0:3.8.1-3.fc19    gnome-robots.x86_64 0:3.8.1-3.fc19    gnome-sudoku.noarch 1:3.8.1-3.fc19     gnome-tetravex.x86_64 0:3.8.1-2.fc19   

Dependency Installed:
  gnuchess.x86_64 0:6.0.3-1.fc19                                                                                                                              

Complete!
$

This isn’t a master hack, but it’s good to think about using command line magic to do your day-to-day tasks.

Happy gaming,

James

PS: I also updated the blog’s theme. Let me know if you hate it.

Bittorent sync for repository mirroring

Theron Conrey writes about using:

BitTorrent Sync as Geo-Replication for Storage

We got a chance to talk about this idea at Linuxcon. I’m not entirely convinced there aren’t some problem edge cases with this solution, but I think it will be hard to tell as long as the BitTorrent sync library is proprietary. I did come up with a special case of Theron’s idea that I believe could work well.

The special case uses the optimization that the synchronization (or file transferring) is unidirectional. This avoids any coherency complications involved if both sides were to write to the same file. Combined with the BitTorrent protocol, this does what normal torrent usage does, except with BitTorrent sync, we’re looking at a folder full of files.

What kind of synchronization would benefit from this model? Repository mirroring! This is exactly a folder full of files, but going in only one direction. Instead of yum or deb mirrors each running rsync, they could use BitTorrent sync, and because of the large amount of available upload bandwidth usually available on these mirrors, “seeding”, wouldn’t be a problem, and the worldwide pool would synchronize faster.

Can we apply this to user mirroring, net installers, and machine updating? Absolutely. I believe someone has already looked into the updates scenario, but it didn’t progress for some reason. The more convincing case is still the server geo-replication of course.

Obviously, using glusterfs with puppet-gluster to host the mirrors could be a good fit. You might not even need to use any gluster replication when you have built-in geo-replication via other mirrors.

If someone works up the open source BitTorrent parts, I’m happy to hack together the puppet parts to turn this into a turn-key solution for mirror hosts.

Hope you liked this idea.

Happy hacking,

James

Gluster Community Day, Thursday

I’m here in New Orleans hacking up a storm and getting to meet fellow gluster users IRL. John Mark Walker started off with a great “State of the GlusterFS union” style talk.

Today Louis (semiosis) gave a great talk about running glusterfs on amazon. It was highly pragmatic and he explained how he chose the number of bricks per host. The talk will be posted online shortly.

Marco Ceppi from Canonical gave a talk about juju and gluster. I haven’t had much time to look at juju, so it was good exposure. Marco’s gluster charm suffers from a lack of high availability peering, but I’m sure that is easily solved, and it isn’t a big issue. I had the same issue when working on puppet-gluster. I’ve written an article about how I solved this problem. I think it’s the most elegant solution, but if anyone has a better idea, please let me know. The solutions I used for puppet, can be applied to juju too. Marco and I talked about porting puppet-gluster to ubuntu. We also talked about using puppet inside of juju, with a puppetmaster, but we’re not sure how useful that would be beyond pure hack value.

Joe Julian gave a talk on running a MySQL (MariaDB) on glusterfs and getting mostly decent performance. That man knows his gluster internals.

I presented my talk about puppet-gluster. I had a successful live demo, which ran over ssh+screen across the conference centre internet to my home cluster Montreal. With interspersed talking, the full deploy took about eight minutes. Hope you enjoyed it. Let me know if you have any trouble with your setup and what features you’re missing. The video will be posted shortly.

Thanks again to John Mark Walker, RedHat and gluster.org for sponsoring my trip.

Happy hacking,

James

Linuxcon day three, Wednesday

After hacking away on Monday and Tuesday and meeting fellow nerds IRL, I’ve landed even more changes to puppet-gluster. My git master branch now sits at 47 commits.

$ git clone https://github.com/purpleidea/puppet-gluster.git
Cloning into 'puppet-gluster'...
remote: Counting objects: 317, done.
remote: Compressing objects: 100% (144/144), done.
remote: Total 317 (delta 187), reused 275 (delta 148)
Receiving objects: 100% (317/317), 82.17 KiB | 12.00 KiB/s, done.
Resolving deltas: 100% (187/187), done.
$ cd puppet-gluster/
$ git log | grep '^commit' | wc -l
47
$ git log | head
commit fa3fd2eb4bab499031274e0918a40e7a99fe0086
Author: James Shubin <hidden>
Date:   Wed Sep 18 17:53:13 2013 -0400

    Added fancy volume creation.
    
    This moves the command into a separate file. This also adds temporary
    saving of stdout and stderr to /tmp for easy debugging of command
    output.

As you can see above, volume creation is now “fancier” and more robust. In case things go wrong, it’s easy to get fast access to gluster command line output (saved in /tmp/), and the volume creation commands are individually stored in your puppet-gluster working directory. Usually this is /var/lib/puppet/tmp/gluster/, and each volume creation command is in the volume subdirectory.

I also met gluster expert Joe Julian. He’s been recently hired at Rackspace. Congratulations Joe. We talked about puppet and gluster, and is very knowledgeable about gluster internals and PRN source diving.

I was interviewed by Aaron Delp and Brian Gracely, on The Cloudcast. These two gentlemen are a pleasure to sit and chat with. Check out their podcast. We talked about puppet, gluster, puppet-gluster and how to dive in. Feel free to comment or email me if you have any questions about something that we didn’t cover in the interview.

All week, I’ve been hacking along side Jayneil Dalal in the speaker room. He was kind enough to give me a Beagle Bone black! Where will its hack potential take you? Two features which are particularly useful are on-board Ethernet, and 2GB of flash storage. He’s at the conference showing off some Minnow boards. They’ve got an Intel atom chip on board if you need something a little beefier.

I’m giving my puppet-gluster talk tomorrow (Thursday) here at Linuxcon! I hope you can make it. I’ll even have a live demo. Until then,

Happy hacking,

James

Linuxcon day two, Tuesday

Continuing on from yesterday, I’ve met even more interesting people. I chatted with Dianne Mueller about some interesting ideas for gluster+openshift. More to come on that front soon. Hung out with Jono Bacon and talked a bit about puppet-gluster on Ubuntu. If there is interest in the community for this, please let me know. Thanks to John Mark Walker and RedHat for sponsoring me and introducing me to many of these folks. Hello to all the others that I didn’t mention.

On the hacking side of things, I added proper xml parsing, and a lot of work on fancier firewalling to puppet-gluster. At the moment, here’s how the firewall support works:

  1. Initially, each host doesn’t know about the other nodes.
  2. Puppet runs and each host exports host information to each other node. This opens up the firewall for glusterd so that the hosts can peer.
  3. Now that we know which hosts are in a common pool, we can open up the firewall for each volume’s bricks. Since the volume has not yet been started (or even created) we can’t know which ports are needed, so all incoming ports are permitted from other gluster nodes.
  4. Once the volume is created, and started, the TCP port information will be available, and can be consumed as facts. These facts then refine the previously defined firewall rules, to only allow the needed ports.
  5. Your white-listed firewall setup is now complete.
  6. For users who wish to avoid using this module to configure your firewall, you can set shorewall => false in your gluster::server class. If you want to specify the allowed ip access control manually, that is possible too.

I hope you find this useful. I know I do. Let me know, and

Happy Hacking,

James