GNOME Montreal Summit

This October 12th to 14th Montreal hosted the GNOME boston summit. Many thanks to Canonical for sponsoring breakfast, Savoir Faire Linux for hosting a great 6 à 10 with fancy snacks, and RedHat for sponsoring a pool night. What follows is some technical commentary about stuff that went on.

JHBuild

JHBuild is a tool to make it easy to download/clone (from git) and compile all the GNOME modules and applications. It was easy to get going. I (mostly) followed the steps listed on the JHBuild HowDoI wiki page. I kept my .jhbuildrc in my home directory instead of ~/.config/. On my Fedora 19 machine, I found certain unlisted dependencies were missing to build everything. You can figure these out yourself when your builds fail, or just run the following command to install them beforehand:

# yum install /usr/bin/{g++,a2x,gnome-doc-prepare} python-rdflib lua-devel

The abridged list of commands that I ran includes:

$ git clone git://git.gnome.org/jhbuild
$ ./autogen.sh --simple-install
$ make
$ make install
$ ln -s ~/.local/bin/jhbuild ~/bin/jhbuild
$ jhbuild sysdeps --install
$ jhbuild buildone gnome-themes-standard    # i was impatient to get this early
$ jhbuild update                            # at home on fast network
$ jhbuild build --no-network                # can be run offline if you like
$ jhbuild run gedit
$ jhbuild shell                             # then run `env` inside the shell

You want to properly understand the context and working directory for each of these, but it should help get you to understand the process more quickly. One thing to realize is that the jhbuild run actually just runs a command from your $PATH, but from within the jhbuild environment, with modified $PATH and prefix variables. Run:

$ jhbuild run env

to get a better understanding of what changes. When patching, I recommend you clone locally from the mirrored git clones into your personal ~/code/ hacking directory. This way JHBuild can continue to do its thing, and keep the checkouts at master without your changes breaking the process.

Boxes

Boxes is a simple virtual machine manager for the GNOME desktop.

Zeeshan talked about some of the work he’s been doing in boxes. William Jon McCann reviewed some of the UI bugs, and I proposed the idea of integrating puppet modules so that a user could pick a module, and have a test environment running in minutes. This could be a way for users to try GlusterFS or FreeIPA.

GNOME Continuous

GNOME continuous (formerly known as GNOME-OSTree) is a project by Colin Walters. It is amazing for two reasons in particular:

1. It does continuous integration testing on a running OS image built from GNOME git master. This has enabled Colin to catch when commits break apps. It even takes screenshots so that you see what the running apps look like.
2. It provides bootable GNOME images that you can run locally in a vm. The images are built from GNOME git master. While there are no security updates, and this is not recommended for normal use, it is a great way to test out the bleeding edge of GNOME and report bugs early. It comes with an atomic upgrade method to keep your image up to date with the latest commits.

My goal is to try to run the bootable image for an hour every month. This way, I’ll be able to catch GNOME bugs early before they trickle down into Fedora and other GNOME releases.

Abiword 3.0

Hubert Figuière released AbiWord 3.0 which supports GTK+ 3. Cool.

A11y

Some of the A11y team was present and hacking away. I learned more about how the accessibility tools are used. This is an important technology even if you aren’t using them at the moment. With age, and accident, the current A11y technologies might one day be useful to you!

Hacking Hearts

Karen Sandler was kind enough to tell me about her heart condition and the proprietary software inside of her pacemaker defibrillator. Hopefully hackers can convince manufacturers that we need to have access to the source.

Yorba and Geary

Jim Nelson is a captivating conversationalist, and it was kind of him to tell me about his work on Geary and Vala, and to listen to my ideas about GPG support in email clients. I look forward to seeing Geary’s future. I also learned a bit more about IMAP.

Future?

I had a great time at the summit, and it was a pleasure to meet and hangout with the GNOME hackers. You’re always welcome in Montreal, and I hope you come back soon.

Happy Hacking,

James

Show current git branch in PS1 when branch is not master

Short post, long command…

I’ve decided to start showing the current git branch in my PS1. However, since I don’t want to know when I’m on master, I had to write a new PS1 that I haven’t yet seen anywhere. Add the following to your .bashrc:

PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
if [ -e /usr/share/git-core/contrib/completion/git-prompt.sh ]; then
    . /usr/share/git-core/contrib/completion/git-prompt.sh
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w$([ "$(__git_ps1 %s)" != "" -a "$(__git_ps1 %s)" != "master" ] && (echo -e " (\[33[32m\]"$(__git_ps1 "%s")"\[33[0m\])") || echo "")\$ '
fi

This keeps my PS1 short for when I’m hacking on personal repositories that only have a single branch. Keep in mind that you might have to change the path to git-prompt.sh depending on what OS you’re using.

Example:

james@computer:~/code/puppet$ cd puppet-gluster
james@computer:~/code/puppet/puppet-gluster$ git checkout -b cool-new-feature
Switched to a new branch 'cool-new-feature'
james@computer:~/code/puppet/puppet-gluster (cool-new-feature)$ # tada !

The branch name is coloured to match the default colours that git uses to colour branches.

Happy hacking,

James

 

Finite state machines in puppet

In my attempt to push puppet to its limits, (for no particular reason), to develop more powerful puppet modules, to build in a distributed lock manager, and to be more dynamic, I’m now attempting to build a Finite State Machine (FSM) in puppet.

Is this a real finite state machine, and why would you do this?

Computer science professionals might not approve of the purity level, but they will hopefully appreciate the hack value. I’ve done this to illustrate a state transition technique that will be necessary in a module that I am writing.

Can we have an example?

Sure! I’ve decided to model thermodynamic phase transitions. Here’s what we’re building:

How does it work?

Start off with a given define that accepts an argument. It could have one argument, or many, and be of whichever type you like, such as an integer, or even a more complicated list type. To keep the example simple, let’s work with a single argument named $input.

define fsm::transition(
        $input = ''
) {
        # TODO: add amazing code here...
}

The FSM runs as follows: On first execution, the $input value is saved to a local file by means of a puppet exec type. A corresponding fact exists to read from that file and create a unique variable for the fsm::transition type. Let’s call that variable $last. This is the special part!

# ruby fact to pull in the data from the state file
found = {}
Dir.glob(transition_dir+'*').each do |d|
    n = File.basename(d)    # should be the fsm::transition name
    if n.length > 0 and regexp.match(n)
        f = d.gsub(/\/$/, '')+'/state'    # full file path
        if File.exists?(f)
            # TODO: future versions should unpickle (but with yaml)
            v = File.open(f, 'r').read.strip    # read into str
            if v.length > 0 and regexp.match(v)
                found[n] = v
            end
        end
    end
end

found.keys.each do |x|
    Facter.add('fsm_transition_'+x) do
        #confine :operatingsystem => %w{CentOS, RedHat, Fedora}
        setcode {
            found[x]
        }
    end
end

On subsequent runs, the process gets more interesting: The $input value and the $last value are used to decide what to run. They can be different because the user might have changed the $input value. Logic trees then decide what actions you’d like to perform. This lets us compare the previous state to the new desired state, and as a result, be more intelligent about which actions need to run for a successful state transition. This is the FSM part.

# logic tree modeling phase transitions
# https://en.wikipedia.org/wiki/Phase_transition
$transition = "${valid_last}" ? {
        'solid' => "${valid_input}" ? {
               'solid' => true,
               'liquid' => 'melting',
               'gas' => 'sublimation',
               'plasma' => false,
               default => '',
        },
        'liquid' => "${valid_input}" ? {
               'solid' => 'freezing',
               'liquid' => true,
               'gas' => 'vaporization',
               'plasma' => false,
               default => '',
        },
        'gas' => "${valid_input}" ? {
               'solid' => 'deposition',
               'liquid' => 'condensation',
               'gas' => true,
               'plasma' => 'ionization',
               default => '',
        },
        'plasma' => "${valid_input}" ? {
               'solid' => false,
               'liquid' => false,
               'gas' => 'recombination',
               'plasma' => true,
               default => '',
        },
        default => '',
}

Once the state transition actions have completed successfully, the exec must store the $input value in the local file for future use as the unique $last fact for the next puppet run. If there are errors during state transition execution, you may choose to not store the updated value (to cause a re-run) and/or to add an error condition fact that the subsequent puppet run will have to read in and handle accordingly. This is the important part.

$f = "${vardir}/transition/${name}/state"
$diff = "/usr/bin/test '${valid_input}' != '${valid_last}'"

# TODO: future versions should pickle (but with yaml)
exec { "/bin/echo '${valid_input}' > '${f}'":
        logoutput => on_failure,
        onlyif => "/usr/bin/test ! -e '${f}' || ${diff}",
        require => File["${vardir}/"],
        alias => "fsm-transition-${name}",
}

Can we take this further?

It might be beneficial to remember the path we took through our graph. To do this, on each transition we append the new state to a file on our local puppet client. The corresponding fact, is similar to the $last fact, except it maintains a list of values instead of just one. There is a max length variable that can be used to avoid storing unlimited old states.

Does this have a practical use?

Yes, absolutely! I realized that something like this could be useful for puppet-gluster. Stay tuned for more patches.

Hopefully you enjoyed this. By following the above guidelines, you should now have some extra tricks for building state transitions into your puppet modules. Let me know if you found this hack awesome and unique.

I’ve posted the full example module here.

Happy Hacking,

James

 

No Ads!

I just shelled out to wordpress.com to buy the No Ads upgrade. I think they’re good people. I hope you all appreciate it. Let me know if you do.

Happy hacking,

James

 

Installing missing GNOME games

I just realized that my Fedora 19 installation didn’t have any of the GNOME games installed by default any more. I guess there’s no love for nibbles. Here’s a quick one-liner to get them all back:

$ sudo yum search game | grep gnome | awk '{print $1}' | xargs sudo yum install -y
Loaded plugins: etckeeper, langpacks, refresh-packagekit
Package gnome-nibbles-3.8.0-2.fc19.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package gnome-chess.x86_64 0:3.8.3-2.fc19 will be installed
--> Processing Dependency: gnuchess for package: gnome-chess-3.8.3-2.fc19.x86_64
---> Package gnome-hearts.x86_64 0:0.3-12.fc19 will be installed
---> Package gnome-klotski.x86_64 0:3.8.2-2.fc19 will be installed
---> Package gnome-mahjongg.x86_64 0:3.8.0-3.fc19 will be installed
---> Package gnome-mines.x86_64 0:3.8.1-3.fc19 will be installed
---> Package gnome-robots.x86_64 0:3.8.1-3.fc19 will be installed
---> Package gnome-sudoku.noarch 1:3.8.1-3.fc19 will be installed
---> Package gnome-tetravex.x86_64 0:3.8.1-2.fc19 will be installed
--> Running transaction check
---> Package gnuchess.x86_64 0:6.0.3-1.fc19 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package               Arch          Version                Repository     Size
================================================================================
Installing:
 gnome-chess           x86_64        3.8.3-2.fc19           fedora        2.6 M
 gnome-hearts          x86_64        0.3-12.fc19            fedora        365 k
 gnome-klotski         x86_64        3.8.2-2.fc19           fedora        1.2 M
 gnome-mahjongg        x86_64        3.8.0-3.fc19           fedora        3.8 M
 gnome-mines           x86_64        3.8.1-3.fc19           fedora        2.7 M
 gnome-robots          x86_64        3.8.1-3.fc19           fedora        1.4 M
 gnome-sudoku          noarch        1:3.8.1-3.fc19         fedora        2.5 M
 gnome-tetravex        x86_64        3.8.1-2.fc19           fedora        1.7 M
Installing for dependencies:
 gnuchess              x86_64        6.0.3-1.fc19           fedora        2.6 M

Transaction Summary
================================================================================
Install  8 Packages (+1 Dependent package)

Total download size: 19 M
Installed size: 54 M
Downloading packages:
(1/9): gnome-chess-3.8.3-2.fc19.x86_64.rpm                 | 2.6 MB   00:04     
(2/9): gnome-mines-3.8.1-3.fc19.x86_64.rpm                 | 2.7 MB   00:05     
(3/9): gnome-klotski-3.8.2-2.fc19.x86_64.rpm               | 1.2 MB   00:05     
(4/9): gnome-robots-3.8.1-3.fc19.x86_64.rpm                | 1.4 MB   00:01     
(5/9): gnome-tetravex-3.8.1-2.fc19.x86_64.rpm                                                                                          | 1.7 MB  00:00:03     
(6/9): gnome-hearts-0.3-12.fc19.x86_64.rpm                                                                                             | 365 kB  00:00:10     
(7/9): gnuchess-6.0.3-1.fc19.x86_64.rpm                                                                                                | 2.6 MB  00:00:04     
(8/9): gnome-mahjongg-3.8.0-3.fc19.x86_64.rpm                                                                                          | 3.8 MB  00:00:11     
(9/9): gnome-sudoku-3.8.1-3.fc19.noarch.rpm                                                                                            | 2.5 MB  00:00:06     
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                         1.5 MB/s |  19 MB     00:12     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
etckeeper: pre transaction commit
  Installing : gnuchess-6.0.3-1.fc19.x86_64                                                                                                               1/9 
  Installing : gnome-chess-3.8.3-2.fc19.x86_64                                                                                                            2/9 
  Installing : gnome-klotski-3.8.2-2.fc19.x86_64                                                                                                          3/9 
  Installing : gnome-tetravex-3.8.1-2.fc19.x86_64                                                                                                         4/9 
  Installing : gnome-mines-3.8.1-3.fc19.x86_64                                                                                                            5/9 
  Installing : 1:gnome-sudoku-3.8.1-3.fc19.noarch                                                                                                         6/9 
  Installing : gnome-hearts-0.3-12.fc19.x86_64                                                                                                            7/9 
  Installing : gnome-mahjongg-3.8.0-3.fc19.x86_64                                                                                                         8/9 
  Installing : gnome-robots-3.8.1-3.fc19.x86_64                                                                                                           9/9 
etckeeper: post transaction commit
  Verifying  : gnome-chess-3.8.3-2.fc19.x86_64                                                                                                            1/9 
  Verifying  : gnuchess-6.0.3-1.fc19.x86_64                                                                                                               2/9 
  Verifying  : gnome-robots-3.8.1-3.fc19.x86_64                                                                                                           3/9 
  Verifying  : gnome-mahjongg-3.8.0-3.fc19.x86_64                                                                                                         4/9 
  Verifying  : gnome-hearts-0.3-12.fc19.x86_64                                                                                                            5/9 
  Verifying  : 1:gnome-sudoku-3.8.1-3.fc19.noarch                                                                                                         6/9 
  Verifying  : gnome-mines-3.8.1-3.fc19.x86_64                                                                                                            7/9 
  Verifying  : gnome-tetravex-3.8.1-2.fc19.x86_64                                                                                                         8/9 
  Verifying  : gnome-klotski-3.8.2-2.fc19.x86_64                                                                                                          9/9 

Installed:
  gnome-chess.x86_64 0:3.8.3-2.fc19    gnome-hearts.x86_64 0:0.3-12.fc19     gnome-klotski.x86_64 0:3.8.2-2.fc19    gnome-mahjongg.x86_64 0:3.8.0-3.fc19   
  gnome-mines.x86_64 0:3.8.1-3.fc19    gnome-robots.x86_64 0:3.8.1-3.fc19    gnome-sudoku.noarch 1:3.8.1-3.fc19     gnome-tetravex.x86_64 0:3.8.1-2.fc19   

Dependency Installed:
  gnuchess.x86_64 0:6.0.3-1.fc19                                                                                                                              

Complete!
$

This isn’t a master hack, but it’s good to think about using command line magic to do your day-to-day tasks.

Happy gaming,

James

PS: I also updated the blog’s theme. Let me know if you hate it.

Bittorent sync for repository mirroring

Theron Conrey writes about using:

BitTorrent Sync as Geo-Replication for Storage

We got a chance to talk about this idea at Linuxcon. I’m not entirely convinced there aren’t some problem edge cases with this solution, but I think it will be hard to tell as long as the BitTorrent sync library is proprietary. I did come up with a special case of Theron’s idea that I believe could work well.

The special case uses the optimization that the synchronization (or file transferring) is unidirectional. This avoids any coherency complications involved if both sides were to write to the same file. Combined with the BitTorrent protocol, this does what normal torrent usage does, except with BitTorrent sync, we’re looking at a folder full of files.

What kind of synchronization would benefit from this model? Repository mirroring! This is exactly a folder full of files, but going in only one direction. Instead of yum or deb mirrors each running rsync, they could use BitTorrent sync, and because of the large amount of available upload bandwidth usually available on these mirrors, “seeding”, wouldn’t be a problem, and the worldwide pool would synchronize faster.

Can we apply this to user mirroring, net installers, and machine updating? Absolutely. I believe someone has already looked into the updates scenario, but it didn’t progress for some reason. The more convincing case is still the server geo-replication of course.

Obviously, using glusterfs with puppet-gluster to host the mirrors could be a good fit. You might not even need to use any gluster replication when you have built-in geo-replication via other mirrors.

If someone works up the open source BitTorrent parts, I’m happy to hack together the puppet parts to turn this into a turn-key solution for mirror hosts.

Hope you liked this idea.

Happy hacking,

James

Gluster Community Day, Thursday

I’m here in New Orleans hacking up a storm and getting to meet fellow gluster users IRL. John Mark Walker started off with a great “State of the GlusterFS union” style talk.

Today Louis (semiosis) gave a great talk about running glusterfs on amazon. It was highly pragmatic and he explained how he chose the number of bricks per host. The talk will be posted online shortly.

Marco Ceppi from Canonical gave a talk about juju and gluster. I haven’t had much time to look at juju, so it was good exposure. Marco’s gluster charm suffers from a lack of high availability peering, but I’m sure that is easily solved, and it isn’t a big issue. I had the same issue when working on puppet-gluster. I’ve written an article about how I solved this problem. I think it’s the most elegant solution, but if anyone has a better idea, please let me know. The solutions I used for puppet, can be applied to juju too. Marco and I talked about porting puppet-gluster to ubuntu. We also talked about using puppet inside of juju, with a puppetmaster, but we’re not sure how useful that would be beyond pure hack value.

Joe Julian gave a talk on running a MySQL (MariaDB) on glusterfs and getting mostly decent performance. That man knows his gluster internals.

I presented my talk about puppet-gluster. I had a successful live demo, which ran over ssh+screen across the conference centre internet to my home cluster Montreal. With interspersed talking, the full deploy took about eight minutes. Hope you enjoyed it. Let me know if you have any trouble with your setup and what features you’re missing. The video will be posted shortly.

Thanks again to John Mark Walker, RedHat and gluster.org for sponsoring my trip.

Happy hacking,

James