Remote execution in mgmt

Bootstrapping a cluster from your laptop, or managing machines without needing to first setup a separate config management infrastructure are both very reasonable and fundamental asks. I was particularly inspired by Ansible‘s agent-less remote execution model, but never wanted to build a centralized orchestrator. I soon realized that I could have my ice cream and eat it too.

Prior knowledge

If you haven’t read the earlier articles about mgmt, then I recommend you start with those, and then come back here. The first and fourth are essential if you’re going to make sense of this article.

Limitations of existing orchestrators

Current orchestrators have a few limitations.

  1. They can be a single point of failure
  2. They can have scaling issues
  3. They can’t respond instantaneously to node state changes (they poll)
  4. They can’t usually redistribute remote node run-time data between nodes

Despite these limitations, orchestration is still very useful because of the facilities it provides. Since these facilities are essential in a next generation design, I set about integrating these features, but with a novel twist.

Implementation, Usage and Design

Mgmt is written in golang, and that decision was no accident. One benefit is that it simplifies our remote execution model.

To use this mode you run mgmt with the --remote flag. Each use of the --remote argument points to a different remote graph to execute. Eventually this will be integrated with the DSL, but this plumbing is exposed for early adopters to play around with.

Startup (part one)

Each invocation of --remote causes mgmt to remotely connect over SSH to the target hosts. This happens in parallel, and runs up to --cconns simultaneous connections.

A temporary directory is made on the remote host, and the mgmt binary and graph are copied across the wire. Since mgmt compiles down to a single statically compiled binary, it simplifies the transfer of the software. The binary is cached remotely to speed up future runs unless you pass the --no-caching option.

A TCP connection is tunnelled back over SSH to the originating hosts etcd server which is embedded and running inside of the initiating mgmt binary.

Execution (part two)

The remote mgmt binary is now run! It wires itself up through the SSH tunnel so that its internal etcd client can connect to the etcd server on the initiating host. This is particularly powerful because remote hosts can now participate in resource exchanges as if they were part of a regular etcd backed mgmt cluster! They don’t connect directly to each other, but they can share runtime data, and only need an incoming SSH port open!

Closure (part three)

At this point mgmt can either keep running continuously or it can close the connections and shutdown.

In the former case, you can either remain attached over SSH, or you can disconnect from the child hosts and let this new cluster take on a new life and operate independently of the initiator.

In the latter case you can either shutdown at the operators request (via a ^C on the initiator) or when the cluster has simultaneously converged for a number of seconds.

This second possibility occurs when you run mgmt with the familiar --converged-timeout parameter. It is indeed clever enough to also work in this distributed fashion.


I’ve used by poor libreoffice draw skills to make a diagram. Hopefully this helps out my visual readers.


If you can improve this diagram, please let me know!


I find that using one or more vagrant virtual machines for the remote endpoints is the best way to test this out. In my case I use Oh-My-Vagrant to set up these machines, but the method you use is entirely up to you! Here’s a sample remote execution. Please note that I have omitted a number of lines for brevity, and added emphasis to the more interesting ones.

james@hostname:~/code/mgmt$ ./mgmt run --remote examples/remote2a.yaml --remote examples/remote2b.yaml --tmp-prefix 
17:58:22 main.go:76: This is: mgmt, version: 0.0.5-3-g4b8ad3a
17:58:23 remote.go:596: Remote: Connect...
17:58:23 remote.go:607: Remote: Sftp...
17:58:23 remote.go:164: Remote: Self executable is: /home/james/code/gopath/src/
17:58:23 remote.go:221: Remote: Remotely created: /tmp/mgmt-412078160/remote
17:58:23 remote.go:226: Remote: Remote path is: /tmp/mgmt-412078160/remote/mgmt
17:58:23 remote.go:221: Remote: Remotely created: /tmp/mgmt-412078160/remote
17:58:23 remote.go:226: Remote: Remote path is: /tmp/mgmt-412078160/remote/mgmt
17:58:23 remote.go:235: Remote: Copying binary, please be patient...
17:58:23 remote.go:235: Remote: Copying binary, please be patient...
17:58:24 remote.go:256: Remote: Copying graph definition...
17:58:24 remote.go:618: Remote: Tunnelling...
17:58:24 remote.go:630: Remote: Exec...
17:58:24 remote.go:510: Remote: Running: /tmp/mgmt-412078160/remote/mgmt run --hostname '' --no-server --seeds '' --file '/tmp/mgmt-412078160/remote/remote2a.yaml' --depth 1
17:58:24 etcd.go:2088: Etcd: Watch: Path: /_mgmt/exported/
17:58:24 main.go:255: Main: Waiting...
17:58:24 remote.go:256: Remote: Copying graph definition...
17:58:24 remote.go:618: Remote: Tunnelling...
17:58:24 remote.go:630: Remote: Exec...
17:58:24 remote.go:510: Remote: Running: /tmp/mgmt-412078160/remote/mgmt run --hostname '' --no-server --seeds '' --file '/tmp/mgmt-412078160/remote/remote2b.yaml' --depth 1
17:58:24 etcd.go:2088: Etcd: Watch: Path: /_mgmt/exported/
17:58:24 main.go:291: Config: Parse failure
17:58:24 main.go:255: Main: Waiting...
^C17:58:48 main.go:62: Interrupted by ^C
17:58:48 main.go:397: Destroy...
17:58:48 remote.go:532: Remote: Output...
|    17:58:23 main.go:76: This is: mgmt, version: 0.0.5-3-g4b8ad3a
|    17:58:47 main.go:419: Goodbye!
17:58:48 remote.go:636: Remote: Done!
17:58:48 remote.go:532: Remote: Output...
|    17:58:24 main.go:76: This is: mgmt, version: 0.0.5-3-g4b8ad3a
|    17:58:48 main.go:419: Goodbye!
17:58:48 remote.go:636: Remote: Done!
17:58:48 main.go:419: Goodbye!

You should see that we kick off the remote executions, and how they are wired back through the tunnel. In this particular case we terminated the runs with a ^C.

The example configurations I used are available here and here. If you had a terminal open on the first remote machine, after about a second you would have seen:

[root@omv1 ~]# ls -d /tmp/file*  /tmp/mgmt*
/tmp/file1a  /tmp/file2a  /tmp/file2b  /tmp/mgmt-412078160
[root@omv1 ~]# cat /tmp/file*
i am file1a
i am file2a, exported from host a
i am file2b, exported from host b

You can see the remote execution artifacts, and that there was clearly data exchange. You can repeat this example with --converged-timeout=5 to automatically terminate after five seconds of cluster wide inactivity.

Live remote hacking

Since mgmt is event based, and graph structure configurations manifest themselves as event streams, you can actually edit the input configuration on the initiating machine, and as soon as the file is saved, it will instantly remotely propagate and apply the graph differential.

For this particular example, since we export and collect resources through the tunnelled SSH connections, it means editing the exported file, will also cause both hosts to update that file on disk!

You’ll see this occurring with this message in the logs:

18:00:44 remote.go:973: Remote: Copied over new graph definition: examples/remote2b.yaml

While you might not necessarily want to use this functionality on a production machine, it will definitely make your interactive hacking sessions more useful, in particular because you never need to re-run parts of the graph which have already converged!


In case you’re wondering, mgmt can look in your ~/.ssh/ for keys to use for the auth, or it can prompt you interactively. It can also read a plain text password from the connection string, but this isn’t a recommended security practice.

Hierarchial remote execution

Even though we recommend running mgmt in a normal clustered mode instead of over SSH, we didn’t want to limit the number of hosts that can be configured using remote execution. For this reason it would be architecturally simple to add support for what we’ve decided to call “hierarchial remote execution”.

In this mode, the primary initiator would first connect to one or more secondary nodes, which would then stage a second series of remote execution runs resulting in an order of depth equal to two or more. This fan out approach can be used to distribute the number of outgoing connections across more intermediate machines, or as a method to conserve remote execution bandwidth on the primary link into your datacenter, by having the secondary machine run most of the remote execution runs.


This particular extension hasn’t been built, although some of the plumbing has been laid. If you’d like to contribute this feature to the upstream project, please join us in #mgmtconfig on Freenode and let us (I’m @purpleidea) know!


There is some generated documentation for the mgmt remote package available. There is also the beginning of some additional documentation in the markdown docs. You can help contribute to either of these by sending us a patch!

Novel resources

Our event based architecture can enable some previously improbable kinds of resources. In particular, I think it would be quite beautiful if someone built a provisioning resource. The Watch method of the resource API normally serves to notify us of events, but since it is a main loop that blocks in a select call, it could also be used to run a small server that hosts a kickstart file and associated TFTP images. If you like this idea, please help us build it!


I hope you enjoyed this article and found this remote execution methodology as novel as we do. In particular I hope that I’ve demonstrated that configuration software doesn’t have to be constrained behind a static orchestration topology.

Happy Hacking,


One hour hacks: Remote LUKS over SSH

I have a GNU/Linux server which I mount a few LUKS encrypted drives on. I only ever interact with the server over SSH, and I never want to keep the LUKS credentials on the remote server. I don’t have anything especially sensitive on the drives, but I think it’s a good security practice to encrypt it all, if only to add noise into the system and for solidarity with those who harbour much more sensitive data.

This means that every time the server reboots or whenever I want to mount the drives, I have to log in and go through the series of luksOpen and mount commands before I can access the data. This turned out to be a bit laborious, so I wrote a quick script to automate it! I also made sure that it was idempotent.

I decided to share it because I couldn’t find anything similar, and I was annoyed that I had to write this in the first place. Hopefully it saves you some anguish. It also contains a clever little bash hack that I am proud to have in my script.

Here’s the script. You’ll need to fill in the map of mount folder names to drive UUID’s, and you’ll want to set your server hostname and FQDN to match your environment of course. It will prompt you for your root password to mount, and the LUKS password when needed.

Example of mounting:

Running on: myserver...
[sudo] password for james: 
Mount/Unmount [m/u] ? m
music: mkdir ✓
LUKS Password: 
music: luksOpen ✓
music: mount ✓
files: mkdir ✓
files: luksOpen ✓
files: mount ✓
photos: mkdir ✓
photos: luksOpen ✓
photos: mount ✓
Connection to closed.

Example of unmounting:

Running on: myserver...
[sudo] password for james: 
Sorry, try again.
[sudo] password for james: 
Mount/Unmount [m/u] ? u
music: umount ✓
music: luksClose ✓
music: rmdir ✓
files: umount ✓
files: luksClose ✓
files: rmdir ✓
photos: umount ✓
photos: luksClose ✓
photos: rmdir ✓
Connection to closed.

It’s worth mentioning that there are many improvements that could be made to this script. If you’ve got patches, send them my way. After all, this is only a: one hour hack.

Happy hacking,


PS: One day this sort of thing might be possible in mgmt. Let me know if you want to help work on it!

Vagrant clustered SSH and ‘screen’

Some fun updates for vagrant hackers… I wanted to use the venerable clustered SSH (cssh) and screen with vagrant. I decided to expand on my vsftp script. First read:

Vagrant on Fedora with libvirt


Vagrant vsftp and other tricks

to get up to speed on the background information.

Vagrant screen:

First, a simple screen hack… I often use my vssh alias to quickly ssh into a machine, but I don’t want to have to waste time with sudo-ing to root and then running screen each time. Enter vscreen:

# vagrant screen
function vscreen {
	[ "$1" = '' ] || [ "$2" != '' ] && echo "Usage: vscreen <vm-name> - vagrant screen" 1>&2 && return 1
	wd=`pwd`		# save wd, then find the Vagrant project
	while [ "`pwd`" != '/' ] && [ ! -e "`pwd`/Vagrantfile" ] && [ ! -d "`pwd`/.vagrant/" ]; do
		#echo "pwd is `pwd`"
		cd ..
	cd $wd
	if [ ! -e "$pwd/Vagrantfile" ] || [ ! -d "$pwd/.vagrant/" ]; then
		echo 'Vagrant project not found!' 1>&2 && return 2

	# hostname extraction from user@host pattern
	p=`expr index "$1" '@'`
	if [ $p -gt 0 ]; then
		let "l = ${#h} - $p"

	# if mtime of $f is > than 5 minutes (5 * 60 seconds), re-generate...
	if [ `date -d "now - $(stat -c '%Y' "$f" 2> /dev/null) seconds" +%s` -gt 300 ]; then
		mkdir -p "$d"
		# we cache the lookup because this command is slow...
		vagrant ssh-config "$h" > "$f" || rm "$f"
	[ -e "$f" ] && ssh -t -F "$f" "$1" 'screen -xRR'

I usually run it this way:

$ vscreen root@machine

which logs in as root, to machine and gets me (back) into screen. This is almost identical to the vsftp script which I explained in an earlier blog post.

Vagrant cssh:

First you’ll need to install cssh. On my Fedora machine it’s as easy as:

# yum install -y clusterssh

I’ve been hacking a lot on Puppet-Gluster lately, and occasionally multi-machine hacking demands multi-machine key punching. Enter vcssh:

# vagrant cssh
function vcssh {
	[ "$1" = '' ] && echo "Usage: vcssh [options] [user@]<vm1>[ [user@]vm2[ [user@]vmN...]] - vagrant cssh" 1>&2 && return 1
	wd=`pwd`		# save wd, then find the Vagrant project
	while [ "`pwd`" != '/' ] && [ ! -e "`pwd`/Vagrantfile" ] && [ ! -d "`pwd`/.vagrant/" ]; do
		#echo "pwd is `pwd`"
		cd ..
	cd $wd
	if [ ! -e "$pwd/Vagrantfile" ] || [ ! -d "$pwd/.vagrant/" ]; then
		echo 'Vagrant project not found!' 1>&2 && return 2

	cat='cat '

	for i in "$@"; do	# loop through the list of hosts and arguments!
		#echo $i

		if [ "$special" = 'debug' ]; then	# optional arg value...
			if [ "$1" -ge 0 -o "$1" -le 4 ]; then
				cmd="$cmd $i"

		if [ "$multi" = 'y' ]; then	# get the value of the argument
			cmd="$cmd '$i'"

		if [ "${i:0:1}" = '-' ]; then	# does argument start with: - ?

			# build a --screen option
			if [ "$i" = '--screen' ]; then
				screen=' -o RequestTTY=yes'
				cmd="$cmd --action 'screen -xRR'"

			if [ "$i" = '--debug' ]; then
				cmd="$cmd $i"

			if [ "$i" = '--options' ]; then
				options=" $i"

			# NOTE: commented-out options are probably not useful...
			# match for key => value argument pairs
			if [ "$i" = '--action' -o "$i" = '-a' ] || \
			[ "$i" = '--autoclose' -o "$i" = '-A' ] || \
			#[ "$i" = '--cluster-file' -o "$i" = '-c' ] || \
			#[ "$i" = '--config-file' -o "$i" = '-C' ] || \
			#[ "$i" = '--evaluate' -o "$i" = '-e' ] || \
			[ "$i" = '--font' -o "$i" = '-f' ] || \
			#[ "$i" = '--master' -o "$i" = '-M' ] || \
			#[ "$i" = '--port' -o "$i" = '-p' ] || \
			#[ "$i" = '--tag-file' -o "$i" = '-c' ] || \
			[ "$i" = '--term-args' -o "$i" = '-t' ] || \
			[ "$i" = '--title' -o "$i" = '-T' ] || \
			[ "$i" = '--username' -o "$i" = '-l' ] ; then
				multi='y'	# loop around to get second part
				cmd="$cmd $i"
			else			# match single argument flags...
				cmd="$cmd $i"

		# hostname extraction from user@host pattern
		p=`expr index "$i" '@'`
		if [ $p -gt 0 ]; then
			let "l = ${#h} - $p"

		# if mtime of $f is > than 5 minutes (5 * 60 seconds), re-generate...
		if [ `date -d "now - $(stat -c '%Y' "$f" 2> /dev/null) seconds" +%s` -gt 300 ]; then
			mkdir -p "$d"
			# we cache the lookup because this command is slow...
			vagrant ssh-config "$h" > "$f" || rm "$f"

		if [ -e "$f" ]; then
			cmd="$cmd $i"
			cat="$cat $f"	# append config file to list

	cat="$cat > $cssh"
	#echo $cat
	eval "$cat"			# generate combined config file

	#echo $cmd && return 1
	#[ -e "$cssh" ] && cssh --options "-F ${cssh}$options" $cmd
	# running: bash -c glues together --action 'foo --bar' type commands...
	[ -e "$cssh" ] && bash -c "cssh --options '-F ${cssh}${screen}$options' $cmd"

This can be called like this:

$ vcssh annex{1..4} -l root

or like this:

$ vcssh root@hostname foo user@bar james@machine --action 'pwd'

which, as you can see, passes cssh arguments through! Can you see any other special surprises in the code? Well, you can run vcssh like this too:

$ vcssh root@foo james@bar --screen

which will perform exactly as vscreen did above, but in cssh!

You’ll see that the vagrant ssh-config lookups are cached, so this will be speedy when it’s running hot, but expect a few seconds delay when you first run it. If you want a longer cache timeout, it’s easy to change yourself in the function.

I’ve uploaded the code here, so that you don’t have to copy+paste it from my blog!

Happy hacking,


Desktop Notifications for Irssi in Screen through SSH in Gnome Terminal

I’m usually on IRC, but I don’t often notice incoming pings until after the fact. I had to both write, and modify various scripts to get what I wanted, but now it’s all done, and you can benefit from my hacking by following along…

The Setup

Laptop -> Gnome-Terminal -> SSH -> Screen -> Irssi

This way, I’m connected to IRC, even when my laptop isn’t. I run irssi in a screen session on an SSH server that I manage, and I use gnome-terminal on my laptop. If you don’t understand this setup, then you’ll need to get more comfortable with these tools first.


The first trick is getting irssi to store notifications in a uniform way. To do this, I modified an irssi script called fnotify. My changed version is available here. Installation is easy:

# on your ssh server:
cd /tmp; wget
cp /tmp/ ~/.irssi/scripts/
# in irssi:
irssi> /load perl
irssi> /script load fnotify

When someone sends you a direct message, or highlights your nick on IRC, this script will append a line to the ~/.irssi/fnotify file on the SSH server.

Watching fnotify

On your local machine, we need a script to tail the fnotify file. This was surprisingly hard to get right. The fruit of my labour is available here. You’ll want to copy this script to your local ~/bin/ directory. I’ve named this script This script watches the remote fnotify file, and runs notify-send and paplay locally to notify you of any incoming messages, each time one comes in.

SSH Activation

We want the script to run automatically when we connect to our SSH server. To do this, add the following lines to your ~/.ssh/config file:

# home
Host home
    PermitLocalCommand yes
    LocalCommand ~/bin/ --start %r@%h

You might also want to have other directives listed here as well, but that is outside the scope of this article. Now each time you run:

ssh home

The command will automatically run.


I’ve left out some important details:

  • The LocalCommand that you use, must return before ssh will continue. As a result, it daemonizes itself into the background when you invoke it with –start.
  • My program watches the parent ssh $PID. When it exits, it will run a cleanup routine to purge old notifications from the fnotify file. This requires a brief SSH connection back to the server. This is a useful feature!
  • You may wish to modify to paplay a different alert sound, or to avoid making noise entirely. The choice is yours.
  • When runs, it will tail the fnotify file over ssh. If there are “unread” messages, tail will try to “download” up to ten. You can edit this behaviour in if you want a larger initial backlog.
  • The script doesn’t attempt to prevent flooding, nor does it filter weird characters from incoming messages. You may want to add this yourself, and or /kb users who cause you to need these features.

Here’s a little screenshot (with shameless plug) of the result in action: notification screenshot

Here’s an example of how this helps me to be more responsive in channel:

helping out in #gluster

helping out in #gluster

I hope you found this useful.

Happy Hacking,


how to use ssh escape characters

So you’ve learned screen, ssh and vim. Time to take your skills to level two.

Day one: You’ve logged in to your server remotely via ssh. You run “screen -xRR”, and two minutes later you’re busy chatting away in irssi and vim is running in the other window, because, you know, real sysadmins don’t use emacs.

Lunch time: You grab your laptop and head off for lunch. When you open the lid and look at your terminals, they’re all frozen, because the tcp connections have died. You force quit the terminals, and you’re back in 30 seconds with new tcp connections.

Day two: Since lunch is a daily occurence, it would be nice to avoid this altogether. Enter ssh escape characters. Do a: “man ssh” and search for: “ESCAPE CHARACTERS”.

Lunch time: Hit: ~ . (tilde-period) in an ssh session. This will probably require you hold <shift> to get a “tilde”, (then release) and enter a period (you should know how to type!) Instead of period, you can enter a ? in case you want see about the other cool commands. If ever this doesn’t work, press <enter> at least once to “unconfuse” the escape sequence listener and you can now try again.

Day three: You learn about SCTP and decide this is the future for your multihomed life. Bonus points for someone who comments about how they use it.

Happy Hacking!


git, gitosis, gitweb and friends…

In case it wasn’t already obvious, I am a huge fan of git, and often prefer it over sliced bread. Recently to help a small team of programmers collaborate, I decided to setup a private git server for them to use. By no claim of mine is the following tutorial unique, however I am writing this to aid those who had trouble following other online tutorials.

Setup a central git server for private or public source sharing, without having to give everyone a separate shell account.

Step 1:
Install git, gitosis, and gitweb by the method of your choosing. Most distributions probably have packages for all of these.

Step 2:
Create a user account named “git”, “gitosis”, or something sensible if it hasn’t already been done by some packagers install script. The shell command to do this yourself looks something like:

sudo adduser --system
--shell /bin/sh
--gecos 'git version control'
--home /srv/gitosis

In my particular case, I edited the /etc/passwd file to change the automatically added account, however running sudo dpkg-reconfigure gitosis is probably an easier way to do this.

Step 3:
Authentication is done by public ssh key, and gitosis takes care of the magic relationships between a users key and the read/write access per repository. As such, gitosis needs initialization, and it needs the public key of the administrator. If you aren’t familiar with ssh public key authentication, go learn about this now.

To initialize gitosis most tutorials alledge that you should run something like:

sudo -H -u gitosis gitosis-init <

In my case, this didn’t work (likely due to environment variable problems, but try it first anyways) so I cut to the chase and ran:

sudo su - gitosis
gitosis-init < /tmp/

which worked perfectly right away. Note that you have to copy your public key to a publicly readable location like /tmp/ first.

Step 4:
Now it’s time to change the gitosis configuration file to your liking. Instead of editing the file directly on the server, the model employed is quite clever: git-clone a special repository, edit what’s necessary and commit, then push the changes back up. Once this is done, git runs a special commit-hook that generates special files needed for correct operation. The code:

git clone gitosis@SERVER:gitosis-admin.git

Step 5:
To add a new repository, and its users, into the machine, the obvious bits are done by making changes in the recently cloned git directory. Once a user has access, setup the new remote, and push.

git remote add origin gitosis@SERVER:the_repository_name.git
git push origin master:refs/heads/master

Subsequent pushes don’t need the master:refs/heads/master part, and everything else should function as normal.

I still don’t have a happy gitweb+gitosis installation. I’m using apache as a webserver, and I wanted to leave the main /etc/gitweb.conf as unchanged as possible. The gitweb package that I installed, comes with an: /etc/apache2/conf.d/gitweb and all I added was:

SetEnv GITWEB_CONFIG /srv/gitosis/.gitweb.conf

between the <directory> braces. I used the template gitweb configuration file as provided by gitosis.

The last small change that I needed for perfection is to store the gitweb configuration in the gitosis-admin.git directory. To do this, I added a symlink to /srv/gitosis/repositories/gitosis-admin.git/gitweb.conf from the above gitweb config location. The problem is that the file isn’t in the git repo. This would require patching gitosis to recognize it as a special file, and since the author doesn’t respond to patch offers, and since the gitweb config is usually completely static, I didn’t bother taking this any further.

It seems there is a well maintained and more fine grained alternative to gitosis called gitolite. The author was very friendly and responsive, and it seems his software provides finer grained control than gitosis. If I hadn’t already setup gitosis, I would have surely investiaged this first.

If you haven’t yet used this tool, then go have a look. It can be quite useful, and I only have one addition to propose. It should keep a configurable mapping (as an etckeeper config file) with commands to run based on what gets updated. For example, if I update /etc/apache2/httpd.conf, then run /etc/init.d/apache2 reload.

Happy hacking!

piping data through ssh

not that what i’m about to tell you is brilliant, new or revolutionary, however i thought i’d mention it in case you’re not doing it. also feel free to let me know if there is a better way.

problem: i have some stdout which comes from a command and i want it in a file on another machine.

i could first send it to a temp file, scp that over, and then remove the temp file; but instead, i’ll just:

echo here is some stdout | ssh tee filename

which gives me the added bonus of seeing the contents fly by my screen as they get sent through tee.