Send/Recv in mgmt

I previously published “A revisionist history of configuration management“. I meant for that to be the intro to this article, but it ended up being long enough that it deserved a separate post. I will explain Send/Recv in this article, but first a few clarifications to the aforementioned article.


I mentioned that my “revisionist history” was inaccurate, but I failed to mention that it was also not exhaustive! Many things were left out either because they were proprietary, niche, not well-known, of obscure design or simply for brevity. My apologies if you were involved with Bcfg2, Bosh, Heat, Military specifications, SaltStack, SmartFrog, or something else entirely. I’d love it if someone else wrote an “exhaustive history”, but I don’t think that’s possible.

It’s also worth re-iterating that without the large variety of software and designs which came before me, I wouldn’t have learned or have been able to build anything of value. Thank you giants!  By discussing the problems and designs of other tools, then it makes it easier to contrast with and explaining what I’m doing in mgmt.


If you’re not familiar with the directed acyclic graph model for configuration management, you should start by reviewing that material first. It models a system of resources (workers) as the vertices in that DAG, and the edges as the dependencies. We’re going to add some additional mechanics to this model.

There is a concept in mgmt called notifications. Any time the state of a resource is successfully changed by the engine, a notification is emitted. These notifications are emitted along any graph edge (dependency) that has been asked to relay them. Edges with the Notify property will do so. These are usually called refresh notifications.

Any time a resource receives a refresh notification, it can apply a special action which is resource specific. The svc resource reloads the service, the password resource generates a new password, the timer resource resets the timer, and the noop resource prints a notification message. In general, refresh notifications should be avoided if possible, but there are a number of legitimate use cases.

In mgmt notifications are designed to be crash-proof, that is to say, undelivered notifications are re-queued when the engine restarts. While we don’t expect mgmt to crash, this is also useful when a graph is stopped by the user before it has completed.

You’ll see these notifications in action momentarily.


I mentioned in the revisionist history that I felt that Chef opted for raw code as a solution to the lack of power in Puppet. Having resources in mgmt which are event-driven is one example of increasing their power. Send/Recv is another mechanism to make the resource primitive more powerful.

Simply put: Send/Recv is a mechanism where resources can transfer data along graph edges.

The status quo

Consider the following pattern (expressed as Puppet code):

# create a directory
file { '/var/foo/':
    ensure => directory,
# download a file into that directory
exec { 'wget -O - > /var/foo/data':
    creates => '/var/foo/data',
    require => File['/var/foo/'],
# set some property of the file
file { '/var/foo/data':
    mode => 0644,
    require => File['/var/foo/data'],

First a disclaimer. Puppet now actually supports an http url as a source. Nevertheless, this was a common pattern for many years and that solution only improves a narrow use case. Here are some of the past and current problems:

  • File can’t take output from an exec (or other) resource
  • File can’t pull from an unsupported protocol (sftp, tftp, imap, etc…)
  • File can get created with zero-length data
  • Exec won’t update if http endpoint changes the data
  • Requires knowledge of bash and shell glue
  • Potentially error-prone if a typo is made

There’s also a layering violation if you believe that network code (http downloading) shouldn’t be in a file resource. I think it adds unnecessary complexity to the file resource.

The solution

What the file resource actually needs, is to be able to accept (Recv) data of the same type as any of its input arguments. We also need resources which can produce (Send) data that is useful to consumers. This occurs along a graph (dependency) edge, since the sending resource would need to produce it before the receiver could act!

This also opens up a range of possibilities for new resource kinds that are clever about sending or receiving data. an http resource could contain all the necessary network code, and replace our use of the exec { 'wget ...': } pattern.


in this graph, a password resource generates a random string and stores it in a file

in this graph, a password resource generates a random string and stores it in a file; more clever linkages are planned


As a proof of concept for the idea, I implemented a Password resource. This is a prototype resource that generates a random string of characters. To use the output, it has to be linked via Send/Recv to something that can accept a string. The file resource is one such possibility. Here’s an excerpt of some example output from a simple graph:

03:06:13 password.go:295: Password[password1]: Generating new password...
03:06:13 password.go:312: Password[password1]: Writing password token...
03:06:13 sendrecv.go:184: SendRecv: Password[password1].Password -> File[file1].Content
03:06:13 file.go:651: contentCheckApply: Invalidating sha256sum of `Content`
03:06:13 file.go:579: File[file1]: contentCheckApply(true)
03:06:13 noop.go:115: Noop[noop1]: Received a notification!

What you can see is that initially, a random password is generated. Next Send/Recv transfers the generated Password to the file’s Content. The file resource invalidates the cached Content checksum (a performance feature of the file resource), and then stores that value in the file. (This would normally be a security problem, but this is for example purposes!) Lastly, the file sends out a refresh notification to a Noop resource for demonstration purposes. It responds by printing a log message to that effect.


Ultimately, mgmt will have a DSL to express the different graphs of configuration. In the meantime, you can use Puppet code, or a raw YAML file. The latter is primarily meant for testing purposes until we have the language built.

Lastly, you can also embed mgmt and use it like a library! This lets you write raw golang code to build your resource graphs. I decided to write the above example that way! Have a look at the code! This can be used to embed mgmt into your existing software! There are a few more examples available here.

Resource internals

When a resource receives new values via Send/Recv, it’s likely that the resource will have work to do. As a result, the engine will automatically mark the resource state as dirty and then poke it from the sending node. When the receiver resource runs, it can lookup the list of keys that have been sent. This is useful if it wants to perform a cache invalidation for example. In the resource, the code is quite simple:

if val, exists := obj.Recv["Content"]; exists && val.Changed {
    // the "Content" input has changed

Here is a good example of that mechanism in action.

Future work

This is only powerful if there are interesting resources to link together. Please contribute some ideas, and help build these resources! I’ve got a number of ideas already, but I’d love to hear yours first so that I don’t influence or stifle your creativity. Leave me a message in the comments below!

Happy Hacking,


Securely managing secrets for FreeIPA with Puppet

Configuration management is an essential part of securing your infrastructure because it can make sure that it is set up correctly. It is essential that configuration management only enhance security, and not weaken it. Unfortunately, the status-quo of secret management in puppet is pretty poor.

In the worst (and most common) case, plain text passwords are found in manifests. If the module author tried harder, sometimes these password strings are pre-hashed (and sometimes salted) and fed directly into the consumer. (This isn’t always possible without modifying the software you’re managing.)

On better days, these strings are kept separate from the code in unencrypted yaml files, and if the admin is smart enough to store their configurations in git, they hopefully separated out the secrets into a separate repository. Of course none of these solutions are very convincing to someone who puts security at the forefront.

This article describes how I use puppet to correctly and securely setup FreeIPA.


FreeIPA is an excellent piece of software that combines LDAP and Kerberos with an elegant web ui and command line interface. It can also glue in additional features like NTP. It is essential for any infrastructure that wants single sign on, and unified identity management and security. It is a key piece of infrastructure since you can use it as a cornerstone, and build out your infrastructures from that centrepiece. (I hope to make the puppet-ipa module at least half as good as what the authors have done with FreeIPA core.)


Passing a secret into the FreeIPA server for installation is simply not possible without it touching puppet. The way I work around this limitation is by generating the dm_password on the FreeIPA server at install time! This typically looks like:

/usr/sbin/ipa-server-install --hostname='' --domain='' --realm='EXAMPLE.COM' --ds-password=`/usr/bin/pwgen 16 1 | /usr/bin/tee >( /usr/bin/gpg --homedir '/var/lib/puppet/tmp/ipa/gpg/' --encrypt --trust-model always --recipient '24090D66' > '/var/lib/puppet/tmp/ipa/gpg/dm_password.gpg' ) | /bin/cat | /bin/cat` --admin-password=`/usr/bin/pwgen 16 1 | /usr/bin/tee >( /usr/bin/gpg --homedir '/var/lib/puppet/tmp/ipa/gpg/' --encrypt --trust-model always --recipient '24090D66' > '/var/lib/puppet/tmp/ipa/gpg/admin_password.gpg' ) | /bin/cat | /bin/cat` --idstart=16777216 --no-ntp --selfsign --unattended

This command is approximately what puppet generates. The interesting part is:

--ds-password=`/usr/bin/pwgen 16 1 | /usr/bin/tee >( /usr/bin/gpg --homedir '/var/lib/puppet/tmp/ipa/gpg/' --encrypt --trust-model always --recipient '24090D66' > '/var/lib/puppet/tmp/ipa/gpg/dm_password.gpg' ) | /bin/cat | /bin/cat`

If this is hard to follow, here is the synopsis:

  1. The pwgen command is used generate a password.
  2. The password is used for installation.
  3. The password is encrypted with the users GPG key and saved to a file for retrieval.
  4. The encrypted password is (optionally) sent out via email to the admin.

Note that the email portion wasn’t shown since it makes the command longer.

Where did my GPG key come from?

Any respectable FreeIPA admin should already have their own GPG key. If they don’t, they probably shouldn’t be managing a security appliance. You can either pass the public key to gpg_publickey or specify a keyserver with gpg_keyserver. In either case you must supply a valid recipient (-r) string to gpg_recipient. In my case, I use my keyid of 24090D66, which can be used to find my key on the public keyservers. In either case, puppet knows how to import it and use it correctly. A security audit is welcome!

You’ll be pleased to know that I deliberately included the options to use your own keyserver, or to specify your public key manually if you don’t want it stored on any key servers.

But, I want a different password!

It’s recommended that you use the secure password that has been generated for you. There are a few options if you don’t like this approach:

  • The puppet module allows you to specify the password as a string. This isn’t recommended, but it is useful for testing and compatibility with legacy puppet environments that don’t care about security.
  • You can use the secure password initially to authenticate with your FreeIPA server, and then change the password to the one you desire. Doing this is outside the scope of this article, and you should consult the FreeIPA documentation.
  • You can use puppet to regenerate a new password for you. This hasn’t been implemented yet, but will be coming eventually.
  • You can use the interactive password helper. This takes the place of the pwgen command. This will be implemented if there is enough demand. During installation, the admin will be able to connect to a secure console to specify the password.

Other suggestions will be considered.

What about the admin password?

The admin_password is generated following the same process that was used for the dm_password. The chance that the two passwords match is probably about:

1/((((26*2)+10)^16)^2) = ~4.4e-58

In other words, very unlikely.

Testing this easily:

Testing this out is quite straightforward. This process has been integrated with vagrant for easy testing. Start by setting up vagrant if you haven’t already:

Vagrant on Fedora with libvirt (reprise)

Once you are comfortable with vagrant, follow these steps for using Puppet-IPA:

git clone --recursive
cd vagrant/
vagrant status
# edit the puppet-ipa.yaml file to add your keyid in the recipient field
# if you do not add a keyid, then a password of 'password' will be used
# this default is only used in the vagrant development environment
vagrant up puppet
vagrant up ipa

You should now have a working FreeIPA server. Login as root with:

vscreen root@ipa


Hope you enjoyed this.

Happy hacking,