New puppet-gluster features before Linuxcon

Hey there,

I’ve done a bit of puppet-gluster hacking lately to try to squeeze some extra features and testing in before Linuxcon. Here’s a short list:

• SELinux fixes to keep Dan Walsh happy :)
• Ping and status checks before volume creation. Now puppet-gluster will be less noisy about failures or missing executions that are due to the necessary incremental nature of puppet-gluster runs. You’ll need multiple puppet runs to get a complete setup, so don’t let puppet complain part way through. This is due to the design of puppet.
• Simple filesystem paths for bricks. This is very useful for prototyping with virtual machines.
• Automatic UUID assignment for hosts. This requires exported resources.
• Updated examples
• And more…

If there are features or bugs that you’d like to see added or removed (respectively) please let me know ASAP so that I can try to get something ready for you before my Linuxcon talk. I also don’t have any hardware RAID or physical hardware to test external storage partitions (bricks) on. If you have any that you can donate or let me hack on for a while, there are some features I’d like to test. Contact me!

I’ve got a few more things queued up too, but you’ll have to wait and see. Until then,

Happy hacking,

James

Puppet-Gluster and me at Linuxcon

John Mark Walker, (from Redhat) has been kind enough to invite me to speak at the Linuxcon Gluster Workshop in New Orleans. I’ll be speaking about puppet-gluster, giving demos, and hopefully showing off some new features. I’m also looking forward to meeting up with gluster expert Joe Julian.

If there are features that puppet-gluster is missing, or you have a use case that I haven’t covered, please let me know, and I’ll try to work on it for you ahead of the conference. If you want to meet up for some puppet-gluster help, or to hack on code, I’ll be around from the 16th to the 20th of September. My talk is on the 19th.

Special thanks to John Mark and Redhat for sponsoring this trip. Without them, none of this would be possible.

Happy hacking,

James

Fresh releases! puppet-ipa, puppet-nfs, puppet-gluster

I’ve been a little slow in making release announcements, so here’s some news:

I’ve just released the third stage of my puppet-ipa module. At the moment it now supports installation, managing of hosts, and managing of services. It integrates with my puppet-nfs module to allow you to easily setup and run an NFSv4 kerberized server and client.

While we’re at it, that’s some more news: I’ve just released a puppet-nfs module to make your /etc/exports management easier. It’s designed to manage other security types, or even to work without kerberos or any authentication at all, but I haven’t tested those.

Back to puppet-ipa for a moment. I’d like you to know that I went to great lengths to make this a very versatile module. Some users probably want certain resources managed by puppet, and others not. With the included features, you can even specify exclusion criteria so that a certain pattern of hosts aren’t touched by puppet. This is useful if you’re slowly converting your ipa setup to be managed by puppet.

You can use $watch and $modify, two special parameters that I added to precisely control what kind of changes you want to allow puppet to make. These are kind of complicated to explain, but suffice it to say that this module should handle whatever situation you’re in.

For the security minded folks, puppet-ipa, never transfers or touches a keytab file. It will securely and automatically provision your hosts and services without storing secret information in puppet. The module isn’t finished, but it’s built right.

Gluster users might find this particular trio useful for offering gluster backed, kerberized, NFS exports. Here’s an example that I made just for you.

Since you sound like you’re having fun deploying servers like crazy, it’s probably useful to have a puppet-cobbler module. I’ve released this module because it’s useful to me, however it really isn’t release ready, but I think it’s better than some (most?) of the other puppet-cobbler code that’s out there. One other warning is that I have a large rearchitecturing planned for this module, so don’t get too attached. It’s going to get better!

So that’s your lot for today, have fun, and

Happy Hacking!

James

PS: If you’re in a giving mood, I’m in the need for some x86_64 compatible test hardware. If you’re able to donate, please let me know!

puppet lsi hardware raid module

In response to some discussion in the gluster community, I am releasing my puppet-lsi module. It’s quite simple, but it is very useful for rebuilding machines. It could do a lot more, but I wanted to depend on the proprietary LSI tools as little as possible. Running megacli with puppet would be a very doable hack, but I’m not sure enough devops out there who would use that feature.

Usage is straightforward if you like the sensible defaults:

include lsi::msm

The general idea is that you’ve probably already setup all your “virtual drive” RAID configurations initially, and now you’re deploying your setup using cobbler and puppet-gluster. This puppet-lsi module should install all the client side LSI tools, and make sure monitoring for the hardware RAID is working. Megacli and all the (evil?) vivaldi framework stuff will be up and running after puppet has run.

I haven’t tested this on a wide array of hardware, and there might even be newer LSI gear on the market. Please don’t test it on production servers. If you want help with this, you might have to sponsor some hardware, or send me somewhere where I can hack on some, because I don’t have a gluster test rig at the moment.

I am curious to hear what kind of RAID you’re using with gluster. Hardware? Software? Details rock. SGPIO with mdadm, and you’re my hero. I want to hear about that!

https://github.com/purpleidea/puppet-lsi/

I hope this was useful to you, and in the meantime,

Happy hacking,

James

PS: The most useful feature of this module, is that it sets up monitoring of your RAID, and lets you access the management daemon through the now installed LSI services.

Knowing when to release and deploy your code (…and a mini script)

Knowing when to release and deploy your code can turn into a complicated discussion. In general, In general, I tend to support releasing early and often, for some value of $early and $often. I’ve decided to keep this simple and introduce you to one metric that I use…

I think that I am fairly diligent in adding plenty of comments to my source code. I might even sometimes add too many. I create plenty of XXX, FIXME, or TODO tagged comments as reminders of things to work on.

To me, XXX represents an important problem that should get looked at or fixed; FIXME, reminds me that I should definitely look into something, and finally, TODO gives me homework or things to pursue when I’m in need of a new project.

I try to resolve most if not all XXX tagged comments before making a 0.1 release, FIXME’s to consider something very stable, and a lack of TODO’s mean something is completely done for now.

To count all these, I wrote a little tool that greps through the top-level directories in my ~/code/ folder, and displays the results in a table. Feel free to give it a try, and use it for your own projects.

While I don’t see this as a particularly game changing utility, it scratches my itch, and helps me keep up my bash skills. The code is available here. Let me know if you have any improvements, or if the source isn’t enough documentation for you.

Happy hacking,

James

continuous display of log files (better tail -f)

All good sysadmins know about using tail -f to follow a log file. I use this all the time to follow /var/log/messages and my gluster logs in particular. Maybe everyone already knows this, but it deserves a PSA: after a certain amount of time (~days) it seems that new messages don’t appear!

What happens by default is that tail -f follows the file descriptor, not the file name, so when your log files get rotated, the file descriptor still points to the (now renamed) file which no longer gets updates any more.

The solution is to get tail to follow the file name you’re interested in:

tail --follow=name /var/log/messages

EDIT: Fortunately there is a shorter way of running this too, you can use:

tail -F

on any up to date version of tail to get the same result. This adds in –retry to the –folllow=name argument.

Happy hacking!

James

 

How to avoid cluster race conditions or: How to implement a distributed lock manager in puppet

I’ve been working on a puppet module for gluster. Both this, my puppet-gfs2 module, and other puppet clustering modules all share a common problem: How does one make sure that only certain operations happen on one node at a time?

The inelegant solutions are simple:

1. Specify manually (in puppet) which node the “master” is, and have it carry out all the special operations. Downside: Single point of failure for your distributed cluster, and you’ve also written ugly asymmetrical code. Build a beautiful, decentralized setup instead.
2. Run all your operations on all nodes. Ensure they’re idempotent, and that they check the cluster state for success first. Downside: Hope that they don’t run simultaneously and race somehow. This is actually how I engineered my first version of puppet-gluster! It was low risk, and I just wanted to get the cluster up; my module was just a tool, not a product.
3. Use the built-in puppet DLM to coordinate running of these tasks. Downside: Oh wait, puppet can’t do this, you misunderstand the puppet architecture. I too thought this was possible for a short duration. Woops! There is no DLM.

Note: I know this is ironic, since by default puppet requires a master node for coordination, however you can have multiple masters, and if they’re down, puppetd still runs on the clients, it just doesn’t receive new information. (You could also reconfigure your manifests to work around these downsides as they arise, but this takes the point out of puppet: keeping it automatic.)

Mostly elegant: Thoughts of my other cluster crept into my head. Out of nowhere, I realized the solution was: VRRP! You may use a different mechanism if you like, but at the moment I’m using keepalived. Keepalived runs on my gluster pool to provide a VIP for the cluster. This allows my clients to use a highly available IP address to download volume files (mount operation) otherwise, if that particular server were down, they wouldn’t be about to mount. The trick: I tell each node what the expected VIP for the cluster is, and if that IP is present in a facter $ipaddress_, then I let that node execute!

The code is now available, please have a look, and let me know what you think.

Happy hacking,
James

PS: Inelegant modes 1 and 2 are still available. For mode 1, set “vip” in your config to the master node IP address you’d like to use. For mode 2, leave the “vip” value at the empty string default.

PPS: I haven’t had a change to thoroughly test this, so be warned of any dust bunnies you might find.