puppet gluster module now in git

The thoughtful bodepd has been kind enough to help me get my puppet-gluster module off the ground and publicized a bit too. My first few commits have been all clean up to get my initial hacking up to snuff with the puppet style guidelines. Sadly, I love indenting my code with tabs, and this is against the puppet rules :(

http://www.emacswiki.org/emacs/TabsSpacesBoth

I’ll be accepting patches by email, but I’d prefer discussion first, especially since I’ve got a few obvious things brewing in my mental queue that should hit master shortly.

Are you a gluster expert who’s weak at puppet? I’m keen to implement many of the common raid, file system and gluster performance optimization’s directly into the module, so that the out of box experience for new users is a fast, streamlined, experience.

Are you a puppet expert who knows a bit of gluster? I’m not sure what the best way to handle large config changes, such as expanding volumes, or replacing bricks is. I can imagine a large state diagram that would be very hard to wholly implement in puppet. So for now, I’m missing a few edge cases, but hopefully this module will be able to solve more of them over time.

I’ve included an examples/ directory in the repository, to give you an idea of how this works for now. Stay tuned for more commits!

git clone https://github.com/purpleidea/puppet-gluster.git

Happy hacking,
James

a puppet module for gluster

I am an avid cobbler+puppet user. This allows me to rely on my cobbler server and puppet manifests to describe how servers/workstations are setup. I only backup my configs and data, and I regenerate failed machines PRN.

I’ll be publishing my larger cobbler+puppet infrastructure in the future once it’s been cleaned up a bit, but for now I figured I’d post my work-in-progress “puppet-gluster” module, since it seems there’s a real interest.

Warning: there are some serious issues with this module! I’ve used this as an easy way to build out many servers with cobbler+puppet automatically. It’s not necessarily the best long-term solution, and it certainly can’t handle certain scenarios yet, but it is a stepping stone if someone would like to think about writing such a module this way.

For lack of better hosting, it’s now available here: https://dl.dropbox.com/u/48553683/puppet-gluster.tar.bz2 Once I finish cleaning up a bit of cruft, I’ll post my git tree somewhere sane. All of this code is AGPL3+ so share and enjoy!

What’s next? My goal is to find the interested parties and provoke a bit of discussion as to whether this is useful and where to go next. It makes sense to me, that the gluster experts chirp in and add gluster specific optimization’s into this module, so that it’s used as a sort of de-facto documentation on how to set up gluster properly.

I believe that Dan Bode and other gluster developers are already interested in the “puppet module” project, and that work is underway. I spoke to him briefly about collaborating. He is most likely a more experienced puppet user than I, and so I look forward to the community getting a useful puppet-gluster module from somewhere. Maybe even native gluster types?

Happy hacking,
James

 

now syndicated on “planet gluster”

Many thanks to johnmark in #gluster for syndicating my “gluster” tagged blog posts on http://www.gluster.org/blog/

I aim to keep these posts technical and informative, aimed mostly at other sysadmins and gluster users. Please don’t  be shy to comment on my writing style or to let me know if you need more information about a particular subject. If you have any ideas about things you’d like me to write about, let me know and I’ll try to do my best. I like feedback!

Happy hacking,
James

PS: My main blog (https://ttboj.wordpress.com/) will still contain other technical articles not relating to gluster, should that be useful to you.

building intel nic driver (igb) for gluster on centos

I’ve been having some strange networking issues with gluster. “Eco__” from #gluster suggested I try an up to date Intel nic driver. Here are the steps I followed to make that happen. No news yet on if that solved the problem.

Currently my system is using the igb (intel gigabit) driver. To find out what version you are running:

# modinfo -F version igb
3.2.10-k

I found a newer version from the supermicro ftp site. A download and a decompress later, I found an: igb-3.4.7.tar.gz file hiding out. Thanks to the kind people at Intel, this was fairly easy to compile and install. First install some deps:

# yum install /usr/bin/{rpmbuild,gcc,make} kernel-devel

Use rpmbuild to make yourself an rpm:

# rpmbuild -tb igb-3.4.7.tar.gz
[snip]

Your rpm package should appear in rpmbuild/RPMS/
In my case, I added this to my local cobbler repo, and pushed it to all my gluster nodes. You might prefer a simple:

 # yum install igb-3.4.7-1.x86_64.rpm

Please note that I believe it’s important to build this module on the same kind of OS/Hardware that you’re using it for. Since my storage hosts are all identical, this wasn’t a problem.

Happy hacking!
James

EDIT: tru_tru from #gluster pointed out that this module actually exists in elrepo, the wonderful people who also provide drbd modules. I haven’t tested it, but I’m sure it’s excellent.

my gluster setup, described

For the last ~two or so years I’ve played and tested gluster on and off and hanging out in the awesome #gluster channel on Freenode. In case you haven’t heard, gluster was acquired by RedHat back in October 2011. This post describes my current setup. I urge you to send your comments and suggestions for improvement. I’ll update this as needed.

Hardware:
Ideology: I wanted to build individual self-contained storage hosts. I didn’t want to have servers with separate (serial) attached storage (SAS) like Dell is often pushing. Supermicro fit the design goal, and sold me when I realized I could have the OS drives swappable out the back.

• 4 x Supermicro 6047R-E1R24N
• 4 x 24 x 2TiB, 3.5″ HDD (front, hot swappable main storage)
• 4 x 2 x 600GiB, 2’5″ HDD (rear, hot swappable os drives, awesome feature!)
• 2 x quality stacked switches (with one leg of each bond device out to each switch)
• IPMI: absolutely required (It seems it’s a bit buggy! I’ve had problems where the SOL console stops responding when dealing with a big stream of data, and I can only rescue it with a cold reset of the BMC.) Overall it’s been sufficient to get me up and running.

OS:

• CentOS 6.3+. I would consider using RHEL if their sales department could get organized and when RHEL integrates into my cobbler+puppet build system.
• Bonded (eth0,eth1 -> bond0) ethernet for each machine. Possible upgrade to bonded 10GbE if ever needed. Interface eth0 on each machine plugs into switch0 and eth1 on each machine plugs into switch1.
• The 24 storage HDD’s are split into two separate RAID 6’s per machine.
• OS HDD’s in software raid 1. Unfortunately anaconda/kickstart doesn’t support RAID 1 for the EFI boot partitions. Maybe someone could fix this! (HINT, HINT)
• The machines pxeboot, kickstart and configure themselves automatically with cobbler+puppet.
• The LSI MSM tool (for monitoring the RAID) seems to give me a lot of trouble with false positive warnings about temperature thresholds. Apart from being stuck with proprietary crap ware, it does actually email me when drives fail. Alternatives welcome! I deploy this with a puppet module that I wrote. If it weren’t for that, this step would drive me insane.

Gluster:

• Each host has its drives split into two bricks. A gluster engineer recommended this for the type of setup I’m running.
• Each RAID6 set is formatted with xfs.
• Keepalived maintains a VIP (will replace with cman/corosync one day) which serves as the client hostname to connect to. This makes my setup a bit more highly available if one or more nodes go down.
• I have a puppet module which I use to describe/build my gluster setup. It’s not perfect, but it works for me ™. I’m cleaning it up, and will post it shortly.
• I’m using a distributed-replicate setup, with eight bricks (2 per node).
• I originally used the official packages to get my gluster rpm’s, but recently I switched to using: kkeithle‘s. Thanks for your hard work!

Conclusion:

Let me know what other nitpick details you want to know about and I’ll post them. A lot of things can also be inferred by reading my puppet module.

Happy Hacking,
James

gnome3+others glipper/klipper replacement

So a friend of mine uses kde4 for its klipper feature. Turns out he’s right that it’s an awesome feature! I realized I couldn’t let gnome3 take second place to a clipboard app, so after a bit of searching…

$ sudo yum install gnome-shell-extension-gpaste gpaste xsel

Next hit up: https://extensions.gnome.org/local/ to flip on the extension. I had to first type: ALT-F2, “r” (to restart the gnome shell). Don’t worry your apps won’t die. And then I flipped it on.

Clicking on the new shell icon will let you change your pastes, as well as the gpaste preferences that you use. I like to combine this with xsel, so that I can:

$ echo this will be seen in gpaste | xsel

…and presto!

Happy hacking!