my gluster setup, described

For the last ~two or so years I’ve played and tested gluster on and off and hanging out in the awesome #gluster channel on Freenode. In case you haven’t heard, gluster was acquired by RedHat back in October 2011. This post describes my current setup. I urge you to send your comments and suggestions for improvement. I’ll update this as needed.

Hardware:
Ideology: I wanted to build individual self-contained storage hosts. I didn’t want to have servers with separate (serial) attached storage (SAS) like Dell is often pushing. Supermicro fit the design goal, and sold me when I realized I could have the OS drives swappable out the back.

• 4 x Supermicro 6047R-E1R24N
• 4 x 24 x 2TiB, 3.5″ HDD (front, hot swappable main storage)
• 4 x 2 x 600GiB, 2’5″ HDD (rear, hot swappable os drives, awesome feature!)
• 2 x quality stacked switches (with one leg of each bond device out to each switch)
• IPMI: absolutely required (It seems it’s a bit buggy! I’ve had problems where the SOL console stops responding when dealing with a big stream of data, and I can only rescue it with a cold reset of the BMC.) Overall it’s been sufficient to get me up and running.

OS:

• CentOS 6.3+. I would consider using RHEL if their sales department could get organized and when RHEL integrates into my cobbler+puppet build system.
• Bonded (eth0,eth1 -> bond0) ethernet for each machine. Possible upgrade to bonded 10GbE if ever needed. Interface eth0 on each machine plugs into switch0 and eth1 on each machine plugs into switch1.
• The 24 storage HDD’s are split into two separate RAID 6’s per machine.
• OS HDD’s in software raid 1. Unfortunately anaconda/kickstart doesn’t support RAID 1 for the EFI boot partitions. Maybe someone could fix this! (HINT, HINT)
• The machines pxeboot, kickstart and configure themselves automatically with cobbler+puppet.
• The LSI MSM tool (for monitoring the RAID) seems to give me a lot of trouble with false positive warnings about temperature thresholds. Apart from being stuck with proprietary crap ware, it does actually email me when drives fail. Alternatives welcome! I deploy this with a puppet module that I wrote. If it weren’t for that, this step would drive me insane.

Gluster:

• Each host has its drives split into two bricks. A gluster engineer recommended this for the type of setup I’m running.
• Each RAID6 set is formatted with xfs.
• Keepalived maintains a VIP (will replace with cman/corosync one day) which serves as the client hostname to connect to. This makes my setup a bit more highly available if one or more nodes go down.
• I have a puppet module which I use to describe/build my gluster setup. It’s not perfect, but it works for me ™. I’m cleaning it up, and will post it shortly.
• I’m using a distributed-replicate setup, with eight bricks (2 per node).
• I originally used the official packages to get my gluster rpm’s, but recently I switched to using: kkeithle‘s. Thanks for your hard work!

Conclusion:

Let me know what other nitpick details you want to know about and I’ll post them. A lot of things can also be inferred by reading my puppet module.

Happy Hacking,
James

gnome3+others glipper/klipper replacement

So a friend of mine uses kde4 for its klipper feature. Turns out he’s right that it’s an awesome feature! I realized I couldn’t let gnome3 take second place to a clipboard app, so after a bit of searching…

$ sudo yum install gnome-shell-extension-gpaste gpaste xsel

Next hit up: https://extensions.gnome.org/local/ to flip on the extension. I had to first type: ALT-F2, “r” (to restart the gnome shell). Don’t worry your apps won’t die. And then I flipped it on.

Clicking on the new shell icon will let you change your pastes, as well as the gpaste preferences that you use. I like to combine this with xsel, so that I can:

$ echo this will be seen in gpaste | xsel

…and presto!

Happy hacking!

fedora 14 to 17 upgrade

I’ve been reluctantly dreading the switch to gnome 3 + shell until it’s been ironed out a little bit more than gnome 3.0 – finally took the plunge. overall it’s working well. here are some (hopefully) useful notes:

• preupgrade 14->17 in one step doesn’t work. it lets you wait an hour for all the downloads to finish, but once you’ve rebooted, the preupgrade installer tells you it can’t jump this far. fail. reboot into f14, yum clean all, rm -rf /var/cache/yum/preupgrade/* ? and preupgrade to f15. boot into that, and then jumping directly to f17 works great.
• remove default f17 ugly fireworks background. the gnome3 default is pretty.
• gnome-tweak-tool is essential. try all the options until you figure out which ones you want. listing my personal preferences here is far too boring for the internet.
• dash click fix extension was essential for me. also: https://extensions.gnome.org/review/download/1323.shell-extension.zip which does the same thing, but for application launcher entries. this allows gnome-shell to behave properly and launch applications where i want them.
• i also added: “Remove user name” and “Remove a11y” extensions.
• “system-monitor”, is close to what i want, but is not good enough yet. disks access/writes don’t show, and it’s got a super ugly popup window. old gnome-system-monitor applet was PERFECT. Please bring it back or fix this one.
• choose non-ugly background for gdm: # sudo -u gdm dbus-launch gnome-control-center

Hope this helped! Happy hacking,

James

IPMI for linux professionals

The nostalgia of serial console servers, kvm’s and switched PDU’s is hopefully no longer visible in your server room. If not, you must definitely start playing catch up. Please forgive my ignorance, but these things might still be common for big windows shops, however if that’s the case, you’ve got an entirely different set of problems ;)

IPMI is an IP based protocol that allows you to talk directly to a little computer, usually built in to your server. It lets you remotely manage power (on, off, reboot, cycle…) get a serial console, collect sensor readings like temperatures, and do other magical things too if you care to figure them out.

The web talks a lot about all this. I’ll give you the short “need to know” list to get you going.

1. It probably makes sense to have the IPMI device of your DHCP server (or whatever network dependencies you have) set statically, so that this works if DHCP is down. I’ve actually never heard of anyone who had this problem, but it seems logical enough that I figured I’d mention it.
2. Set an IPMI password and put the device on a separate layer two network behind your router and firewall. Most servers bond the IPMI device to your “eth0” by default (at layer2), or let you split it off to a separate physical interface if so desired. Do the split and plug it into your management network. Remind me to talk about my dual router topology one day.
3. When you use cobbler to kickstart your machines, you’ll need this in your kopts:
   console=ttyS1,115200
   Don’t bother wasting your time configuring that manually when anaconda takes care of this for you :)
4. Almost all server hardware uses the second serial device (ttyS1) as the one that is linked to the IPMI hardware. In some insane default BIOS’es you might have to enable this.
5. Once installed, the kopt will usually know to have added the correct magic to grub, and also to whatever spawn’s your serial tty. Feel free to grep to see what your $OS did.
6. ipmitool -I lanplus -H <ip address of ipmi device> -U ADMIN sol activate
   if ever this gets stuck, run a ‘deactivate’ first.
7. Learn the ~. disconnect sequence. If you’re connected over ssh to your ipmi client (which I always am since it’s my router) you’ll need to “~~” to skip “through” the ssh escape character, and then period “.”, exactly how ssh disconnects. Similarly the same logic applies if you’re insane and run screen -> ssh -> screen.
8. You might need to do a “reset+clear” if the bios throws crap down the wire at you. I haven’t found a way to avoid this. It’s generally not a big problem for me, because this only happens if I’m watching the bios at boot, which only really happens if I’m bored on first install.

Happy Hacking!

James

 

how to use ssh escape characters

So you’ve learned screen, ssh and vim. Time to take your skills to level two.

Day one: You’ve logged in to your server remotely via ssh. You run “screen -xRR”, and two minutes later you’re busy chatting away in irssi and vim is running in the other window, because, you know, real sysadmins don’t use emacs.

Lunch time: You grab your laptop and head off for lunch. When you open the lid and look at your terminals, they’re all frozen, because the tcp connections have died. You force quit the terminals, and you’re back in 30 seconds with new tcp connections.

Day two: Since lunch is a daily occurence, it would be nice to avoid this altogether. Enter ssh escape characters. Do a: “man ssh” and search for: “ESCAPE CHARACTERS”.

Lunch time: Hit: ~ . (tilde-period) in an ssh session. This will probably require you hold <shift> to get a “tilde”, (then release) and enter a period (you should know how to type!) Instead of period, you can enter a ? in case you want see about the other cool commands. If ever this doesn’t work, press <enter> at least once to “unconfuse” the escape sequence listener and you can now try again.

Day three: You learn about SCTP and decide this is the future for your multihomed life. Bonus points for someone who comments about how they use it.

Happy Hacking!

James

Dear internets

Dear internets,

During my daily grind of computer architecturing, I often learn something new that I wish a mentor had taught me earlier. I will post my babbles here, so that the interested might learn, and for the wiser to comment on.

Happy Hacking!

James

PS: I had a previous blog, but that server was taken down. It has been merged into this one!

sparse rsync magic

Dear internets,

Once upon a time, I had one of those “brain teaser-of-the-day” calendars. Free business tip: someone should make an ‘rsync’ version. That would be brilliant. (If you don’t know what rsync is, then you should probably go read a different blog.)

Anyone who’s used kvm might know, linux virtual machines can have sparse .raw file backing stores. This means the operating system could think it has one TiB of space, when it is actually sharing a one TiB physical harddrive with many others.

Tip #1: man ls
# ls -lAhs
29G -rwxr-xr-x. 1 qemu qemu 1000G Feb 22 18:39 cobbler.raw

You’ll notice that the ‘s’ flag shows you how much space the file is actually taking up on your harddrive. This gives me lots of room to grow my cobbler virtual machine, without having to actually resize partitions, lv’s or filesystems.

Warning: Make sure to monitor your individual virtual machines, because if the .raw files grow bigger than the size of your host filesystem then they’ll freeze.

Tip #2: use rsync like a boss
This all started out about rsync, remember ? As anyone who’s tried might know, running an rsync on your images/ directory as the source will cause your destination to swell to the size of the virtual hard drives! Solution? add in -S and enjoy the rsync magic.

Happy hacking!

James