IPMI for linux professionals

The nostalgia of serial console servers, kvm’s and switched PDU’s is hopefully no longer visible in your server room. If not, you must definitely start playing catch up. Please forgive my ignorance, but these things might still be common for big windows shops, however if that’s the case, you’ve got an entirely different set of problems ;)

IPMI is an IP based protocol that allows you to talk directly to a little computer, usually built in to your server. It lets you remotely manage power (on, off, reboot, cycle…) get a serial console, collect sensor readings like temperatures, and do other magical things too if you care to figure them out.

The web talks a lot about all this. I’ll give you the short “need to know” list to get you going.

1. It probably makes sense to have the IPMI device of your DHCP server (or whatever network dependencies you have) set statically, so that this works if DHCP is down. I’ve actually never heard of anyone who had this problem, but it seems logical enough that I figured I’d mention it.
2. Set an IPMI password and put the device on a separate layer two network behind your router and firewall. Most servers bond the IPMI device to your “eth0” by default (at layer2), or let you split it off to a separate physical interface if so desired. Do the split and plug it into your management network. Remind me to talk about my dual router topology one day.
3. When you use cobbler to kickstart your machines, you’ll need this in your kopts:
   console=ttyS1,115200
   Don’t bother wasting your time configuring that manually when anaconda takes care of this for you :)
4. Almost all server hardware uses the second serial device (ttyS1) as the one that is linked to the IPMI hardware. In some insane default BIOS’es you might have to enable this.
5. Once installed, the kopt will usually know to have added the correct magic to grub, and also to whatever spawn’s your serial tty. Feel free to grep to see what your $OS did.
6. ipmitool -I lanplus -H <ip address of ipmi device> -U ADMIN sol activate
   if ever this gets stuck, run a ‘deactivate’ first.
7. Learn the ~. disconnect sequence. If you’re connected over ssh to your ipmi client (which I always am since it’s my router) you’ll need to “~~” to skip “through” the ssh escape character, and then period “.”, exactly how ssh disconnects. Similarly the same logic applies if you’re insane and run screen -> ssh -> screen.
8. You might need to do a “reset+clear” if the bios throws crap down the wire at you. I haven’t found a way to avoid this. It’s generally not a big problem for me, because this only happens if I’m watching the bios at boot, which only really happens if I’m bored on first install.

Happy Hacking!

James

 

how to use ssh escape characters

So you’ve learned screen, ssh and vim. Time to take your skills to level two.

Day one: You’ve logged in to your server remotely via ssh. You run “screen -xRR”, and two minutes later you’re busy chatting away in irssi and vim is running in the other window, because, you know, real sysadmins don’t use emacs.

Lunch time: You grab your laptop and head off for lunch. When you open the lid and look at your terminals, they’re all frozen, because the tcp connections have died. You force quit the terminals, and you’re back in 30 seconds with new tcp connections.

Day two: Since lunch is a daily occurence, it would be nice to avoid this altogether. Enter ssh escape characters. Do a: “man ssh” and search for: “ESCAPE CHARACTERS”.

Lunch time: Hit: ~ . (tilde-period) in an ssh session. This will probably require you hold <shift> to get a “tilde”, (then release) and enter a period (you should know how to type!) Instead of period, you can enter a ? in case you want see about the other cool commands. If ever this doesn’t work, press <enter> at least once to “unconfuse” the escape sequence listener and you can now try again.

Day three: You learn about SCTP and decide this is the future for your multihomed life. Bonus points for someone who comments about how they use it.

Happy Hacking!

James