SElinux causes pain when using puppet 2.x with hiera

So hiera wasn’t working when used through my puppetmaster. It worked perfectly when I was running my scripts manually with: puppet apply site.pp but the moment I switched over to regular puppetmasterd usage, everything went dead.

I realized a while back, that I could always expect some hiera failures from time to time. Whether this is hiera’s fault or not is irrelevant, the relevant part is that I quickly added:

$test = hiera('test', '')    # expects: 'Hiera is working!'
if "${test}" == '' {
    fail('The hiera function is not working as expected!')
}

to my site.pp and now I don’t risk breaking/changing a node because some data is missing. Naturally you’ll need to add a little message in your globals.yaml or similar. If you’re really cautious, you could change the above to an inequality and have your code “expect” a magic message.

To continue with the story, blkperl from #puppet recommended that I try running puppetmasterd manually to watch for any messages it might print. To my surprise, things started working! This just shows you how helpful it is to have a second set of eyes to help you on your way. So why did this work ?

To make a long story short, after following a few dead ends, it hit me, and so I looked in /var/log/audit/audit.log:

type=AVC msg=audit(1358404083.789:55416): avc:  denied  { getattr } for  pid=13830 comm="puppetmasterd" path="/etc/puppet/hiera.yaml" dev=dm-0 ino=18613624 scontext=unconfined_u:system_r:puppetmaster_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file

to find that selinux was once again causing me pain. When I had started puppetmasterd manually, I was root, which allowed me to bypass selinux rules. Sadly, I like selinux, but since I’m not nearly clever enough to want to learn how to fix this the right way, it just got disabled on another one of my machines.

Running:

restorecon -v /etc/puppet/hiera.yaml

fixed the bad selinux context I had on that file.

Hope this saves you some time and,

Happy hacking,

James

Dynamically including classes in puppet

As you might already know, I like pushing the boundaries of what puppet is able to do. Today, I realized that I needed to include a class by variable name. A simple way to do this is possible with:

$foo = 'world'
class { "hello::${foo}::params":
}

I then realized that you could also do this with the standard include keyword:

$foo = 'world'
include "hello::${foo}::params"

If you’re really insane enough to need to have your include depend on a variable, then this should suit your needs. The advantage of the second form is that if that statement appears more than once, you won’t cause a “duplicate definition” error.

Recently, I’ve been playing around with hiera. For completeness, I will mention one more form. It turns out that the hiera puppet functions offer a hiera_include function. As with all the hiera functions, the second argument lets you specify a default:

$foo = 'world'
hiera_include('...', "hello::${foo}::params")

which finishes off the trilogy. Hope this was useful, now start getting creative and

Happy hacking,

James

 

Clustering virtual machines with rgmanager and clusvcadm

This could be a post detailing how to host clustered virtual machines with rgmanager and clusvcadm, but that is a longer story and there is much work to do. For now, I will give you a short version including an informative “gotcha”.

With my cluster up and running, I added a virtual machine entry to my cluster.conf:

<vm name="test1" domain="somedomain" path="/shared/vm/" autostart="0" exclusive="0" recovery="restart" use_virsh="1" />

This goes inside the <rm> block. As a benchmark, please note that starting the machine with virsh worked perfectly:

[root@server1 ~]# virsh create /shared/vm/test1.xml --console
(...The operation worked perfectly!)

However, when I attempted to use the cluster aware tools, all I got was failure:

[root@server1 ~]# clusvcadm -e 'vm:test1' -m server1
Member server1 trying to enable vm:test1...Failure

Whenever I think I’ve done everything right, but something is still not working, I first check to see if I can blame someone else. Usually that someone is selinux. Make no mistake, selinux is a good thing^™, however it does still cause me pain.

The first clue is to remember that /var/log/ contains other files besides “messages“. Running a tail on /var/log/audit/audit.log while simultaneously running the above clusvcadm command revealed:

type=AVC msg=audit(1357202069.310:10904): avc:  denied  { read } for  pid=15675 comm="virsh" name="test1.xml" dev=drbd0 ino=198628 scontext=unconfined_u:system_r:xm_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=file
type=SYSCALL msg=audit(1357202069.310:10904): arch=c000003e syscall=2 success=no exit=-13 a0=24259e0 a1=0 a2=7ffff03af0d0 a3=7ffff03aee10 items=0 ppid=15609 pid=15675 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="virsh" exe="/usr/bin/virsh" subj=unconfined_u:system_r:xm_t:s0 key=(null)

I am not a magician, but if I was, I would probably understand what all of that means. For now, let’s pretend that we do. Closer inspection (or grep) will reveal:

• “test1.xml” (the definition for the virtual machine)

and:

• “/usr/bin/virsh” (the command that I expect rgmanager’s /usr/share/cluster/vm.sh script to run)

A quick:

[root@server1 ~]# selinuxenabled && echo t || echo f
t

to confirm that selinux is auditing away, and a short:

[root@server1 ~]# /bin/echo 0 > /selinux/enforce

to temporarily test my theory, and:

[root@server1 ~]# clusvcadm -e 'vm:test1' -m server1
Member server1 trying to enable vm:test1...Success
vm:test1 is now running on server1

Presto change-o, the diagnosis is complete. This is a development system, and so for the time being, I will accept defeat and workaround this problem by turning selinux off, but this is most definitely the wrong solution. If you’re an selinux guru who knows the proper fix, please let me know! Until then,

Happy Hacking,

James

 

How I broke (and fixed) my rgmanager service

Rgmanager, clustat and clusvcadm are useful tools in cluster land. I recently built a custom resource which I added to one of my service chains. Upon inspecting clustat, I noticed:

[root@server1 ~]# clustat
Member Status: Quorate

Member Name                             ID   Status
------ ----                             ---- ------
server1                                 1 Online, Local, rgmanager
server2                                 2 Online, rgmanager

Service Name                   Owner (Last)                   State
------- ----                   ----- ------                   -----
service:service-main-server1   (server1)                      failed

Looking at /var/log/messages, I found:

server1 rgmanager: [script] script:shorewall-reload: start of shorewall-reload.sh failed (returned 2)
server1 rgmanager: start on script "shorewall-reload" returned 1 (generic error)

This was peculiar, because my script didn’t have an exit code of 2 anywhere. It was due to a syntax error (woops)! Moving on with the syntax error fixed, I had trouble getting the service going again. In the logs I found these:

server1 rgmanager: #68: Failed to start service:service-main-server1; return value: 1
server1 rgmanager: Stopping service service:service-main-server1
server1 rgmanager: #12: RG service:service-main-server1 failed to stop; intervention required
server1 rgmanager: Service service:service-main-server1 is failed
server1 rgmanager: #13: Service service:service-main-server1 failed to stop cleanly

Running commands like: clusvcadm -e service-main-server1 didn’t help. It turns out that you have to first convince rgmanager that you truly fixed the problem, by first disabling the service. Now you can safely enable it and things should work smoothly:

clusvcadm -d service-main-server1
clusvcadm -e service-main-server1

Hopefully you’ve now got your feet wet with this clustering intro! Remember that you can look in the logs for clues and run clustat -i 1 in a screen session to keep tabs on things.

Happy Hacking,

James

 

visible close buttons on every firefox tab (including the last one)

After hacking around with a few firefox internals the other day, I decided there was another little annoyance that I had… When I’m acting insane and using my mouse to open and close tabs, the per tab close button disappears when there is only one tab left! This is apparently a feature, and while I can respect it as a default, it certainly isn’t an option for me. Thankfully, there are a lot of common minded individuals on the net, and the (hack) solution already existed. Here’s to explaining it clearly:

1. Create/edit a userChrome.css file in your profile directory. For me this path was:

   ~/.mozilla/firefox/xxxxxxxx.default/chrome/userChrome.css

2. Insert the following and restart firefox:

   .tab-close-button {display: -moz-box !important; }

3. Share and Enjoy!

Happy hacking,

James

PS: In case it’s not blatantly obvious, I didn’t invent any of this, but I am writing about it for your enjoyment, and for my own lousy memory.

more rows and columns on firefox new tab page

Firefox has a “new tab” speed dial type page available. I use it as my homepage (hint, use: about:newtab) and find it very useful for launching my often used favourites.

My one gripe is that it only shows you a default grid of 3 x 3. You can easily change that if you look in the secret preferences. Open a new tab and type in: about:config, accept the warning, and then search for: browser.newtabpage.rows and browser.newtabpage.columns. These values are easily editable by double clicking on the row. (Bold indicates non defaults.)

I chose a rectangular (column) size of 4, and now I’ve got just enough favourites to suit my frequent browsing habits.

Happy hacking,

James

PS: Now if firefox would only integrate natively with gnome-keyring like chrome and epiphany already do!

 

hello planet puppet

Hello planet puppet readers!

Brice Figureau[1] who runs planet puppet has just syndicated my blog, so thanks go out to him.

To all readers, I hope you enjoy my content, and please don’t be shy to comment and let me know if there are particular subjects or posts that I should elaborate on. I’ve got a whack of technical posts in the archives, so feel free to browse and let me know what’s good!

Only my puppet related posts should appear on planet puppet, so if you’re interested in other linux/sysadmin/technical posts, feel free to drop by.

Happy hacking,

James

[1] He seems to have a trusting, inclusive policy, so I hope to not disappoint :)