Knowing when to release and deploy your code (…and a mini script)

Knowing when to release and deploy your code can turn into a complicated discussion. In general, In general, I tend to support releasing early and often, for some value of $early and $often. I’ve decided to keep this simple and introduce you to one metric that I use…

I think that I am fairly diligent in adding plenty of comments to my source code. I might even sometimes add too many. I create plenty of XXX, FIXME, or TODO tagged comments as reminders of things to work on.

To me, XXX represents an important problem that should get looked at or fixed; FIXME, reminds me that I should definitely look into something, and finally, TODO gives me homework or things to pursue when I’m in need of a new project.

I try to resolve most if not all XXX tagged comments before making a 0.1 release, FIXME’s to consider something very stable, and a lack of TODO’s mean something is completely done for now.

To count all these, I wrote a little tool that greps through the top-level directories in my ~/code/ folder, and displays the results in a table. Feel free to give it a try, and use it for your own projects.

While I don’t see this as a particularly game changing utility, it scratches my itch, and helps me keep up my bash skills. The code is available here. Let me know if you have any improvements, or if the source isn’t enough documentation for you.

Happy hacking,

James

running your file manager from a terminal

I do a lot of my work in a terminal. For the unfamiliar, this might seem strange, however once you’re comfortable with your shell, this is the best place to be. I don’t restrict myself to it though. I often want to spawn a file manager, or a graphical text editor. When I run nautilus, I usually see something like this:

james@computer:~/some/awesome/directory$ nautilus .
Initializing nautilus-open-terminal extension
Shutting down nautilus-open-terminal extension
james@computer:~/some/awesome/directory$

This is useful, because I can open a file browser right where I want it, it’s annoying, because nautilus runs in that terminal until I close it. (This doesn’t happen if the nautilus process is always running, but since GNOME 3, it isn’t.)

My solution is a short bash script that runs nautilus, and leaves your terminal alone. I named my script nautilus, and placed it inside my ~/bin/. Here is the script:

#!/bin/bash
# run nautilus from a terminal, without being attached to it; similar to nohup
# use the full path of nautilus to avoid it calling itself (recursion!)
{ `/usr/bin/nautilus "$@" &> /dev/null`; } &

I hope this is useful for you too. Feel free to do the same for gedit, nemo, and any other app which you often find convenient to run from the terminal. You can generalize this by leaving out the nautilus program:

#!/bin/bash
if [ "$1" == "" ]; then
        echo "usage: ./"`basename $0`"  (to run a command nohup style)"
        exit 1
fi
# do a nohup bash style according to:
{ `"$@" &> /dev/null`; } &

I name the above script run.sh, and it helps me out from time to time, when I don’t want to touch my mouse.

In case you haven’t heard about it, there’s also an open-terminal extension for nautilus and nemo which lets you get to a terminal, from your file manager. A quick internet search should help you install it.

If you found this information useful, please let me know, and as always,

Happy hacking,

James

PS: If you plan to do this for gedit, you probably want to preserve stdin, so that you can still pipe things in. To do this, you’ll probably want:

{ `/usr/bin/gedit "$@" &> /dev/null`; } < /dev/stdin &    # accept stdin too!

adding range support to python’s http server to kickstart with anaconda

I’ve been working on automatic installs using kickstart and puppet. I’m using a modified python httpserver because it’s lightweight, and easy to integrate into my existing python code base. The server was churning away perfectly until anaconda started downloading the full rpm’s for installation. What was going wrong?

Traceback (most recent call last):
[...]
error: [Errno 32] Broken pipe
BorkedError: See TTBOJ for explanation and discussion

As it turns out, anaconda first downloads the headers, and then later requests the full rpm with an http range request. This second range request which begins at byte 1384, causes the “simple” httpserver to bork, because it doesn’t support this more elaborate feature.

After a bit of searching, I found rangehttpserver and was very grateful that I wouldn’t have to write this feature myself. This work by smgoller was based on the similar httpserver by xyne. Both of these people have been very responsive and kind in giving me special permission to the relevant portions of their code that I needed under the GPLv2/3+. Thanks to these two and their contribution to Free Software this let’s us all see further, instead of having to reinvent previously solved problems.

This derivative work is only one part of a larger software release that I have coming shortly, but I wanted to put this out here early to thank these guys and to make you all aware of the range issue and solution.

Thank you again and,
Happy Hacking,

James

A quick anaconda trick

Here’s a quick anaconda solution that I am now using in some of my kickstart files…

I wanted to bootstrap a machine and do all the partitioning and logical volume creation, but not format or mount one of the logical volumes. The magic parameter I needed was:

--fstype=none

This seems to work perfectly for me. It’s not 100% intuitive to me, but it does work. I hope it’s not an accidental bug in the anaconda code! The full text of my partitioning is:

clearpart --all --drives=sda
part /boot --fstype=ext4 --size=1024
part pv.01 --grow --size=1024
volgroup VolGroup00 --pesize=4096 pv.01
logvol / --fstype=ext4 --name=root --vgname=VolGroup00 --size=65536
logvol swap --name=swap --vgname=VolGroup00 --size=16384
logvol /foo --name=foo --vgname=VolGroup00 --fstype=none --grow --size=1

Now all that anaconda is missing is support for RAID1 EFI /boot.

Happy hacking,

James

Automatic hiera lookups in puppet 3.x

Dear readers,

I’ve started the slow migration of code from puppet 2.6 all the way to 3.x+. There were a few things I wasn’t clear on, so hopefully this will help to discuss these and make your migration easier!

I used hiera in 2.6, and I actually like it a lot so far. I was concerned that automatic lookups would pull in values that I wasn’t expecting. This is not the case or a worry. Let’s dive in and let the code speak:

# create a class in a module or site.pp for testing...
class foo(
        $a = 'apple',
        $b = 'banana'
) {
        notify { 'foo':
                message => "a is: ${a}, b is: ${b}",
        }
}

and

# define it using :: as a prefix because we want to search in the
# top level, module namespace. optional if we only have one foo.
class { '::foo':
}

and

# /etc/puppet/hiera.yaml
:backends:
        - yaml

:hierarchy:
        - globals
        - whatever
        - youlike

:yaml:
        :datadir: /etc/puppet/hieradata/

and

# /etc/puppet/hieradata/whatever.yaml (because of - whatever above)
---
foo::a: 'somevalue' # this many colons is actually valid syntax
dude: 'sweet'

will produce:

[...]
Notice: a is somevalue, b is: banana
Notice: Finished catalog run in 3.14159265359 seconds

This is the automatic lookup. You probably have zero risk of collision with earlier data in your hiera yaml files, because these lookups use keys that match the classname::paramname pattern. If you had used :: (double colons) in your keys before, then you’re insane, and you should check for any collisions! The downside to this is that my whatever.yaml looks awkward with all those colons, but I got over that very quickly.

The full lookup order is first:

# directly specified values first (of course)
class { '::foo':
        a => 'this value is used first if set.',
}

and then:

# values matching an appropriate yaml key:
---
foo::a: 'this value is used next if found.'

and finally:

class foo(
        $a = 'this parameter default value is used last.'
        $b = 'b is still for banana...'
) {
        # do stuff...
}

all as detailed in: http://projects.puppetlabs.com/issues/11608. Finding this link and setting me down the path to knowledge was all thanks to eric0 in #puppet. Thanks Eric!

Make sure to reload your puppetmaster after you make any changes to /etc/puppet/hiera.yaml, and as always:

Happy hacking,

James

How to send and receive files like a professional

Everyone needs to send and receive files sometimes. Traditionally people send files as email attachments. This still works great, and supports encryption, but many mail servers are slow and cap the upper file size limit.

ICQ was a great solution back in the 1990’s, but those days are now over. (I still remember my number.)

A lot of folks use dropbox, which requires a dropbox account, and for you to trust them with your files.

If you want a simple solution that doesn’t need internet access (if you’re on a LAN, for example) you can use droopy and woof. These are two shell scripts that I keep in my ~/bin/. Droopy lets you receive a file from a sender, and woof lets you send one their way. Unfortunately, they don’t support ssl. This could be a project for someone. (Hint)

I recently patched droopy to add inline image support. I’ve emailed my patch to the author, but until it gets merged, you can get my patched version here. (AGPLv.3+)

Hopefully these are helpful to you.

Happy hacking,

James

Clustering virtual machines with rgmanager and clusvcadm

This could be a post detailing how to host clustered virtual machines with rgmanager and clusvcadm, but that is a longer story and there is much work to do. For now, I will give you a short version including an informative “gotcha”.

With my cluster up and running, I added a virtual machine entry to my cluster.conf:

<vm name="test1" domain="somedomain" path="/shared/vm/" autostart="0" exclusive="0" recovery="restart" use_virsh="1" />

This goes inside the <rm> block. As a benchmark, please note that starting the machine with virsh worked perfectly:

[root@server1 ~]# virsh create /shared/vm/test1.xml --console
(...The operation worked perfectly!)

However, when I attempted to use the cluster aware tools, all I got was failure:

[root@server1 ~]# clusvcadm -e 'vm:test1' -m server1
Member server1 trying to enable vm:test1...Failure

Whenever I think I’ve done everything right, but something is still not working, I first check to see if I can blame someone else. Usually that someone is selinux. Make no mistake, selinux is a good thing^™, however it does still cause me pain.

The first clue is to remember that /var/log/ contains other files besides “messages“. Running a tail on /var/log/audit/audit.log while simultaneously running the above clusvcadm command revealed:

type=AVC msg=audit(1357202069.310:10904): avc:  denied  { read } for  pid=15675 comm="virsh" name="test1.xml" dev=drbd0 ino=198628 scontext=unconfined_u:system_r:xm_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=file
type=SYSCALL msg=audit(1357202069.310:10904): arch=c000003e syscall=2 success=no exit=-13 a0=24259e0 a1=0 a2=7ffff03af0d0 a3=7ffff03aee10 items=0 ppid=15609 pid=15675 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="virsh" exe="/usr/bin/virsh" subj=unconfined_u:system_r:xm_t:s0 key=(null)

I am not a magician, but if I was, I would probably understand what all of that means. For now, let’s pretend that we do. Closer inspection (or grep) will reveal:

• “test1.xml” (the definition for the virtual machine)

and:

• “/usr/bin/virsh” (the command that I expect rgmanager’s /usr/share/cluster/vm.sh script to run)

A quick:

[root@server1 ~]# selinuxenabled && echo t || echo f
t

to confirm that selinux is auditing away, and a short:

[root@server1 ~]# /bin/echo 0 > /selinux/enforce

to temporarily test my theory, and:

[root@server1 ~]# clusvcadm -e 'vm:test1' -m server1
Member server1 trying to enable vm:test1...Success
vm:test1 is now running on server1

Presto change-o, the diagnosis is complete. This is a development system, and so for the time being, I will accept defeat and workaround this problem by turning selinux off, but this is most definitely the wrong solution. If you’re an selinux guru who knows the proper fix, please let me know! Until then,

Happy Hacking,

James