I needed a base image “box” for my Puppet-Gluster+Vagrant work. It would have been great if good boxes already existed, and even better if it were easy to build my own. As it turns out, I wasn’t able to satisfy either of these conditions, so I’ve had to build one myself! I’ve published all of my code, so that you can use these techniques and tools too!
Status quo:
Having an NIH problem is bad for your vision, and it’s best to benefit from existing tools before creating your own. I first tried using vagrant-cachier, and then veewee, and packer. Vagrant-cachier is a great tool, but it turned out not being very useful because there weren’t any base images available for download that met my needs. Veewee and packer can build those images, but they both failed in doing so for different reasons. Hopefully this situation will improve in the future.
Writing a script:
I started by hacking together a short shell script of commands for building base images. There wasn’t much programming involved as the process was fairly linear, but it was useful to figure out what needed getting done.
I decided to use the excellent virt-builder command to put together the base image. This is exactly what it’s good at doing! To install it on Fedora 20, you can run:
$ sudo yum install libguestfs-tools
It wasn’t available in Fedora 19, but after a lot of pain, I managed to build (mostly correct?) packages. I have posted them online if you are brave (or crazy?) enough to want them.
Using the right tool:
After building a few images, I realized that a shell script was the wrong tool, and that it was time for an upgrade. What was the right tool? GNU Make! After working on this for more hours than I’m ready to admit, I present to you, a lovingly crafted virtual machine base image (“box”) builder:
The Makefile itself is quite compact. It uses a few shell scripts to do some of the customization, and builds a clean image in about ten minutes. To use it, just run make.
Customization:
At the moment, it builds x86_64, CentOS 6.5+ machines for vagrant-libvirt, but you can edit the Makefile to build a custom image of your choosing. I’ve gone out of my way to add an $(OUTPUT) variable to the Makefile so that your generated files get saved in /tmp/ or somewhere outside of your source tree.
Download the image:
If you’d like to download the image that I generated, it is being generously hosted by the Gluster community here. If you’re using the Vagrantfile from my Puppet-Gluster+Vagrant setup, then you don’t have to download it manually, this will happen automatically.
Open issues:
The biggest issue with the images is that SELinux gets disabled! You might be okay with this, but it’s actually quite unfortunate. It is disabled to avoid the SELinux relabelling that happens on first boot, as this overhead defeats the usefulness of a fast vagrant deployment. If you know of a way to fix this problem, please let me know!
Example output:
If you’d like to see this in action, but don’t want to run it yourself, here’s an example run:
$ date && time make && date Mon Jan 20 10:57:35 EST 2014 Running templater... Running virt-builder... [ 1.0] Downloading: http://libguestfs.org/download/builder/centos-6.xz [ 4.0] Planning how to build this image [ 4.0] Uncompressing [ 19.0] Resizing (using virt-resize) to expand the disk to 40.0G [ 173.0] Opening the new disk [ 181.0] Setting a random seed [ 181.0] Setting root password [ 181.0] Installing packages: screen vim-enhanced git wget file man tree nmap tcpdump htop lsof telnet mlocate bind-utils koan iftop yum-utils nc rsync nfs-utils sudo openssh-server openssh-clients [ 212.0] Uploading: files/epel-release-6-8.noarch.rpm to /root/epel-release-6-8.noarch.rpm [ 212.0] Uploading: files/puppetlabs-release-el-6.noarch.rpm to /root/puppetlabs-release-el-6.noarch.rpm [ 212.0] Uploading: files/selinux to /etc/selinux/config [ 212.0] Deleting: /.autorelabel [ 212.0] Running: yum install -y /root/epel-release-6-8.noarch.rpm && rm -f /root/epel-release-6-8.noarch.rpm [ 214.0] Running: yum install -y bash-completion moreutils [ 235.0] Running: yum install -y /root/puppetlabs-release-el-6.noarch.rpm && rm -f /root/puppetlabs-release-el-6.noarch.rpm [ 239.0] Running: yum install -y puppet [ 254.0] Running: yum update -y [ 375.0] Running: files/user.sh [ 376.0] Running: files/ssh.sh [ 376.0] Running: files/network.sh [ 376.0] Running: files/cleanup.sh [ 377.0] Finishing off Output: /home/james/tmp/builder/gluster/builder.img Output size: 40.0G Output format: qcow2 Total usable space: 38.2G Free space: 37.3G (97%) Running convert... Running tar... ./Vagrantfile ./metadata.json ./box.img real 9m10.523s user 2m23.282s sys 0m37.109s Mon Jan 20 11:06:46 EST 2014 $
If you have any other questions, please let me know!
Happy hacking,
James
PS: Be careful when writing Makefile‘s. They can be dangerous if used improperly, and in fact I once took out part of my lib/ directory by running one. Woops!
UPDATE: This technique now exists in it’s own repo here: https://github.com/purpleidea/vagrant-builder
The Technical Blog of James 
This tip should work, at least as an interim measure, to allow you to enable SELinux and do the labelling step at build time:
https://github.com/libguestfs/libguestfs/commit/20a4bfde9628cfeb8bea441cab7dcc94843b34e3
” first tried using vagrant-cachier, and then veewee, and packer. Vagrant-cachier is a great tool, but it turned out not being very useful because there weren’t any base images available for download that met my needs. Veewee and packer can build those images, but they both failed in doing so for different reasons.”
It would be very enlightening and interesting to hear why Packer failed. I managed to get Packer building vagrantboxes relatively easily, and the process is robust enough, and provides a lot of options in the workflow. Before venturing into YASS (Yet Another Set of Scripts) I’d like to understand why Packer failed?
I don’t have my packer notes handy, but they’re fairly easy to reproduce:
1) Try installing and building a vagrant-libvirt compatible box on Fedora 20.
2) Fail…
If this has changed since my tests, please let me know and I’ll consider giving it another shot!
You might have different success build images for virtualbox. I believe it’s the default target for packer, but personally, I find this useless because virtualbox is proprietary, and if I can’t hack on something, then I can’t build awesome things. From a technical point of view, I prefer libvirt/qemu/kvm.
I can’t comment on the design of packer because I haven’t looked at those details, but I can tell you that it’s almost assuredly much more complicated, than my 66 line Makefile. I also think that using a Makefile is a very elegant design. I’m able to achieve such a low LOC count because I’m using great tools such as virt-builder and make.
$ grep -v '^#' Makefile | grep -v '^$' | wc -l
66
Let me know if you have any other questions, and good luck!
Thanks for the clarification. I missed the context of this being a libvirt based box, which explains why you had a difficult time I guess.
Pingback: [Howto] Vagrant: libvirt, (Multi-)Multi-Machine, Ansible and Puppet | /home/liquidat – Open Source, Linux and Business
I came across this trying to resize some vagrant boxes, but will attempt to use this to make a custom box.
I found this link broken https://github.com/purpleidea/puppet-gluster/blob/master/builder/Makefile
Added a link to https://github.com/purpleidea/vagrant-builder at the bottom, since this is out of date now. Cheers!