• Jul 11, 2007

    Better JSON output from Rails with the Jsonifier plugin

    Update: Edge Rails now does whatever Jsonifier does (and more). Check out my blog post on JSON serialization maturity in edge Rails. This means that this plugin is virtually obsolete if you're using edge Rails.

    If you've tried to output JSON from your Rails applications before, you'd probably have noticed how inadequate it all seems. Let's look at the kind of output (pretty-printed for easier reading) you get from calling to_json on your typical User model:

    {
  attributes: {
    id: 1,
    name: "Konata",
    awesome: true,
    created_at: "07/01/2007"
  }
}

    Some of the common problems developers who write applications that speak JSON have with this are:

    • The attributes part is often unnecessary cruft when consuming ActiveRecord objects as JSON.
    • There's no obvious way to control which attributes are shown in the JSON (e.g. you may want to hide private or lengthy attributes). The way to workaround this is either to write your own to_json instance method for your ActiveRecord model, or to use a Rails plugin (like acts_as_json) or a gem.
    • Including 2nd (or higher order) associations is tricky (as above, you can achieve this by defining your own to_json instance methods).

    Under the hood of the current ActiveRecord#to_json

    The to_json method for ActiveRecord objects falls back on using that defined in ActiveSupport's JSON classes. The way it's currently done, there are a bunch of JSON encoder classes defined for various types - when you call to_json on an ActiveRecord object it simply uses Object#to_json.

    By the way, see that created_at: "07/01/2007" bit in the example above which is the JSON output for a Date attribute? That's in MM/DD/YYYY format. Yup, 1st July 2007 not 7th January 2007 you "rest of the world you". Ouch, but at least you can use JavaScript's Date.parse() on it directly as detailed in this patch. And before you ask, yes, Date.parse() doesn't understand the YYYY-MM-DD format.

    Can has Rails patch?

    So about 2 days ago, I wrote a Rails patch to boost ActiveRecord#to_json with an options hash. Yep much like what you can do with ActiveRecord#to_xml.

    I hope it gets committed into the core some day. I mean, gosh, isn't Rails all about Web 2.0 mashups. It even has that fancy render :json thing. I do believe JSON output needs to be treated more seriously in Rails rather than the half-hearted attempt at JSON encoding. No? Well, if the patch doesn't get accepted I'll still have my plugin (see below).

    Anyway, let's see some examples:

    Some examples, see?

    Assuming User and Post models where User has_many Posts:

    david = User.find(1)
david.to_json  # {id: 1, name: "David", awesome: true, created_at: "07/01/2007"}

    No more attributes cruft!

    :only and :except work the same way that as for ActiveRecord#to_xml:

    david.to_json(:only => :name)                 # {name: "David"}
david.to_json(:only => [:id, :name])          # {id: 1, name: "David"}
david.to_json(:except => :created_at)         # {id: 1, name: "David", awesome: true}
david.to_json(:except => [:id, :created_at])  # {name: "David", awesome: true}

    You can use the :methods options as well to include any methods on the object.

    david.to_json(:methods => :permalink)
  # {id: 1, name: "David", awesome: true, created_at: "07/01/2007", permalink => "1-David"}
david.to_json(:methods => [:permalink, :interestingness])
  # {id: 1, name: "David", awesome: true, created_at: "07/01/2007", \
  #   permalink => "1-David", :interestingness => 666}

    The :include option lets you include associations.

    david.to_json(:include => :posts)
  # {id: 1, name: "David", awesome: true, created_at: "07/01/2007", \
  #    posts: [{id: 1, author_id: 1, title: "Welcome to the weblog"}, \
  #            {id: 2, author_id: 1, title: "So I was thinking"}]}

    :only, :except, and :methods works on the included associations as well:

    david.to_json(:include => { :posts => { :only => :title } })
  # {id: 1, name: "David", awesome: true, created_at: "07/01/2007", \
  #    posts: [{title: "Welcome to the weblog"}, \
  #            {title: "So I was thinking"}]}

    Of course, 2nd level (and higher order) associations work too:

    david.to_json(:include => { :posts => { \
                              :include => { :comments => { \
                                              :only => :body } }, \
                              :only => :title } })
  # {id: 1, name: "David", awesome: true, created_at: "07/01/2007", \
  #    posts: [{comments: [{body: "1st post!"}, {body: "OMGWTFBBQ!"}], \
  #             title: "Welcome to the weblog"}, \
  #            {comments: [{body: "Don't think too hard"}], \
  #             title: "So I was thinking"}]}

    Please do give any feedback at the Rails patch for ActiveRecord#to_json or here. I wanna know what you feel about adding this functionality into the Rails core!

    Jsonifier plugin

    I've rolled the patch into a plugin: Jsonifier. The Jsonifier Trac is at http://trac.codefront.net/jsonifier/.

    Git repository:

    git://github.com/chuyeow/jsonifier.git

    Github project page:

    http://github.com/chuyeow/jsonifier/

    To install the plugin:

    ruby script/plugin install git://github.com/chuyeow/jsonifier.git

    or git clone the repository manually into the vendor/plugins directory.

    A note on valid JSON

    The JSON Rails spits out by default is not strictly valid JSON since the JSON specifications require keys to be double quoted. To get strictly valid JSON, add

    ActiveSupport::JSON.unquote_hash_key_identifiers = false

    in your environment.rb (or in the Rails initializers directory if you're on edge). See my blog post on how to get strictly valid JSON from Rails for more info.

  • Jul 6, 2007

    Haruhi Season 2 announced!

    Season 2 of The Melancholy of Haruhi Suzimiya was announced in the August 2007 edition of NewType Japan - check out the scans. The announcement isn't official, but there'll be one on the 7th - that's 7/7/2007!

  • Jul 2, 2007

    WTF is this? (Or how NIS and portmapper on Ubuntu Dapper Drake drove me nuts)

    Not too long ago a couple of the Ubuntu Dapper Drake (6.06 LTS) VPSs we use abruptly became inaccessible. These VPSs were NIS clients that suddenly could no longer connect to the NIS master. Great. And as far as I could tell I didn't do anything to those servers. I even managed to replicate the problem on another similar VPS (that was still accessble) simply by restarting the NIS daemon.

    Thanks to Rimuhosting's fantabulous support, I managed to regain access to the VPSs. Oh yeah, big lesson learnt here: when configuring NIS clients, be sure to leave at least one user account with sudo privileges or set the root account's password (Ubuntu doesn't come with a root account enabled by default - run sudo passwd to set it and enable the root account as a side effect). Big big lesson learnt. Don't let this bite you in the ass if you're a sysadmin-ish person.

    Anyway, /var/log/syslog was complaining this:

    ypbind[1026]: Unable to register (YPBINDPROG, YPBINDVERS, udp)

    Which turned out to be a portmap problem. So yeah portmap wasn't running! For a moment there I thought we had an easy solution. Sadly, /etc/init.d/portmap start was throwing this inexplicable error (i.e. Google wasn't of much help):

    root@zoidberg:/sbin# /etc/init.d/portmap  restart
 * Stopping portmap daemon...
   ...done.
 * Starting portmap daemon...
dhclient.4.x 2.*.5, December 30th, 2000
Compiled on Dec 12 2003 09:55:54
Features: DBG, SEE, ALS, SEF
init: Couldn't open the file ./libuuid.so ->
   ...fail!

    It sure isn't caturday. After hours spent scrambling to find out why portmap can't find libuuid.so, reinstalling the NIS .deb, comparing configuration files between working NIS clients and these busted ones, it finally dawned on me that I have yet to re-install portmap (not that it was logically sound that I'd have to reinstall it, but at this point I was willing to try anything). Annoyingly enough, it worked!

    apt-get remove portmap nis
apt-get install nis

    I still don't know WTF happened there. If anyone has any clue please please point me to it.

  • Jun 26, 2007

    Installing Xen on Ubuntu Feisty Fawn - the complete newbie's guide

    I had to spend some time setting up Xen on one of the new Dell servers we bought and while there was some documentation around, I had to constantly refer to the different sources since there wasn't a complete page which had everything I needed for my particular situation - I have a host machine with Ubuntu Feisty Fawn installed and wanted to install a couple of Xen guest domains on it. One other thing I found lacking in the documentation is the explanation of commands or answers to the question "so why the hell am I doing this?". I'm not considering LVM at the moment and am happy with loopback disk file images too.

    So I think it's a good idea to write this down somewhere for the next time I have to do the same bloody thing over again on any new servers.

    Some caveats before we start:

    • Like I said, I'm not gonna talk about LVM for disk images since I know next to nothing about configuring it at the moment. The host machine I have isn't setup with LVM and I haven't got the time to look into LVM either.
    • I'm quite the networking newbie, so my setup uses the default Xen bridge installed via apt-get. Check out Xen networking on the Xen wiki for more complex network setups.
    • I'm not gonna talk about putting a Linux distro other than that of the host machine into the Xen virtual machines. In other words, I have Feisty Fawn for both dom0 and all my domUs.
    • I probably did some completely stupid things. Stuff like xen-tools exist to simplify Xen administration, but I haven't had a chance to use them. Let me know if you spot an error or have a better way of doing something please!

    Some terms that deserve explaining (most guides out there just use "DomU" and "Dom0" without any ceremony, and assume you know what they mean):

    • Domain-0 or Dom0: This refers to the host machine OS. You know, the OS of the actual physical server that you have.
    • DomU: This refers to a Xen guest domain. A DomU is a single Xen virtual machine. The "U" stands for "unprivileged" I believe.

    OK let's go!

    Installing the Xen Server

    First off, we need to install the Xen server on the host machine (or Dom0). Thankfully, there is a Xen server package in Feisty Fawn's apt-get repository.

    • Make sure 'universe' is enabled in /etc/apt/sources.list and run: apt-get install ubuntu-xen-server
    • Reboot the machine. After boot, running sudo xm list should show you Domain-0 (which is the host machine that's running Xen).
    • Edit /etc/xen/xend-config.sxp and uncomment this line: 
      (network-script network-bridge)
    • Comment out this line: 
      (network-script network-dummy)
    • Restart xend: 
      sudo xend restart
    • When you run 'ifconfig', you should see the 'xenbr0' interface. This is the Xen network bridge.

    Creating a base image

    Now let's create a base Xen image. We'll use this as a template for any future images (domUs).

    • Use 'dd' to create the images (I place them in /xen-images/): 
      dd if=/dev/zero of=/xen-images/feisty_base.img bs=1024k count=4000
dd if=/dev/zero of=/xen-images/feisty_base_swap.img bs=1024k count=1000

      These commands create image files of 4GB and 1GB for your virtual OS and its swap respectively. 'count' is the number of blocks (block size = 1024k).

    • Create the filesystem (we're using ext3 in this case) within the virtual OS image file. 
      mkfs.ext3 /xen-images/feisty_base.img

      Answer 'yes' to proceed anyway when it complains about the file not being a block device.

    • Do the same for the swap file: 
      mkswap /xen-images/feisty_base_swap.img
    • Now, restrict the permissions on the images so no one else but root can screw around with them. 
      chmod 640 /xen-images/feisty_base*
    • Create a mount point for the image. We need this to mount the image to install and configure Ubuntu Feisty Fawn the way we like it. We can then replicate this base image for any future virtual machines. 
      mkdir /xen-images/mnt
    • Mount the base image via the loop device: 
      mount -o loop /xen-images/feisty_base.img /xen-images/mnt

      Now we have full access to the base image's filesystem. What we need to do now is to put a base install of Feisty Fawn on it.

    Installing and configuring the base image

    Next, we have to put Ubuntu onto the image and configure it to our liking.

    • Install debootstrap, a handy tool that bootstraps a basic Debian (Ubuntu in this case) system. 
      apt-get install debootstrap
    • Bootstrap the base image with 'feisty': 
      debootstrap feisty /xen-images/mnt

      After this is done you should see the basic Ubuntu file hierarchy:

      ls /xen-images/mnt
    • Copy your sources.list into the base image: 
      cp /etc/apt/sources.list /xen-images/mnt/etc/apt/
    • Copy your kernel modules as well (replace 2.6.19-4-server with whatever kernel version you have): 
      cp -a /lib/modules/2.6.19-4-server/ /xen-images/mnt/lib/modules/
    • Install the libc6-xen package, which is a Xen-compatible glibc that allows applications to use Thread-Local Storage (TLS). 
      apt-get install libc6-xen

      Taken from the Xen FAQ:

      Some modern distributions ship with a 'TLS' version of glibc that is not fully compatible with Xen. To use Xen reliably and with maximum performance you must disable the incompatible glibc.

    • Configure networking by editing /xen-images/mnt/etc/network/interfaces: 
      auto lo
iface lo inet loopback

# Uncomment this and fill up with your networks settings.
#auto eth0
#iface eth0 inet static
#  address 192.168.0.201
#  netmask 255.255.255.0
#  broadcast 192.168.255.255
# gateway 192.168.0.1
#  dns-nameservers 192.168.0.1

      Notice how the network settings are commented out - this is because we are leaving the base image as a template from which other images would be made from.

    • Edit the hosts file (/xen-images/mnt/etc/hosts): 
      127.0.0.1       localhost localhost.localdomain
127.0.0.1       guest

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
    • Give the base image a generic hostname by editing /xen-images/mnt/etc/hostname. In this case I'm just naming it 'guest' - actual virtual machines would have sensible hostnames.
    • Edit the filesystem table (/xen-images/mnt/etc/fstab). 
      proc            /proc           proc    defaults        0       0
/dev/hda1       /               ext3    defaults,errors=remount-ro 0       1
/dev/hda2       none            swap    sw              0       0

      Don't worry, the /dev/hda* would be configured to point to the images you created, not to your actual physical harddisks.

    • You're done with the initial configuration and have enough to bootup the image. Let's unmount the image: 
      umount /xen-images/mnt
    • Now, we need to create a Xen config for the base image so we can boot into it. Create a file named /etc/xen/vm_feisty_base.config.example.sxp (you can name it whatever you like of course): 
      name = "feisty_base"
kernel = "/boot/vmlinuz-2.6.19-4-server"
ramdisk = "/boot/initrd.img-2.6.19-4-server"
root = "/dev/hda1 ro"
memory = 512
disk = ['file:/xen-images/feisty_base.img,hda1,w','file:/xen-images/feisty_base_swap.img,hda2,w']

# Network.
hostname = "feisty_base"
vif = ['bridge=xenbr0']
dhcp = "off"
ip = "192.168.0.201"
netmask = "255.255.255.0"
gateway = "192.168.0.1"

    An explanation of the configuration options:

    • name - a descriptive name of your image.
    • kernel - path to the kernel.
    • ramdisk - path to the initrd (initial RAM disk) image.
    • root - point this to your host machine's harddisk where your image files are.
    • memory - the amount of memory assigned to the virtual machine in megabytes
    • disk - specify a list of block devices that will be export to the virtual machine. In this example, we're mapping the /xen-images/feisty_base.img image file to /dev/hda1 in the virtual image (same for the swap image). This should match whatever config you did in /xen-images/mnt/etc/fstab.
    • hostname - hostname of your virtual machine.
    • vif - virtual network interface that your virtual machine will use. In this case, we're using the default 'xenbr0' bridge that Xen installs. For alternative configurations, check out the networking page on the Xen wiki.
    • ip - specify your virtual machine's IP here.
    • netmask and gateway - netmask and gateway that your virtual machine should use.
    • Now, start the base virtual machine: 
      xm create /etc/xen/vm_feisty_base.config.example.sxp -c

      You should see a console (the '-c' option connects you to the console immediately after booting up the virtual machine. Login as root (empty password) - you'll want to change the password (with 'passwd').

    • Enable shadowed passwords. 
      shadowconfig on
    • Create a backup of /etc/network/interfaces and edit /etc/network/interfaces to put in correct values so you can access the Internet from the virtual machine: 
      cp /etc/network/interfaces /etc/network/interfaces.bak
vim /etc/network/interfaces
    • Start networking: 
      ifup -a
    • Install the Ubuntu standard system after running an update: 
      apt-get update
apt-get install ubuntu-standard
    • I also like to upgrade any packages that have been updated, if any: 
      apt-get upgrade
    • Install the OpenSSH server so that you can access the virtual machine via ssh: 
      apt-get install ssh
    • At this point, do any configuration or installation of packages that you want this base image to have. Remember that we're going to replicate this base image for future Xen virtual machines so don't go overboard and install stuff you're not gonna use.
    • Stop networking and put back the original /etc/network/interfaces: 
      ifdown -a
mv /etc/network/interfaces.bak /etc/network/interfaces
    • Exit the guest console with Ctrl-]. This will bring you back to dom0's shell.
    • Shutdown the base Xen domU (-w means "wait for the domU to shutdown completely"): 
      xm shutdown feisty_base -w

    You're done setting up the base image! Now you have a nicely configured base image.

    Creating your first virtual server

    The fruits of thy labor are almost at hand - with the base image, we can now make a copy of it for our first Xen virtual machine.

    • Make copies of the base image and the swap image (let's call our virtual server "yuki"): 
      cp /xen-images/feisty_base.img /xen-images/yuki.img
cp /xen-images/feisty_base_swap.img /xen-images/yuki_swap.img

      You may want to resize the base image depending on your disk space needs. (And, of course, you can create your own swap image like we did earlier instead of copying the base image's swap file.)

    • Make a copy of the base Xen configuration file that we created for our base image earlier (you can use whatever name you like): 
      cp /etc/xen/vm_feisty_base.config.example.sxp /etc/xen/vm_yuki.config.sxp
    • Edit the /etc/xen/vm_yuki.config.sxp file, changing the values to suit your virtual server (you probably want to change at least the 'name', 'disk', 'ip, and 'hostname' values). 
      name = "yuki"
disk = ['file:/xen-images/yuki.img,hda1,w','file:/xen-images/yuki_swap.img,hda2,w']
hostname = "yuki"
ip = "192.168.0.202"
    • Now start up your cloned virtual server and log into it like we did earlier with the base image (if you haven't changed the password, you can login as root with an empty password). 
      xm create -c /etc/xen/vm_yuki.config.sxp
    • The first thing you should do is to secure your root password (if you haven't done so in the base image earlier). 
      passwd
    • Now, you'll want to change these files:
      • /etc/network/interfaces
      • /etc/hostname
      • /etc/hosts
    • With the proper network settings, you should be able to bring up networking to connect to the internet (assuming that your dom0 is connected to the internet). 
      ifup -a
ping google.com
    • At this point, you are free to customize your virtual machine as you see fit - install nginx for use as a static file server, Rails, MySQL server, whatever.
    • But first, let's make sure this Xen virtual server starts up automatically on boot. Exit the virtual machine console with Ctrl-] 
      ln -s /etc/xen/vm_yuki.config.sxp /etc/xen/auto/

      Any Xen domain configuration placed in /etc/xen/auto/ will be started up automatically on boot (and shutdown when the host machine is shutdown too).

    That's it! You have a functional Xen guest domain that starts up on boot and that can connect to the Internet. Oh before I forget, some useful commands when dealing with the Xen images:

    • xm create /path/to/config_file - Starts up the guest domain with the given Xen domU configuration file. (xm create -c to login to the console immediately after booting the image.)
    • xm console yuki - Open the console for the 'yuki' Xen domain.
    • xm shutdown yuki - Shutdown the 'yuki' domU.
    • xm list - Lists all the Xen domains you have running.

    Resizing your virtual machine disk

    Let's say you wanna add 1GB of disk space to your virtual machine (named 'yuki').

    Note that LVM is usually recommended for Xen setups, so you may wanna take a look at that instead.

    • Create a temporary 1GB file: 
      dd if=/dev/zero of=/tmp/temp_expand bs=1024k count=1000
    • Bring down the virtual machine (I've found that a shutdown is needed so that the virtual machine recognizes the increased disk space): 
      xm shutdown yuki
    • Append the file to the disk image of the virtual machine (/xen-images/yuki.img in this example). Be sure to use '>>' (which means append) - using a single '>' would overwrite the disk image with your temporary file, which you most definitely don't want! 
      cat /tmp/temp_expand >> /xen-images/yuki.img
    • Resize the file system: 
      resize2fs -f /xen-images/yuki.img
    • Startup the virtual machine again: 
      xm create -c /etc/xen/vm_yuki.config.sxp
    • Remember to login to your virtual machine to verify that the disk space has increased (df -h)
  • Jun 21, 2007

    Prototype 1.5.1 is baaaaad for Safari 1 and 2 users

    Read the bug fix announcement. Upgrade! This was causing extremely puzzling crashes in Safari on Macs (pre-Safari 3 beta) on one of our applications and I'm glad (in some ways) that the problem lay with the Prototype library.

subscribe via RSS