Jeremy Durham

Although my DNS-321 is last year’s model, having been superseded by the awesome DNS-320, I decided last night that I’d love to get OpenVPN running on it and set up a VPN that would give me access to my shared directories and home computer from the office.

OpenVPN requires liblzo, which is available for fun_plug, so I grabbed that and installed it:

wget http://www.inreto.de/dns323/fun-plug/0.5/extra-packages/libs/lzo-2.03-1.tgz

funpkg -i lzo-2.03-1.tgz

There is also an OpenVPN package, but it’s built for the DNS-323, so I installed it:

wget http://www.condor-edv.com/~rubber/Download/openvpn_2_1_2-DNS323.tgz

funpkg -i openvpn_2_1_2-DNS323.tgz

After installing, attempting to start gave me “invalid module format”, with dmesg telling me that the tun module included in the package was compiled for a different kernel than the stock kernel on the DNS-321.

A bit of Googling told me that no one (I could find) had provided a tun module for the DNS-321, so I set out to create one. Having been a Mac convert from Linux for the last 4 or 5 years, I haven’t built a kernel in quite a while, but it proved to be like riding a bike. I won’t get into the dirty details but you need version 2.6.22.7 of the kernel, the .config from the DNS-321 kernel, and an hour or so to kill. “make config” still works and after answering the right questions I was able to “make modules”, install the tun module and configure my VPN.

Although the DNS-321 is not as popular as it once was, I hope this helps someone else (or future me). Here is a copy of the tun.ko built for DNS-321 firmware 1.03: tun.ko

Regardless of what Monty Python or Indiana Jones might have us believe, or not believe, there is no Holy Grail. In our case, the Holy Grail is the ability to run browser based Cucumber tests expediently. We’ve been chasing the Holy Grail for some time, and it ends today.

We started using Cucumber as we got more involved in the agile process. Unfortunately, like most other people, we were unable to catch our unicorn, and we never got our “customers” to write acceptance tests. Considering that we mainly do work for the government, I suppose we couldn’t expect to really have a lot of success getting them to write acceptance tests, but we subscribed to the process. The road was paved with good intentions.

How did it all go so wrong? As with many tools, we began to lean heavily on Cucumber. Our model, controller, and view specs started to be neglected, mainly as we spent serious amounts of time trying to get our javascript tests to run. Basically, very little of our applications will function without JavaScript.

The time we spent writing Cucumber features paled in comparison to the time we spent running Cucumber features. I previously blogged about my attempts to mitigate the issue. We now have another project that has hit 30 minutes per build. Obviously, we are in a unique situation and if you can keep your JavaScript tests down to <5 minutes, Cucumber + Selenium is probably fine and you should continue what you’re doing.

Recently, something snapped. Ben mentioned that his friend had rejected Cucumber and exclusively written view specs with graceful degradation. I respect the merits of doing that, but it can be difficult to sell the idea of building functionality in such a way that it will never be used, or specifically for testing purposes.

After a series of conversations, we’ve come up with a good direction. Our intention is to give more time to model and controller specs, use view specs or cucumber without selenium to test our views, and Jasmine to test our JavaScript. We’ll probably even mix in a bit more (GASP) manual testing, at least for the time being. With the time we save not writing javascript tests and running the full hour plus test suite, we’ll should have additional time to invest in researching other testing tools and strategies as well.

Ultimately, we’d love to increase our test coverage and validity, decrease our test running time, and as a by-product increase developer happiness. There’s nothing more demoralizing than finishing a feature, only to have to wait an hour or so to be able to push it and start on your next one. Ideally we’ll return to the days where running tests 50-100 times a day is not unrealistic. Right now, we’re lucky to run the full suite 5-6 times a day.

Cucumber, you’ve been good to us, and although you probably won’t be going away completely, it feels like the end of an era. Hopefully the next phase will bring faster builds and better tests. I’m happy we’re ending our quest for the Holy Grail, and although we’re not holding the Holy Grail in our hands, we’re more knowledgeable and ready to move on.

After reading Gruber’s article about Android and the iPhone, and having been long overdue to write my iPhone/Android article, today is the day.

Before we begin, I want you to understand very well where I stand on the Android/iPhone debate. Android will continue to gain market share, at the expense of Apple and RIM, and will easily be the market leader in the next year or two. There will be no debate. I firmly believe this.

I am heavily invested in both Apple and Google. I ran Windows until around 2000, ran Linux at home until ~2004, and have been exclusively Mac since sometime in 2004. Both my children have iPods, my wife and I both have iPhones and Macbook Pros, and I work on a iMac all day at work. I love the philosophy of Apple, and to be completely honest, I love their products. Short of me spending months to build something perfectly tailored to my flavor of the week, it’s the closest thing to perfection, for me. On the Google side, I use Gmail, Google Voice, Google Reader, Google Analytics, Google Groups, Google Maps, and probably other Google products daily. I have over 6 and a half years of personal e-mail on Gmail, have read over 220,000 items via Google Reader, use Google Analytics to track a number of personal and professional sites, Google Groups to manage my open source projects and football league, and use Google Maps countless times to make sure I get where I need to, when I need to.

With the release of the Nexus S, something really hit me. I watched the Nexus S commercial, and came away fairly unimpressed with the device. It reminded me a lot of the Samsung Galaxy S, actually. What did impress me, of course, was the apps. Not the 3rd party apps, but the Google apps. Google Voice, Google Maps, Google Navigator.

My concern as both platforms mature is that Google will continue to build and improve tools only for their platform as a “Product Differentiator”, and as a consumer on another platform (iPhone), Google Maps, Google Navigator, Google Reader, and Gmail are all good examples of this in varying ways.

If Google built a device like the iPhone: a non-plastic, non-cheap feeling device that didn’t require extensive configuration or tweaking to get it to a state where I’d want to use it, they might be a better candidate for my smart phone business. Maybe then my concerns will be quelled. On the other hand, Google is more a get it out there and iterate it kind of company, which typically means substandard initial releases of products. I don’t want to be your guinea pig when it comes to a $200 phone that will cost over $2000 over the life of the phone.

Are you addicted to Apple and Google? Do you read Google Reader via a Safari window on your iPhone? Do you use Mapquest4Mobile because Google refuses to release Google Navigator for iPhone? Do you use Google Priority Inbox watered down on your iPhone? How do others cope?

A few weeks back, I bought a DNS-321 with the idea of turning it into a sub $200 RAID 1 Time Machine and replacement for my Linux box that was serving Git repositories.

I purchased the DNS-321 for 60 bucks from TigerDirect with a $20 rebate, and managed to get two 1.5TB Western Digital 5400rpm hard drives from Micro Center for $60 a piece. The fun had just begun.

The first thing on the list was to “fun_plug” the DNS-321, which is incredibly trivial and allows you to get root on the device via telnet. “Hacking” the DNS-321 is not required to setup Time Machine, but is essential for installing and configuring Gitosis. Rooting the device is covered very well on the DNS 323 wiki.

Setting up Gitosis was a bit less trivial. First, you need to install Python, setuptools, and PERL from the fun_plug packages. Git depends on PERL, and Gitosis depends on Python and setuptools. You can get Python and setuptools from plord’s repository.

Once the dependencies are installed, you need to fetch git using wget:

wget http://kernel.org/pub/software/scm/git/git-1.7.3.4.tar.bz2

After decompressing, git must be configured to install into /ffp, so run:

./configure --prefix=/ffp

After configuring, git must be built with NO_NSEC because of the version of libc used by the DNS-321, so run:

NO_NSEC=YesPlease make install 

Once git has been installed, which takes a while, you need to clone gitosis:

git clone git://github.com/res0nat0r/gitosis.git

You can pretty much follow the directions at scie.nti.st, with a few caveats:

When adding a user, you need to use the syntax appropriate for BusyBox’s adduser:

adduser -S -g 'git version control' -D -h /mnt/HD_a2/git

After this user is added, you need to run:

store-passwd.sh

to permanently store the git user.

Make sure that you create git’s home directory on the NAS and set the appropriate privileges. You don’t want to accidentally delete the directory when you’re browsing the NAS one day.

The last tweak I had to make was an unfortunate one, but I couldn’t think of a more reasonable way of doing it. I had to disable StrictModes in OpenSSH, because the git user’s home directory is not located in /home. I tried mounting /mnt/HD_a2/git to /home/git or symlinking /mnt/HD_a2/git to /home/git, but neither of those worked. This can be done via editing /ffp/etc/ssh/sshd_config and adding:

StrictModes no

Once you’re done with that remember to restart OpenSSH and you should be good to go. I hope you enjoy your RAID 1 Gitosis repository on your DNS-321 as much as I am enjoying mine. After updating this site to Ruby 1.9 and Rails 3, I began deploying exclusively from my DNS-321.

Webrat or Capybara? With thousands of steps, it’s not a question you ask lightly. We’ve been using Webrat for a long time, but it definitely has it’s issues. It’s dependence on selenium RC to run in Selenium mode being one of them. How many times have you copied the selenium-server.jar into Webrat so that you can run your tests on Firefox 3.6? Also, it’s ability to see and click hidden elements, a lack of case sensitivity (until recently), and other annoying little things. Don’t even get me started about having to run Selenium and non-standard separately.

A few months back, we started pondering the idea of using Capybara instead of Webrat. We even decided to start a new project with Capybara instead of Webrat; that way, if we had to convert it to Webrat because of problems, it would be less work. As luck would have it, at the time I was testing parallel_tests with Webrat and doing some hacking to see if I could get the two to play nicely. Unfortunately, I wasn’t successful. After a few weeks of successfully using Capybara on the other project and my failed attempt to get parallel_tests to work with Webrat in selenium mode, I decided that something needed to be done.

I started the tedious process of converting all of the steps in one of my projects to Capybara, and removing custom steps by adding pickle. I had used pickle in a few pet projects, and found that for creating test data and limiting the amounts of custom paths, it standardized and greatly simplified the process. It took a few weeks of slowly changing in my downtime, rebasing against master, resolving merge conflicts, and converting some more.

I also decided that resurrecting parallel_tests needed to be done; we were spending a significant amount of time trying to optimize one of our projects build times, which had ballooned to around 45 minutes. Through some clever rework by some of my co-workers, they managed to get the build down to the mid 30 minutes.

After creating a small sandbox, I was able to get Capybara running on parallel_tests with very little effort. Although the results were not astronomical, I was able to save approximately 30-35% on my dual core laptop. With our build machine being quad core, I could only imagine the time savings.

The end result of running the old and new features on the build machine:

• Non-parallel (build machine): 11m27.796s

• Dual core parallel_tests: 8m28.36s

• Quad core parallel_tests (build machine): 4m8.45s

So, I succeeded in shaving off 3 minutes locally by running parallel_tests, and 7 minutes off my build by hacking up Capybara, and here’s the hack:

class Capybara::Server

private

  def find_available_port

    @port = 9887 + ENV['TEST_ENV_NUMBER'].to_i

    @port += 1 while is_port_open?(@port) and not is_running_on_port?(@port)

  end

end

Drop this into a file in features/support, setup parallel_tests as specified and enjoy faster Selenium features. You may also want to create a rake task to run your tests in parallel:

namespace :ci do

  task :parallel_all do

    RAILS_ENV = 'test'

    Rake::Task['log:clear'].invoke

    Rake::Task['setup:app'].invoke

    Rake::Task['parallel:migrate'].invoke

    Rake::Task['spec'].invoke

    Rake::Task['parallel:features'].invoke

  end

end

We’re slowly working on converting the project that’s build takes over 30 minutes to Capybara, and will report our results when we’re completed. We’re shooting for under 10 minutes.

A new design and now a new post. I’m on fire!

I’ve been spending a little more time lately optimizing our build times, and ran across a useful snippet I’d written a few months ago.

We have a few projects with over 4000 Selenium features (we love that Javascript!), and every once in a while we run into a few false positives. Features will run slow on the build machine and cause the build to fail, even though it shouldn’t.

Cucumber supports a “rerun” format, where it will output a rerun.txt file of all the files that failed. You can push that rerun.txt back through Cucumber to just rerun those features.

Here’s the code from ci.rake:

namespace :ci do

  task :all do

    rerun = "#{Rails.root}/rerun.txt"

    RAILS_ENV = 'test'

    Rake::Task['log:clear'].invoke

    Rake::Task['setup:app'].invoke

    Rake::Task['spec'].invoke

    FileUtils.rm_f rerun if File.exist?(rerun)

    begin

      Rake::Task['ci:features'].invoke

    rescue Exception => e

      if File.exist?(rerun)

        rerun_features = IO.read(rerun)

        Cucumber::Rake::Task.new(:rerun_selenium_features) do |t|

          t.cucumber_opts = "--format progress -q #{IO.read("#{Rails.root}/rerun.txt")}"

        end

        Rake::Task['rerun_selenium_features'].invoke unless rerun_features.to_s.strip.empty?

      end

    end

  end

  Cucumber::Rake::Task.new(:features) do |t|

    t.cucumber_opts = "-q --format rerun --out #{Rails.root}/rerun.txt"

  end

end

It’s important not to blindly use this task; if you continue to run into the same feature failing over and over again, it could be another issue. We’ve seen instances where stories will set some persistent data in the browser’s local storage, and later stories will fail unless run in isolation. Make sure you’re looking at your build output regularly!

I pushed a new repository to Github and a new gem to Gemcutter. It’s called Seed.

Seed is (another) gem designed to deal with seeding databases.

The main advantage of using Seed is that when creating a seed, or retrieving a seed, an error is throw if you attempt to overwrite a seed or retrieve a non-existent seed. The goal is to make you care about your seed data, and ensure that you’re not overwriting existing seeds or trying to make an association between a seed and a non-existent seed.

How does it work?

Role.seed(:system_administrator, :role => 'system_administrator')

User.seed(:jdurham, :login => 'jdurham', :roles => [Role.seed(:system_administrator])

if you like blocks, we’ve also got those covered:

Role.seed(:system_administrator) do |role|

  role.role = 'system_administrator'

end

User.seed(:jdurham) do |user|

  user.login = 'jdurham'

  user.roles = [Role.seed(:system_administrator)]

end

This will create a Role seed, named “system_administrator”, which we will reused in other seeds. That way, when looking at another seed, it’s very obvious what associated data it has.

Overplanting not allowed

Later on, if I decide I want to create a new seed and I do:

Role.seed(:system_administrator, :role => 'admin')

This will raise an exception, because there will be an ambiguity issue. I’m trying to create another seed named system_administrator.

Trying to get a seed that never existed

User.seed(:normal_user, :roles => [Role.seed(:user)])

This will raise an exception because it doesn’t make sense to try to associate a user with a seed that doesn’t exist.

Great, I’ve got seed data. Now, how do I load it?

Seed comes with a rake task that extends the built-in rake db:seed functionality. In your Rakefile add:

require 'seed/task'

Now, create a db/seeds directory and add a file; for example, development.rb to that directory. Once you do, you’ll automatically have a rake task that can be invoked as follows:

rake db:seed:development

Hopefully you find Seed useful. We’ve been using it in production for a little while and it’s been working well.

This will allow you to do rake db:seed:x where “x” is a file in the db/seeds directory:

namespace :db do

  namespace :seed do

    Dir[File.join(Rails.root, 'db', 'seeds', '*.rb')].each do |seed_file|

      task_name = File.basename(seed_file, '.rb').intern    

      desc "Load the seed data from db/seeds/#{task_name}.rb"

      task task_name => :environment do

        load(seed_file) if File.exist?(seed_file)

      end

    end

  end

end

Last week was R&D week at Beacon, meaning that developers got to work on a project of their choice to increase their understanding of something they may not normally be able to work on.

While doing my R&D week I decided that now would be a good time to learn Emacs. Emacs has been on my list for the last seven or eight years, but never made it to the top.

After a week of Emacs, here’s some of the things that I like:

• Split screen with script/server running and editing. Feels very functional and is a great use of space.

• Unlimited extensions: Magit, Ido, Multi-term, twitter, iBuffer

• Keyboard shortcuts for everything

• The tab key. Emacs really “gets” the tab key. I can tab in the middle of the line and it will correctly tab the beginning of the line, so reformatting is a dream.

• Email, Twitter, RSS, Terminals, etc integrated into one “IDE”

Things I’m trying to get used to or otherwise don’t like:

• Keyboard shortcuts for everything. I want to make sure I’m being productive vs trying to remember as many shortcuts as I can.

• Lack of a beautiful UI

• With infinite customization comes loss of cohesion. I can’t sit at another developer’s desk who uses Emacs and use my shortcuts.

• Why is it so hard to duplicate a line out of the box? In vim I could do “y-y-p”. In Emacs the best I’ve got is C-a C-k C-y C-y, and that’s only if (setq kill-whole-line t) is set. Really?

My last editor change was from vim to Textmate, and though I really miss some vim niceties in Textmate, overall I’ve been fairly happy. I plan to use Emacs for the next few weeks or so, then make a decision at that point.

So, Emacs or VI? Textmate? BBEdit? What’s your favorite editor, and why?

This is the best way I’ve seen so far to get Rails 2.3.5 working with Bundler 0.9.x:

Create config/preinitializer.rb with the following code:

begin

  # Require the preresolved locked set of gems.

  require File.expand_path('../../.bundle/environment', __FILE__)

rescue LoadError => e

  raise RuntimeError, "Please run `bundle lock` and try again."

  # Fallback on doing the resolve at runtime.

  require "rubygems"

  require "bundler"

  Bundler.setup

end

Update config/boot.rb and add the following before Rails.boot!

class Rails::Boot

  def run

    load_initializer

    extend_environment

    Rails::Initializer.run(:set_load_path)

  end

  def extend_environment

    Rails::Initializer.class_eval do

      old_load = instance_method(:load_environment)

      define_method(:load_environment) do

        Bundler.require :default, Rails.env

        old_load.bind(self).call

      end

    end

  end

end

Now, just run bundle install and bundle lock and you’re good to go.