Some thoughts on designing a test framework with Ruby

Here are some of my thoughts after actively working on a test framework in Ruby and scripting and running test scripts for the last couple of months.

Start writing tests first, and then grow your framework. I tried to design a framework before I had a solid idea of what the testing needs were. I had just joined the company, and didn't really understand the use cases and test procedures very well. So, I just made assumptions of what I thought would be good common code and what should be in a "good" test framework. These assumptions came from other test frameworks I had used or designed in the past. I felt like my experience would make up for my lack of product knowledge. In a lot of cases that was true, but I could have delved more into the testing side, instead of just the framework. Sometimes to test something, you just need a very simple script, and a sophisticated framework can just get in the way.

Don't overdesign features before you really need them. For example, I'm thinking of ripping out the logging stuff I implemented, because now it is annoying me. I had designed the test framework so that when you include the test harness class in the test class, a log will be made for any test run. Well, I found out I wasn't really reading the logs too much, I was just relying on the output from Ruby's Test::Unit class. And I hadn't implemented a cleanup mechanism to clear the logs after a certain age was reached. So, I'm constantly typing "rm -f *.log". I think it is wiser to not force the logging in a test class, and have the test developer decide what logging is necessary and the mechanism to do that.

The DRY (Don't Repeat Yourself) concept is great, but don't get too fancy, though, with code reuse and that magical metaprogramming (if you are using a dynamic language like Ruby, as I am). It probably doesn't matter much when you need to quickly crank out tests, since this is just testware, not shippable code. In most cases, I think repeating yourself a couple of times is okay, especially when you are trying to automate as much of your tests by the end of a sprint. But once you get to the magical number of 3, then you should think about your modularity and move your repeated code to a common library.

Don't get hung up about trying to implement tests in only one language. This is probably a good reason to not spend too much time working on a framework, because, you might need a different language or tool to perform a subset of your tests. For example, there are some things that Java is better at than Ruby and vice versa. When it comes to manipulating large amounts of XML, performance-wise, Java would be my choice over Ruby.