Recently while watching my Twitter feed I ran across this:

testobsessed: AACK! RT @dhemery I recently heard of an Agile team accused of cheating for running tests of their own before handing off to QA group.

There is so much wrong with this.  And this was shortly after a similar tweet from Elisabeth:

.@ andrew_paradigm Srsly? You’ve seen Test Mgrs shun FitNesse tests to increase bugs for sys tstrs to find? OMG. Malpractice. Wow. I’m sorry.

I ranted a bit as a result. And ranted again.  And Elisabeth patiently asked, “What can we do to  fix this?”

And I was stuck.  I’ve had a number of test leadership roles in three different companies.  So my breadth of environment experience isn’t huge, but it isn’t super narrow. I’ve never run into anyone who put their turf that far ahead of the quality of the product. 

While I don’t have a reason to doubt Dale or Andrew, I’m pretty sure an exchange like this on twitter would be considered hearsay in a court of law.  Never the less, I can imagine it happening. 

Back to the question Elisabeth asked, “What can we do to fix it?”  I don’t know if I know how to fix it, but I have some thoughts on how to challenge these Test Managers.  And I’m going to use some classic QA/Testing concepts to do it.

Cost of Quality

First off, the classic cost of quality graphic.  This is the classic graph that we’ve seen in testing for a really long time now.  The dollar values you assign at each level can change, but the message is always the same.  The cost of removing a defect goes up as the product moves through the process.  Defects discovered in requirements are cheaper to fix than defects discovered in coding.  Defects discovered when coding are cheaper to fix than the ones found in system test.  And the defects removed in system test are cheaper than those discovered in production.  The slope is significant and the cost gets pretty crazy. 

Agile is all about the cost of quality.  The time between requirements, coding and testing is minimized.  The customer is involved more and the process is focused on building the right thing all the time.  TDD, Agile Testing, Exploratory Testing, Executable specifications; all of them are focused on improving the quality of the product and finding problems earlier.  The QA group in both tweets is working at a higher cost of quality than the agile team.  It doesn’t make them less valuable, but it does mean that they should want fewer bugs to make it to them to test.  This means the project will cost less and the customer will be happier with the product.

Why Do I Always Have to Unit Test the Developers Code?

I can’t tell you how many times I’ve found testers complaining about the code being “thrown over the wall” or “didn’t they even try this?” So often testers complain about the what they perceive as low quality code.  They are frustrated that they can’t even get at the real bugs because they have to wade through all of these little bugs.  I’ll be willing to bet that those same Test Managers have commiserated with their team about the quality of the code that they were getting before the team went Agile.

The Agile team is most likely removing the issues that the testers would find right away, and complain about.  Sure those defects would have made their defect discovery numbers higher, but were keeping those testers from getting to something more sinister that could have made it past the team.  Sure that hidden bug might have been more work, but that’s the job.

Defect Removal Effectiveness Metric

What QA/Test Manager in the 90s didn’t at some time pick up a copy of Metrics and Models in Software Quality Engineering by Steven H. Kan?  I had one (two jobs ago).  One of the metrics he discusses is Defect Remove Effectiveness.  The idea behind the metric is that you should be able to measure the number of issues identified at each phase of development.  This will let you understand what portion of the defects are removed at each level.  Knowing this lets you understand where you need to focus your efforts and which processes are the best at eliminating problems.  This is classic process improvement.  Measure what you’re doing and see where you need to improve.

This last one is the hardest and maybe is me going a step to far.  It’s a hard metric to get, unless you track everything in agile or in waterfall.  That much tracking and measurement isn’t actually very agile and wouldn’t go over very well with the agilists.  The catch is, I think those test managers wouldn’t like this either.  It might point out that they aren’t that effective.  I might point out that they need to evolve.

Where Does This Leave Us?

I don’t know. 

I doubt that my thoughts here are going to be enough to change the minds of the Test Managers Dale and Andrew talk about in their tweets.  So often when people go into defensive mode, they aren’t going to listen to a reasoned debate. At the same time, I don’t know that these folks cannot be convinced.  Maybe they are up for the challenge of finding those tricky bugs that make it through unit tests, TDD, executable specifications, and agile testers.  Maybe not.

I do know this.  I work at a place where we are becoming more agile (and the small ‘a’ is intentional).  We’ve got a ways to go, but we’re on a good path.  We still have what we call the “Traditional QA Team” who are outside of development and we have our Dev Test team that is integrated into development (as are our BAs).  Their manager and I get along really well.  We’re talking and trying to figure out how to make the whole process more effective.  He’s never once told me that we were “cheating”.  He’s never once told me that we removing too many bugs. 

Of course, I did say we’ve still got a ways to go, didn’t I?

I’ve been working with FitNesse for over a year now.  I’m on my second incarnation of fixtures, and now my second incarnation of a Hudson implementation.  Technically, it is the third, but I’ll get to that.  I thought I might share a little bit about this evolution and the final result.

Incarnation One

While I was getting started with my Fit based fixtures (incarnation one of my fixtures), one of the development leads I work with was working to make Hudson our CI tool.  He decided to help out and had a team member write a Hudson plug-in for it.  Unfortunately I wasn’t ready for this and so it sort of got abandoned.  When time came that I really needed to get FitNesse working under Hudson, I partly forgot about it and was partly hoping for an official plug-in that was mentioned on the FitNesse yahoo group.  So I sort of skipped it and went looking.

Incarnation Two

So with my goal to get FitNesse running under Hudson, I went searching and found two things that set me on the path.

1. Showing FitNesse Test Results in Hudson
2. A post by Gregor Gramlich which had an xslt for translating the XML result from test runner to HTML (sorry, can’t find the post any more)

This was great.  A few tweaks to the ant task from Andy Palmer, and I was up and running.  Well, there was some work in Gant to get our stuff running under Hudson due to some environment configuration stuff we need, but otherwise it was working.  I was getting ready to start moving this into use when…FitNesse updated with some changes I needed, and it broke.  It was a minor break, so I tweaked the ant script again to launch FitNesse via a jar, rather than the class.

Incarnation Three

So i was all set.  Things were working great and I was getting both HTML page results and JUnit style results in my local Hudson Instance.  It was great.  It worked just fine, until I ran a lot of tests.  I kept getting Java Heap errors when I ran my whole suite.  No problem when I ran a test or two, but run them all and boom.  I started investigating and I found the memory was blowing out in the same place every time.  It was always blowing up as new results streamed back from the server were appended to the string that was eventually going to be saved out.

I contacted Uncle Bob and he was having the same problem.  Due to the addition of Slim, the amount of data being sent back when a suite runs increased significantly.   This was causing the memory problems.  Uncle Bob thought about it and decided that he needed to trim the fat in the protocol.  So he changed the XML that is returned to the TestRunner to be simpler, smaller and more efficient. 

So here I was, I had an approach for running the tests.  They ran.  They returned results without blowing things up.  But Something was missing.  The XML transform from Gregor still ran, but the output was all wrong; it was missing the HTML version of the page results.  To make things more efficient, Uncle Bob had to stop returning the content, as the page content was the source of all our memory problems.  This would have been a real problem, but Uncle Bob didn’t do this without a backup plan.  Earlier this summer, he added the ability to keep the history of a test run and view it.  Leveraging that work, he now returned the path to the history for the specific test run of each page in that execution.

Now I had to do some thinking.  I have the results and I can transform them into JUnit results for Hudson.  But my boss wants to see the HTML pages.  So I think about this.  I have an XML file that has the page address to the history.  It’s not a full URL, just a page reference with a link to the history responder.  So what do I do?  What if I take the XSL transform and modify it.  So I did some quick Googling and tinkering and I ended up with some changes to the XSL file to take the Page reference, build it into a proper HTML link and change the “format=xml” to be “format=html”.  Now I had a HTML page and links.  (I’ll post this file very soon.  Waiting on permission, as I modified the original.)

However, how do I have a persistent FitNesse server running on my Hudson server?  I probably over thought this, but it seemed to me that I needed to have an instance of FitNesse running as a service on the computer just to serve up the history pages when someone goes exploring a failure from Hudson.  I’ll post my version of this for everyone soon in a FitNesse Tip very soon.

Today all I have the energy for is the story.  Very soon I’ll post information on the following things:

1. The XSL transform that I got to work.
2. An ANT task that runs FitNesse and transforms the XML results TestRunner saves into an HTML page
3. My version of the trick of getting FitNesse running as a service on Windows.

Recently I attended the 2009 AA-FTT workshop.  While I didn’t get to go to the Agile 2009 conference, I enjoyed an opportunity to talk about the tools and techniques people have used to be more successful in their efforts to be Agile (big A or small a).

One of the sessions I attended was Advanced FitNesse Techniques.  Well, it quickly evolved into a different session focused on Tips and Tricks.  There were two of us who had been doing FitNesse for some time and two folks who were just starting out or evaluating it as an option.  Adam Goucher recounted this recently on his blog and summed up a majority of what we discussed (see his post: http://adam.goucher.ca/?p=1153).

So I’m not an expert at FitNesse, merely someone who has been a practitioner and champion in one context.  What works for my project won’t necessarily work for you. So your mileage may vary.

I’m hoping to share with folks a number thoughts, tips and tricks that I’ve found in the past year.  This first post will be a few quick things.  In the later posts, I plan to take more time with a single topic.

So on to some quick tips:

• People often complain about the .zip files under their pages.  These zip files are the rudimentary version control built into the system.  if you add a “ –e 0” argument to the command line when you launch FitNesse, the zip files will go away.  Alternately you can leave them there and just ignore them in your source control (you are using real source control aren’t you?).
• If you do want to make sure that the those zip files never appear, or that the FitNesse server launches on a available port all of the time, create your own .bat file, alias, or shell script. Make sure it launches the way you want it to.
• If you are in a fixture, learn how to rely on Graceful Names and use the automatic behavior to fix the words.  FitNesse will remove the whitespace and let you write more naturally.
• Use the output. In Java you can use System.out.println() to add to the output page.  Log what you are doing .  Output debug information. I’m not sure what happens in other languages under Slim, so this might work differently there.
• Comment out big sections of your tests easily.  } are a powerful tool to comment out big sections of your tests to let you debug your both your product and your tests.
• Create your own styles.  Use the !style_name widget to create your own styles for inserting comments or warnings into your tests.  Add them to the fitnesse.css file and check that file into source control with your code (remember the source control).

It’s a rather random list, but just a start.  In the coming days and weeks I’m going to try to explore a number of different topics in more depth.

So, I’ve been using a range of open source tools for a long time now.  Vim, Eclipse, FitNesse, FreeMind, TuxPaint (for the kids), and even a little bit of Ubuntu.  I haven’t really contributed much back to the communities I’ve particpated in as a consumer.

Today I put forth my first child into the world.  I created a syntax coloring script for Vim/gVim.  It’s still pretty  rough.  I have some more slim and Fit/FitNesse syntax to support.  I also should add some FitLibrary support, even though I don’t use that.

If you love FitNesse, and you use Vim, it’s a natural.  So far that isn’t too many people.  As of this posting, only 5 people have downloaded it.  And none of those are a vanity download.

If you are interested, you can find it here: http://www.vim.org/scripts/script.php?script_id=2651.

WikiWidgets in FitNesse are a great way to extend the functionality of your FitNesse pages. Essentially they let you rewrite the content of you page in creative ways. My first forays into the creation of my own wiki widgets was to add some specially styled text to make things stand out on the page. This has been replaced by the new !style_class widget added by Uncle Bob a few releases back.

That doesn’t mean there isn’t a need for a widget now and then. I wanted to come up with a way to connect my wiki pages to our JIRA defect tracking and workflow system at work. I wanted it to be easy for me and my team to be able to do this with the smallest amount of wiki text. So I created a Widget for it !JIRA{TEAM-11111} which gets transformed into a proper link to the JIRA system.

I’m putting this out here for anyone who wants to reuse it. To make it work in your environment, you will need update the URL to match your environment. It can be further modified to work with any defect tracking system that supports URL as a way to access the system. And if you aren’t using JIRA, and many of you probably are not, you can just rename the Widget as well.

package common.wikitext.widgets; 
import java.util.regex.Matcher; 
import java.util.regex.Pattern; 
import fitnesse.wikitext.WikiWidget; 
import fitnesse.wikitext.widgets.ParentWidget; 

public class JiraWikiWidget extends WikiWidget { 

    public static final String REGEXP = "!JIRA(\\{\\w+.*?\\})?"; 
    public static final Pattern pattern = Pattern.compile(REGEXP); 
    private String originalText = ""; 
    private String token = null; 

    public RpJiraWikiWidget(ParentWidget parent, String text) { 
        super(parent); 
        originalText = text; 
        Matcher match = pattern.matcher(text); 
        if (match.find()) { 
            token = match.group(1); 
        } 
    } 

    private String formatTestIdentifier(String text) { 
        String tempText = removeBraces(text); 
        return "<a class=\"dts\" " + "href=\"http://jiraserver/jira/browse/" + tempText + "\">" + tempText + "</a>"; 
    } 

    private String removeBraces(String format) { 
        return format.replaceAll("\\{", "").replaceAll("\\}", ""); 
    } 

    public String render() throws Exception { 
        return (token != null) ? formatTestIdentifier(token) : originalText; 
    } 
}

It’s not the trickiest code in the world. But it’s yours to take and to use.

So far this is a small contribution to the community. At some point in the not too distant future I’ll share my vim syntax coloring for FitNesse wiki pages. I still am looking to make a couple of improvements before that goes out.

Comments and questions are welcome.