Every year MSOE, one of the local engineering schools, holds a testing competition for their Software Engineering students. It has been going on for six years now and I don’t think I’ve missed a year yet.

The event is called Bugstomp. Students are invited to compete at testing programs selected by faculty members. Generally these are open source projects that the faculty have modified. They inject some additional bugs to keep things interesting. After all, open source doesn’t mean buggy by nature.

The school invites local software professionals to do the judging. Typically it is a collection of testers, developers and an occasional project manager. I sign up every year. It is very important to me to show my interest in the profession. And while I didn’t go to MSOE, i think they have a good program and I want to support it.

The Competition

The are two rounds of competition. In between, we take a break for the traditional food service taco bar.

The students have no idea what they are going to test and get no requirements document or other reference material. In other words, they are pretty much forced into exploratory testing.

Round One

The first round is usually an application that has a purpose. If I recall correctly, last year’s round one app was an SVG editor. This year it was PNotes, a Post-It Note type application.

It was a dizzying program to start with. Full of options. Too many options. PNotes runs as a system tray application and when you right click on it, the menu comes three-quarters of the way up the screen. This is one example of the biggest issues, usability. In addition to the large menu, the user interface tends to be non-intuitive. While the students didn’t report too many pure usability bugs, the issues with usability did lead to a few reports.

A word about the judging process. The process has evolved over the years. It started out a bit too simple and was pretty easy to game. The current process as the students logging into a specially built defect tracking system that queues defects to the judges. We then judge the bug based on quality of the defect report and severity of the problem they find. These two values are multiplied together. Non-reproducable bugs or really incomplete bug reports are can get a score of zero. So, the better the report and the more significant a bug, the higher the score.

When I judge, I tend to be a bit of a harsh critic. While I know that the students aren’t professional testers, I want them to think about what they are doing and I just don’t let poor reports go by. So if it is a mediocre write up or a trivial bug, it won’t get a lot of points. We do get to provide feedback related to our scores, so that the students can improve.

Back to PNotes. The students did a pretty good job finding some very interesting bugs. A lot of them were small ones. The biggest one was an injected bug related to creating a duplicate note. Doing so created multiple windows, more than expected. Lots of folks found this. Most of the students didn’t take it to the next level. Just in their reach was a crash that caused data loss.

I have to find some time to play with the app some more. I only got a little while before the judging started and then it was all I could do to verify bugs.

Round Two

The second round is usually a game of some sort. Last year it was a three-dimensional vision of Minesweeper. This year was a version of Space Invaders.

Games are a different sort of beast to test. And this game was a treat. First off, I could barely run it. It turns out that the game and some pretty significant memory issues related to running on a 64-bit JVM. It crashed on me constantly, so that made reproducing some of the students reports difficult. I finally forced it to use a 32-bit JVM and that got things more stable.

The students found a number of the seeded issues and reported things well. It’s a lot of fun trying to reproduce issues in a game. It’s also really hard sometimes. What can I say, college students generally have better hand-eye coordination than I. Bugs came in bursts, and every so often I got caught up.

At one point, one of judges got Rick-rolled by a student. The student reported a bug claiming that the game needed a mute-option, as it got in the way of him listening to a video. Of course, The link pointed to the infamous music video. I am proud to say that the student in question is an intern at my company. Though if I had gotten that bug, I would have figured out how to give him a negative result.

In Summary

The only downside of the experience is that we only get a few minutes to talk about the reports and give guidance. We don’t get to do much more help than that. The situation doesn’t provide for much more.

It was a very fun day. I really enjoy coming each year to participate. I enjoy also getting to see the other returning judges; other members of our craft. I look forward to doing it again in the future.