Stop giving the power to the big media companies and the other corporate beasts.  Vote.

Day One is probably my favorite app on my iOS devices and on my Mac. It is a wonderful little journal that is changing the way I think about my day and how I capture my thoughts. How is it doing this. Well here are a few ways.

The first, and most obvious, is that I am writing about my day, every day. It’s great. Deciding to have a daily journal makes me stop and think about my day and the things that went on. I capture little things that make me happy or that I think are important. I get notes about my kids, though I know I miss more than 70% of the cute things they say or do, but anything is better than nothing. I get notes about what we did and where we went. Sure, I leave stuff out, but capturing anything is better than nothing. It even let’s you have pictures, weather and geolocation data attached to your entries, so you can know more about what the day was like and what you were doing.

The second way it is helping me is as a safe place to put the thoughts that I might have been tempted to share on twitter or Facebook, but that are too boring, too personal, or to partisan. I don’t really want to share everything that comes to mind, but it don’t want to lose it either. And I don’t really want to get into any sort of political or philosophical arguments with people via social media. If I’m going to put my foot in my mouth, I prefer to do it with my civilized friends who will nicely handle the fact we disagree.

Don’t think that this will save you from my pictures of Legos, beer, or my perfectly roasted chickens. You’re stuck with them.

Third, I’ve implemented Slogger, a creation by the mad scripting scientist known as Brett Terpstra. Slogger let’s me capture my activities on various social media sites like Twitter, Instagram, Instapaper, pinboard and my own blog entries. All of my tweets, starred reads in Instapaper, photos and blog posts get captured and logged as apart of my day. This way I can review the events of my day and the things I’ve done in those services.

Fourth, I capture notes about my day at work. So I know what I did and the things that I found important or interesting. This helps me track how things in my professional life intersect with my private life.

I like Facebook for simple sharing with friends and family, but I still try to keep it simple there. I enjoy twitter, but always know it s public. Day One gives me a place to track my life in a private and contained way, and I love it.

I’ve look at other social networks, such as Path and EveryMe as ways to do a smaller, more private social network. But I think those miss the point a little. Facebook is a very effective way of keeping in touch with folks. Yes, it’s never as private as we would like, but as long as you keep that in mind and share what you wouldn’t mind if it got out, it can be really helpful. If something is private, then it should be private and shouldn’t be in a social network at all. That is where Day One comes in. Record it. Keep it. But don’t over share it.

By the way. I’m going to try to post more, so you might see stuff that isn’t as testing oriented. I still hope you will enjoy and read on.

iOS Test Automation Comparission

For the past couple of weeks I’ve been diving into iOS testing. While I would not describe my investigations as exhaustive (one might even call it a shallow dive), I have learned a few things that I think are worth sharing about a handful of the options out there.

Before I get started with that, I should share a little bit about how I started this process. I like to think of my self as being capable of most anything on a computer. Ok, maybe I should say that I’m willing to try most things. Anyways, I decided that it was important for me to understand the inner workings of things. While I probably will never get a job writing iOS apps, knowing how to connect controls up and make them accessible to automation would be very useful. So I downloaded Xcode and worked my way through the first couple lectures of the Stanford course on iOS development. After doing the first couple of homework assignments, I had what I wanted: an app to test.

Now, it is a very simple app, but one that I could modify at will to make sure it works with automation. I am sure that a more complex app would introduce things that might make one tool or another a better choice.

The Playing Field

There are a few flavors of test automation tools available for testing iOS apps. So far these fit into the following categories:

• UIAutomation via Apple Instruments
• Automation By Wire
• Jailbreak Dependent

UIAutomation via Apple Instruments

Instruments is a tool that comes with Xcode and is used in the process of testing and profiling iOS and OS X applications. UIAutomation runs via Instruments and can execute tests both on the device and in the iOS Simulator.

Automation By Wire

While not an official term, the name seems to fit. Automation By Wire identifies a category of test tool that injects/adds some code to the application and then allows software running on a computer to connect to a server in the app and then run tests against said app.

Worth noting about the Automation By Wire approach:

• Relies on linking a library or building code into the application
• This code can be injected using one build target and the production software can be built with a different target
• The change in binaries between test automation and the app store could be a risk, though it is probably really small

Jailbreak Dependent

One option for being able to interact with and control the app is to Jailbreak the iOS device. Jailbreaking allows the Test Tool to install an application onto the device that allows for interaction via violating the normal security model of iOS. This software is then controlled by remote calls from a computer where the test is run.

I believe that jailbreaking is safe for this purpose. I haven’t done it myself, but i know folks who have. The big concern i would have is if Apple released a version of iOS and the Jailbreak community is unable to hack it for an extended period of time, we would be unable to run our tests. It is unlikely that this would continue for a long time, as the community is pretty fast at getting new versions jailbroken. Never the less, it is worth noting the risk. Your comfort with that risk will guide your decisions.

Commercial Tools

Of course, commercial tools exist for mobile app testing. Most of these tools are relatively new to the market in comparison to their desktop counterparts. My research hasn’t identified a clear leader in this category. Though all of the major players appear to have something.

It seems that most appear to function using the Automation By Wire approach or by Jailbreaking. They inject/add code to the app that allows them to observe the app and make it testable. Other tools claim to be able to record the execution of tests on the device via the USB port and run those tests through the same mechanism (supposedly without jailbreaking).

Challenges and Caveats to iOS Testing

The following challenges have been identified:

• Design for testablilty is as important in iOS applications as it is in any software product. An iOS App can be made in such a way that it is very difficult, or almost imposssible for tools to interact with the app consistently.
• All of these tools when pitted against a basic or standard iOS app can be used without significant extension. Equally so, all of these tools will require additional effort when the application under test is designed in such a way that it makes testing difficult or uses custom controls.
• All of the tools for testing on iOS and Android are much less mature than those for desktop systems.

Tool Notes

Instruments (Apple Official)

Instruments is a collection of tools that allow developers (and testers) to monitor and test their iOS application. Specifically to test an application, the user would choose the Automator tool and would then write automated tests in Javascript. These tests can be run on the device or in the simulator. Additionally, you can analyze memory, cpu and other factors while running your application.

Instruments has no specific parallel on Android or in other platforms. It would be a unique running and authoring experience.

Testability Note: While investigating Instruments, I discovered that UILabel controls are not a good choice for updating text that you might want to check in a test. There is a known issue with them, as sited in this O’Reilley post: How to use UIAutomation to create iPhone UI tests - O’Reilly Answers. I modified my app based on this to use UITextField controls that had been disabled. I also added Accessibility Labels to my buttons to make them easier to work with (something that all tools benefit from).

Additional Notes

• Uses Javascript as the programming language
• Supports suspending and resuming application via home button
• Supports running memory and cpu utilization logging in sync with test execution
• Supports capture/playback
• While it looks like it is getting easier to do, there are challenges running it in CI as Apple does not seem to provide ways that are consistent with other development environments
• Is extendable within it’s own tooling using libraries like tune_up.js which provide assertions and a test declaration format. Tuneup_js also appears to have added a command line runner through a ruby script.
• Tool does not indicate when javascript is invalid (may require other tools to check for obvious syntax issues). In fact, the tool is user hostile when there are javascript errors.
• Logs are both verbose and lacking at the same time

Frank-Cucumber (Open Source)

Frank-Cucumber is a ruby toolset for automating iOS app tests. It uses a BDD (Behavior Driven Development) style of test authoring. This style is focused on developing the tests in the language of the business and then allowing the developers and testers to separately write the wiring software that connects the words to the invocation and checking of the system. Cucumber is a style that is growing in popularity and there are several books now available on the topic. Personally, I have known about Cucumber for some time now, but have not had a reason to try it. The experience is positive, but I should save that for another post.

    Feature: Simple Multiplication
    
    Scenario: Multiplying Two Numbers
    Given I launch the app
    Then I should see a Result of 0
    
    When I enter 75
    And I enter 3
    And I tap *
    Then I should see a Result of 225
    
    Scenario: Multiplying Three Numbers
    When I enter 3
    And I enter 4
    And I enter 5
    And I tap *
    Then I should see a Result of 20
    And I tap *
    Then I should see a Result of 60
    
    Scenario: Multiplying Three Numbers in a different order
    When I enter 3
    And I enter 4
    And I tap *
    Then I should see a Result of 12
    And I enter 5
    And I tap *
    Then I should see a Result of 60

The Given-When-Then style of tests used in Cucumber follow the concept that steps using “given”, “when” and “and” are all steps that drive the application to the point you want to check. The “then” steps are used to check the state of the software. Step descriptions are matched against the sentences and phrases after the given, when, and, then keywords. Developers and/or Test Engineers could write step descriptions. Frank examples of the step descriptions are as follows:

    Then /^I should see a Result of (\d+)$/ do |result|
        list_of_text_contents = frankly_map( "view:'UITextField' marked:'Result'", "text" ) 
        list_of_text_contents.should have(1).item # make sure we only matched one view with our selector 
        list_of_text_contents.first.should == result
    end
    
    When /^I enter (\d+)$/ do |value|
      value.each_char do |number|
        touch "view:'UIButton' marked:'#{number}button"
      end
      touch "view:'UIButton' marked:'Enter'"
    end
    
    When /^I tap \*$/ do
      touch "view:'UIButton' marked:'*button'"
    end

Frank-Cucumber is an Automation by Wire style tool. It provides functionality for preparing your app for testing and to build out a structure for the tests to run within it. You then write your tests and create your step descriptions. When you run your tests, the cucumber tool runs your test files and uses Frank to drive the application. Results are displayed at the command line. The results include pass and fail results.

Additional Notes

• Frank is the fastest of the tools I’ve tried (and it is really fast)
• It does not support the full range of gestures
• It is GPL3 which is a less favorable license for some use

Frank is actively being developed. It has an active community mailing list on Google Groups.

Calabash (Open Source - Commercial Hosted)

Calabash is a similar concept to Frank. It also uses Cucumber and ruby to do it’s thing. It also provides tools to bootstrap your app. In fact, I didn’t rewrite my tests to work in calabash, merely revised the step descriptions to match calabash’s internal APIs. So, let’s look at the step descriptions:

The differences between Calabash and Frank are as follows:

• Calabash has a nice syntax for using steps inside steps, which makes them more readable.
• Calabash supports a broader range of gestures.
• Calabash has a more permissive license.

Additionally Calabash-ios has a cousin called calabash-android. This can be used to make similar tests on android devices. There are commonality in the provided step descriptions and the syntax which would reduce some overhead. In theory, you could share the tests, but in practice there might be challenges to do that.

Finally, Calabash offers an optional hosted solution. This solution allows tests to be authored locally and then uploaded to servers on the hosting company and run on devices there. The hosted solution is not free, though it is reasonablly priced to try out. I haven’t tried the hosted offering.

Calabash is actively being developed. It has active user mailing list on Google Groups. The developer appears to be quite responsive there.

Silk Mobile (Commercial)

SilkMobile is a fairly standard commercial tool offering. It supports capture/playback style of test authoring and seems to hope that that is what users focus on. It supports Android and iOS devices, as well as Blackberry and Windows Mobile. In doing this, it does seem to allow a common user experience in test authoring/execution.

Silk Mobile also supports exporting their test steps into other programming languages for use in unit testing. These exported tests are coupled with the SilkMobile app through the Elements that are captured as a part of the test authoring process. The unit tests can then access image capture and OCR functionality provided by the SilkMobile.

Concerns

SilkMobile appears to be a rebranded version of Experitest. The UI is the same, other than the branding and the videos demonstrating the product are virtually identical.

Additional Notes

• Works best on jailbroken devices, but supports the Automation By Wire style
• Runs only on Windows
• Supports Android and iOS (as well as other devices)

Other Paths, Not Taken

There is a multitude of tools out there that I have not looked at yet. Among them might be a real standout, but I have not had an opportunity to investigate them yet. Oth options that I could have looked at include:

• Soasta: Commercial, hosted solution
• HP Unified Software Testing: Comercial
• Bwoken: Open source tool that uses CoffeScript
• Telerik Test Studio: Commercial
• TestComplete: Commercial

In Conclusion

I don’t have a specific recommendation. I like Frank for its speed. I like Calabash for the broader functionality and the macro functionality. I sort of like instruments for the ability to run tests and diagnostics together.

I don’t recommend SilkMobile. I admit that I have alias against commercial tools, but based on my experience, most of what I want out of any co numerical tool is the engine. Thous I would end up using the junit runner, and I would be spending a lot of money on what is essentially a library.

Whatever the case, our mileage may vary and you may want to investigate all of these for yourself. I encourage you to. Check out the following links for more information:

• Dominik Dary on Calabash-iOS
• Dominiks’s list of open source mobile test tools
• Getting started with Frank

Yesterday Apple was handed a huge victory against Samsung. As an Apple partisan, I was feeling rather vindicated.

Let’s be honest, Samsung copied icons and imagery. For goodness sake, thy made their cables and chargers into black versions of the Apple ones. They imitated too well the Apple style. Then they made fun of fans who love products so much, that people wait in line for them. And what company selling a product doesn’t want that kind of loyalty?

But then the part of me that hates our patent system steps in. The part of me that knows there are small developers who have good ideas and that can’t do anything with them because there are trolls out there that will sue them for making a product that uses some obvious patent. And I think about Apple winning and wI wonder who really lost.

I don’t know if this will really be an apocryphal event in the phone biz. I just want everyone to have he cool phone they want, whatever OS it runs.

I’ve been meaning to do this for a long time. Back when I was discussing how I got FitNesse to work on Hudson, I pointed out this page from Andrew Palmer. Since then, FitNesse has changed a little and I’ve had to adapt my script to work with those changes. So here is my XSL to transform FitNesse results XML into Junit results:

<?xml version="1.0" encoding="ISO-8859-1"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSLTransform"> 
  <xsl:template match="/">
   <xsl:element name="testsuite">
     <xsl:attribute name="tests">
       <xsl:value-of select="sum(suiteResults/finalCounts/\*)" />
     </xsl:attribute>
     <xsl:attribute name="failures">
       <xsl:value-of select="suiteResults/finalCounts/wrong" />
     </xsl:attribute>
     <xsl:attribute name="disabled">
       <xsl:value-of select="suiteResults/finalCounts/ignores" />
     </xsl:attribute>
     <xsl:attribute name="errors">
       <xsl:value-of select="suiteResults/finalCountsexceptions" />
     </xsl:attribute>
     <xsl:attribute name="name"> <xsl:value-of select="suiteResults/rootPath" /> </xsl:attribute>
   <xsl:for-each select="suiteResults/pageHistoryReference">
     <xsl:element name="testcase">
       <xsl:attribute name="classname">
         <xsl:value-of select="/suiteResults/rootPath" />
       </xsl:attribute>
       <xsl:attribute name="name">
         <xsl:value-of select="name" />
       </xsl:attribute>
       <xsl:attribute name="time">
         <xsl:value-of select="runTimeInMillis div 1000" />
       </xsl:attribute>
       <xsl:choose>
         <xsl:when test="counts/exceptions > 0">
           <xsl:element name="error">
             <xsl:attribute name="message">
               <xsl:value-of select="count/exceptions" />
               <xsl:text> exceptions thrown </xsl:text>
               <xsl:if test="counts/wrong> 0">
                 <xsl:text> and  </xsl:text>
                 <xsl:value-of select="counts/wrong" />
                 <xsl:text> assertions failed </xsl:text>
               </xsl:if>
             </xsl:attribute>
           </xsl:element>
         </xsl:when>
         <xsl:when test="counts/wrong > 0">
           <xsl:element name="failure">
             <xsl:attribute name="message">
               <xsl:value-of select="counts/wrong" />
               <xsl:text> assertions failed </xsl:text>
             </xsl:attribute>
           </xsl:element>
         </xsl:when>
       </xsl:choose>
     </xsl:element>
   </xsl:for-each>
   </xsl:element>
  </xsl:template>
  </xsl:stylesheet>

o how do I use it?

First off I have the following ANT targets to run FitNesse and then capture the output:

<!--Run the tests first using the command line runner with text output.  This lets us see the results as tests complete.-->
 
     <java jar="javalib/fitnesse.jar" classpath="${toString:install.classpath}" fork="true" maxmemory="1024m">
           <arg value="-c" />
           <arg value="${fitnesseSuite}?suite&amp;suiteFilter=${fitnesseSuiteFilter}&amp;excludeSuiteFilter=${fitnesseExcludeSuiteFilter}&amp;format=text" />
           <arg value="-p"/>
           <arg value="${fitnesse\_port}" />
        </java>
        <!--Then run the page history responder to get the latest run of fitnesse in xml format-->  
 
        <java jar="/javalib/fitnesse.jar" classpath="${toString:install.classpath}" fork="true" maxmemory="256m" output="${fitnesse.output.file}.temp" >
           <arg value="-c" />
           <arg value="${fitnesseSuite}?pageHistory&amp;resultDate=latest&amp;format=xml" />
           <arg value="-p"/>
           <arg value="${fitnesse\_port}" />
        </java>
 
        <!--Trim off the text that FitNesse sends before and after the XML-->
 
       <exec executable="${env.RPFITDIR}/tools/XMLTrim.exe" output="${fitnesse.output.file}.xml">
          <arg value="${fitnesse.output.file}.temp"/>
       </exec>

The latest result will be written out to a temporary file, but won’t be true XML as there are http headers and traling characters around the XML. XMLTrim.exe is a little executable that can trim off the http headers in front of the XML results temporary file. I can’t share it right now, but will post something cross-platform in the future.

I have the following ANT target to transform the XML to the format we want.

<target name="convert-fitnesse-results-to-junit">
      <xslt style="src/xsl/fitnesse2junit.xsl" in="${fitnesse.output.file}.xml" out="TEST-${fitnesse.output.file}.xml" >
       </xslt>
</target>

Then I configure Hudson to use the results I generated, and we’re all set.