Functional Testing Starling Applications with Cucumber and Melomel Redux - Part 2

Command Line Building

In my last post (http://ttmmhe.blogspot.co.uk/2014/01/functional-testing-starling.html) I discussed the pains of not taking a test first approach to building applications.  One of the core pieces to be successful in feature based development is to be able to consistently build and test your application, this means being able to build it almost anywhere and run your tests without the need for a developer to run the build an army of testers to manually click through it.

The first step to this is to get a build script running which could be executed on by your continuous integration server.

Thankfully there are a number of handy tools out there which you can use to create your build scripts, Ant, Maven, Gradle, Bash, Rake to name but a few, my personal preference is Rake which is ruby based, mainly as it integrates well with lots of other Ruby packages which use for testing and also its quite simple to write.

For this example I am going to use Rake and the Ruby gem ASRake which handles all our compiler commands very nicely.  It is not the purpose of this tutorial to go into the depths of Rake, the internet has many far better than I could write.

We are going to compile the demo which comes with the Starling download, other than changing a few file paths I have not modified the code at all.  All source for this example can be found at:

https://github.com/nicholasjackson/melomel_starling_example

The context of this assumes that you are using a mac or linux environment if you are using windows you will need to change the paths from / to \ however everything should still work.

Flex SDK

You are going to need a Flex SDK, I am currently using the Apache SDK 4.11 you can download this from http://flex.apache.org/ this will also install the Air SDK if you select it as an option.

ASRake requires that you set the environment variable for FLEX_HOME you can set this by executing the terminal command:
"export FLEX_HOME=/path/to/your/sdk"

Rakefile

For our build we have a few dependancies:

Ruby

ruby gem :- asrake

ruby gem :- zip

My personal preference is to use RVM (http://rvm.io/) to manage my Ruby installs and gemsets, the two files .ruby-gemset and .ruby-version let RVM know which gemset repository and version of ruby you would like to use.  Think of gemsets like workspaces, you can install all sorts of gems but they are local to that gem set so if you are working on a different project which has different dependancies there is no conflict.

There is also a file called Gemfile the contents of this are....

source 'https://rubygems.org' 

gem "cucumber", "~> 0.10.0"

gem "asrake", "~> 0.14.1"

gem "zip", "~> 2.0.2"

gem "rspec", "~> 2.14.1"

This file is used by the Ruby bundler command which will download and install the gems from the repository rubygems.org and install them for use in your gemset.  Currently we only need asrake and zip for this part of the process but it will not hurt to install the others.

From your command line execute 

"bundle"

If your system complains you do not have bundler then you may have to install it with "gem install bundler".

Now lets look at our Rakefile, at the top of the file we are defining our ASRake setup:

args = ASRake::Mxmlc.new "bin/Demo_Mobile.swf"

args.file_specs = 'src/Demo_Mobile.as'

args.target_player = 11.9

args.debug = true

args.source_path << "src"

args.library_path << "libs"

#args.external_library_path << "air/nativelibs/NativeGATracker-v2.0.4.ane"

#args.external_library_path << "air/nativelibs/NativeExtension.ane"

args.isAIR = true

current = Dir.pwd

unzipDir = "#{current}/unzippedane"

I'm going to run through this line by line but this is going to do our build.

# Create a new instance of as rake and specify the output directory for our compile binary, in this case the bin folder and Demo_Mobile.swf as a filename.

args = ASRake::Mxmlc.new "bin/Demo_Mobile.swf" 

# This line defines the path to our applications main class.
args.file_specs = 'src/Demo_Mobile.as'

# The version of player you wish to target.
args.target_player = 11.9

# Set to true to enable debug mode, false for your release builds.
args.debug = true

# Set your source path, if you have multiple source paths you can just add this line multiple times with your difference source paths.
args.source_path << "src"

# Set your library paths if you are using any pre-compiled libraries, again this can be set multiple times if you have more than one library path.
args.library_path << "libs"

# This line is commented out for our example as we are not using any native extensions however if you are then you add them like so.
#args.external_library_path << "air/nativelibs/NativeGATracker-v2.0.4.ane"

# We tell the compiler we are going to build an adobe air app
args.isAIR = true

# Set a variable to the current working directory
current = Dir.pwd

# Specify the directory to which we would unzip any ANE files should we have any
unzipDir = "#{current}/unzippedane"

Full details about the package can be found on the Github read me: (https://github.com/nexussays/ASRake)

Build Task

task :build => [args,:unzipane] do 
cp_u "src/Demo_Mobile-app.xml", "bin/"
Dir.glob("assets/*") do |folder|
cp_r folder, "bin/"
end
end

In the definition of this task the => arrow signifies that the task has a dependancy on other tasks, in this case args which is our ASRake build and unzipane which would unzip any native extension so that they can be picked up by the debugger.

By simply making a dependancy on the ASRake builder Rake will automatically invoke the execute method of args, sometimes you may only want to build your application and therefore your task would be nice an neat however in this instance we need to copy some assets which are loaded at runtime by the app.

I am not going to examine the :unzipane task as we are not using it in this example, its just there for reference should you need this.  If I get the time I will fork ASRake and add unzipping native extensions and running the build as rake tasks removing all this ugly code.

Build Our Code

This is the complex part, actually its not really all you have to do is type 

"rake build"

Hopefully you do not have an error message and you are now good to run the application.

Run the Application

I have created a rake task called run to start the Air Debug Launcher, the full settings and help can be found here (http://help.adobe.com/en_US/air/build/WSfffb011ac560372f-6fa6d7e0128cca93d31-8000.html).

The below code executes a shell command which starts the ADL.  The ADL requires certain parameters such as profile (mobile, desktop), screensize (predefined such as iPhone or custom), platform and a path to your application config xml file.  This file contains details such as the swf to start and various settings for the runtime.  Its exactly the same code that IntelliJ or Flash builder would generate for you.

flex = ENV["FLEX_HOME"]

sh "#{flex}/bin/adl -extdir #{unzipDir} -profile extendedMobileDevice -screensize iPhone -XversionPlatform IOS #{current}/bin/Demo_Mobile-app.xml"

To run the app all we have to do is execute the task run like so:

"rake run"

In our rake file you can see that run has a dependancy of build so all the build tasks would be executed before the ADL is started, this is mainly because I generally change code and forget to build then spend ages working out why my bug is still there.  If you just want run to run then remove => :build.

Hopefully now you will have the Starling demo running, the next part is to look at Melomel and how we can use it for our functional testing.

Source Download

https://github.com/nicholasjackson/melomel_starling_example

Functional Testing Starling Applications with Cucumber and Melomel Redux - Part 1

Background

A few months ago we decided to embark on a cross platform application for our store staff, there were three criteria which the application must meet:

1. It must be fun.
2. It must be cross platform.
3. It must be fun.

After spending some time thinking about what we wanted to do we realised that we wanted the app to feel like a game. We then had the harsh realisation that this was not going to be easy to code, in fact after running up some rough estimates of effort we realised that it was also going to be expensive to build native.

We did some research and concluded that HTML5 would not give the performance we wanted from animation on mobile platforms and that we did not really want to use Unity.   Having read a lot about Starling and the amazing cross platform performance we decided we would give this a go and started to build our stack around the Starling framework.  Both Keith and I used to be Flash programmers back in the day when Flash was still owned by Macromedia and both of us have a warm place in our hearts for ActionScript so coding in Starling should not be too alien.  Zan an Android developer who we had just recruited stupidly expressed interest in the project and having no past experience in either ActionScript or Flash was thrown straight into the deep end.  Alex an epic JavaScript developer who saw trouble a mile away was press ganged onto the project and we had our merry band.

One of the things we have been concentrating on at Marks and Spencer has been solid engineering principles, we have spend a lot of time working on Agile practices, Continuous Integration, TDD and BDD using cucumber.  Personally I have bored myself silly telling the developers about the virtues of Test First Development, how important CI is, etc, etc.

When we were putting together our stack for the app we decided that we would use Starling but we would also implement the Pure MVC framework to give us clean separation and testability.  We would write everything full TDD and we would have full automated functional testing.

Tech spike time and it did not go well, due to inexperience we battled with flex unit so dropped it, we could not work out how to test the Pure MVC components in isolation so we didn't, we could not find a functional testing frame work so we decided that our poor automation tester Anu would manually test the app, she still hates us and I don't blame her.

So in effect we decided to do everything that modern development methodology tells you not to and funnily enough we ended up in exactly the same mess that conventional test driven development is intended to avoid.

Post delivery of the app which was not a great success due to a large number of defects, these were eventually fixed but we realised that we could not continue developing in this way and had to either:

1. Change core technology to something which we can write tests for.
2. Write some testing frameworks.

We chose 2, mainly because I like Starling, and secondly because I refuse to admit I was wrong about choosing it in the first place.

Now Paul who works for us used to work for the BBC building their iPlayer application in Flex, he suggested we look at a framework called Melomel written by Ben Johnson (https://github.com/benbjohnson/melomel).  I took a look and it is indeed a very impressive framework however incompatibility between flash.display.DisplayObject and  starling.display.DisplayObject meant we could not use it.

Unless we modified it.

So we did (https://github.com/DigitalInnovation/melomel).

Melomel had not been touched for a couple of years and needed a little love so we abstracted out DisplayObject and its descendants into a common interface, modified all the tests, wrote implementations for starling objects, wrote a whole heap more tests added touch and swipe functionality and ended up with a rather nice Cucumber framework that allows us to interrogate and interact with our Starling applications.

In part two I am going to show you how to build and launch your application using a Rake not an IDE and get your initial cucumber framework setup.

Deprecated uniqueIdentifier in iOS7 work around.

This code is a work around for the deprecated method to retrieve the devices Unique ID in iOS7 [[UIDevice currentDevice] uniqueIdentifier] 

We currently have a variety of services which are using the UDID as an authenticator, in iOS7 the uniqueIdentifier method is deprecated meaning that only an app level unique id is the supported method of identification. If an app upgrades to iOS7 then the app will fail to work as it will no longer authenticate with the server and would require re-registration. 

To avoid this problem it has been proposed to install an update to the existing apps before the upgrade to iOS7 this would copy the UDID to the keychain, which could then still be used for authentication post iOS 7 roll out. 

This code is an example of how this could be implemented, instead of using [[UIDevice currentDevice] uniqueIdentifier] the developer would reference the "MNSAppIdentifier.h" header and then call [MNSAppIdentifier uniqueIdentifier] to retrieve the UDID instead. 

How it works: Internally when you first call [MNSAppIdentifier uniqueIdentifier] the system checks to see if the UDID is stored in the keychain, if it is it simply returns it to the user. If the UDID is not in the keychain and the user is using iOS6 or below then the code internally calls [[UIDevice currentDevice] uniqueIdentifier] and then stores this value in the keychain before returning the value to the calling code. If the user is using iOS7 and there is no stored value in the keychain then [[[UIDevice currentDevice] identifierForVendor] UUIDString] is called and the value again stored in the keychain before being returned to the calling code. Subsequent calls to [MNSAppIdentifier uniqueIdentifier] in both cases would return the keychain value. If the user upgrades from iOS6 to iOS7 and assuming the app has been installed and used on iOS6 then the value from the keychain which corresponds to the original device UDID will be returned and the app will continue to function normally.

NSUInteger DeviceSystemMajorVersion();
NSUInteger DeviceSystemMajorVersion() {
    static NSUInteger _deviceSystemMajorVersion = -1;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        _deviceSystemMajorVersion = [[[[[UIDevice currentDevice] systemVersion]
                                       componentsSeparatedByString:@"."] objectAtIndex:0] intValue];
    });
    return _deviceSystemMajorVersion;
}

#define kIS_LESS_THAN_IOS7 (DeviceSystemMajorVersion() < 7)

#import "MNSAppIdentifier.h"

@implementation MNSAppIdentifier

+(NSString*) uniqueIdentifier {
    
    // check to see if we have the UUID in the keychain
    MNSKeychainItemWrapper * wrapper = [[MNSKeychainItemWrapper alloc] initWithIdentifier:@"GenericUDIDKeychain" accessGroup:nil];
    NSString* udid = [wrapper objectForKey:kSecValueData];
    
    if(udid == nil || [udid length] < 1) {
        udid = [MNSAppIdentifier generateUDID];
        [wrapper setObject:udid forKey:kSecValueData];
    }
    
    
    NSLog(@"Got ID: %@",udid);
    
    return udid;

}

+(NSString*) resetIdentifier {
    
    MNSKeychainItemWrapper * wrapper = [[MNSKeychainItemWrapper alloc] initWithIdentifier:@"GenericUDIDKeychain" accessGroup:nil];
    [wrapper resetKeychainItem];
    
}

+(NSString*) generateUDID {
    
if (kIS_LESS_THAN_IOS7) {
        // get the device UDID
        return [[UIDevice currentDevice] uniqueIdentifier]; // standard udid
} else {
        // we need to generate the UDID from the vendor id this should be shared across team apps
        return [[[UIDevice currentDevice] identifierForVendor] UUIDString];
}
    
}

@end

Click here to download some example code

Monit and Go Applications

For those not aware of it Monit (http://mmonit.com/monit/) is an open source application for process monitoring on unix, well actually monit says it better....

"Monit is a free open source utility for managing and monitoring, processes, programs, files, directories and filesystems on a UNIX system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations."

Why use monit

One of the benefits of using a server like IIS or Apache is that even if your application crashes then the chances are that Apache or IIS is protected and when the next request comes in then your hosted application will re-spawn.  This also means that you do not generally have to wire up any startup code for when the server itself starts.

When you create a go application which is directly serving web requests then you are responsible for starting and keeping the application up.  If for some reason the process crashes or your server reboots then you have to log on the server and restart it or script this to happen automatically.

Monit manages just this behaviour, it also does a lot more such as alerting you at various configurable states.  In this tutorial I am just going to describe how to use monit to manage the startup and stayup for your go application but once you have the basics running you can experiment with additional configuration.

Wrapping your Go Application

Monit works by monitoring the pid (process identification file) file for your application, because Go does not automatically create these you either have to explicitly add this to your startup code.  In my humble opinion this is not so desirable as we want to keep our code cross platform capable without having to write extra nastiness just for the unix environment and also we can never be sure that the pid file directory is going to be the same.  Monit also asks you to declare startup and stop commands, for this reason I created a simple batch file.

 

#!/bin/bash

 case $1 in
    start)
       exec $(cd /var/go/go_application_root; ./application_executable &> log.txt &);
       sleep 5;
       echo $(pgrep application_executable) > /var/run/application_executable.pid
       ;;
     stop)
       kill -9 $(cat /var/run/application_executable.pid) ;;
     *)
       echo "usage: application_executable {start|stop}" ;;
 esac
 exit 0

So what is going on in this file? 

It's a basic bash script which expects one parameter with a value of either start or stop, we are staring the go application application_executable which resides in the same directory as our starup script and piping the output to the file log.txt, note the use of & this tells bash to run the command in the background.  The next line (echo $(pgrep documents_prod) > /var/run/documentserver.pid) we are echoing pid of the documents_prod process to the lock file which monit will monitor.  

I know you can get the pid by using the bash command $$ and $! but I did not have much success with this I was constantly getting a pid which was +/- 1 incorrect.  

The line sleep 5 is also important as before we can pgrep the process id we need to ensure that it has started.

To stop the process we are calling the standard kill comand and using the contents of the pid file we created earlier which contains the pid of the running process.

Note: the file needs to be executable so chmod 755 ./filename.sh

Setting up our environment

I use ubuntu but please feel free to substitute these commands with your favourite package manager.

sudo apt-get install monit

This command will install and start monit running as a daemon, monit has various options that can be enabled such as a http server for checking status, most are disabled as default.  We are only going to configure our mail server for receiving alerts and also set up the config for our go application.  For any other options please see the monit website.

Configuring Monit

First we need to add our mail server to the monit config, we are going to use a local smtp account but you can use anything you choose to

Edit the file /etc/monit/monitrc using your favourite editor, i'm going to use Pico because I could never figure Vi (let the flaming begin)...

sudo pico /etc/monit/monitrc

Find the section which starts with

## Set the list of mail servers for alert delivery. Multiple servers may be

## specified using a comma separator. If the first mail server fails, Monit

# will use the second mail server in the list and so on. By default Monit uses

# port 25 - it is possible to override this with the PORT option.

#

# set mailserver mail.bar.baz, 

and change set mailserver mail.bar.baz to ...

set mailserver localhost

This will send error messages via the local mail server, if you dont have a mail server you can install one by sudo apt-get install sendmail, be sure that you have inbound port 25 blocked by your firewall or you are opening a spam server.

Now lets set the address to send alerts to, add a line beneath the below section in the monit config.

## You can set alert recipients whom will receive alerts if/when a

## service defined in this file has errors. Alerts may be restricted on

## events by using a filter as in the second example below.

#

# set alert sysadm@foo.bar                       # receive all alerts

set alert nic.jackson@myaddress.com { nonexist } # receive only when service is stopped or restarted

Now set the mail from this may be required dependant upon your sendmail setup, edit the below section.

## You can override this message format or parts of it, such as subject

## or sender using the MAIL-FORMAT statement. Macros such as $DATE, etc.

## are expanded at runtime. For example, to override the sender, use:

#

set mail-format { from: nic.jackson@myaddress.com }

save this file "ctrl x"

Now we can create our monitoring config, for convenience separate config files can be placed in /etc/monit/conf.d/ so lets create our file here

check process documents_prod with pidfile /var/run/application_executable.pid

       start program = "/var/go/go_application_root/application_executable.sh start"

       stop program = "/var/go/go_application_root/application_executable.sh stop"

if 5 restarts within 5 cycles then alert

Reload Monit

You can now tell monit to reload your config with monit reload, next time monit runs its checks your process will be started if not already running.  For debugging check out the monit log in /var/log/monit.log

Good luck,

Nic