Whenever app development gets more serious and your code becomes more complex, testing your application automatically is a step you need to go to survive. Xamarin provides two neat ways to write UI Tests for your applications. Calabash is a framework you can write tests in Ruby with and Xamarin.UITests provides everything you need to build test cases in C# the NUnit way. Whichever framework you choose, you can write tests for your application very easily and run them on an emulator or local device.
With both ways you can cover a managable range of devices which might not be enough to test how your applications perform in the real world. Especially when developing for Android, fragmentation is your daily companion and is not to be underestimated. Your application may run fine on all of your test devices and emulators but crashes on 80% of the rest.
Xamarin Test Cloud
This is, where Xamarin Test Cloud comes into play. It provides you a way to upload your app and tests to a huge farm of more than 1000 real devices and let them run there. Afterwards you will get a detailed test report of every single device including screenshots and Stack Traces for a comprehensive overview of which devices run your application nicely and which don’t.
I will not cover how to create these test cases in this blog post. If you are interested in the tests, take a look at the sample code at GitHub. What I want to show you today is how to upload your tests to the Xamarin Test Cloud and test your app on tons of devices in minutes.
To use the Xamarin Test Cloud you need at least version 5.9 of Xamarin Studio and a Test Cloud subscription. If you already subscribe to one of the Xamarin Platform plans you can start immediately due all of theminclude 60 monthly device minutes which should be enough for our demo. If you have neither, you can at least take a look at the demo accout.
Uploading your test
To upload your project to the Xamarin Test Cloud, simply right click on the tests and select “Run in Test Cloud”. The following steps run almost automatically and your application and tests are going to be uploaded into the cloud.
Important: Make sure, that you have changed your build configuration to Release and that it does not use the shared Mono runtime and supports the x86 ABI.
As soon as this is finished, your browser pops up and asks you to define your test environment. Basically this is where you can select the devices that should run your test. You can sure between several filters and pre-selections here. I personally liked the “Top 20 devices” option that selects the devices with most market share for you. Make sure that you also add some low end devices to test your application under extreme conditions.
Warning: Keep an eye on your remaining test hours! By selecting 30 devices and letting the tests run for only two minutes, your 60 device minutes that come with your Xamarin Platform subscription are already burned. When creating serious test scenarios, you should consider buying a dedicated Test Cloud subscription.
When all tests have finished you can take a look at the results dashboard where you can see how the tests performed on each device. For every single device you can follow each test and the screenshots that have been taken. This provides you insights of how your design works on the device. If a test fails you can also take a look at the Stack Trace and find out, why it crashed.
This is how I found out that my demo Pollen application seems to have a memory leak when loading the list of pollen with their images which causes crashes on weaker devices, which I never recognized when testing on my emulators and local devices.
Summarizing it up, Xamarin Test Cloud provides a really attractive way to test your apps on a wide variety of devices within a few minutes. Mobile developers know the pain of fragmentation and no one loves deploying their tests to dozens of physical devices the manual way. And let’s be honest, buying 50 different physical devices is much more expensive than one of these Test Cloud subscriptions!
I hope I managed to make Xamarin Test Cloud tempting for you. If you already have a Xamarin subscription it is definitely worth a closer look. Try it yourself – it might be the child you always wanted! Please also feel free to take a look at the tests in the sample code!