Mobile is becoming increasingly important for companies that build web applications, and that also includes XING. Over 50% of our platform traffic comes from mobile devices. This in turn leads to a constant increase in the complexity and amount of testing work required on mobile devices.
At the beginning of 2015 XING launched a new internal initiative called “Unleashing Mobile”. The idea behind it is to upscale mobile development from a single mobile team to multiple teams within the company. The previous team setup was simply not able to keep pace with the development speed of the web platform and bring more and more features to the Android, iOS and Windows Phone mobile platforms. As things stand, we have 5 mobile feature teams developing features like profile, jobs, content or messages. Besides that, each platform has a central core team divided up into a platform and framework sub-team. The core platform team works on features that haven’t yet been passed on to the domain teams. As well as building its own app features, the core team has adopted more of a consulting role in helping to keep the whole app consistent and clean. Another key task of the central core teams is to integrate all of the code changes every two weeks to make sure that a stable app version can be released to our users.
The framework team works at an architectural level by providing other developers and software test engineers with the tools they need to carry out their duties, and by executing the bi-weekly releases.
The unleashing initiative also has a major impact when it comes to testing. Every team needs to have its own test devices or borrow some from the core teams. Handing out devices to the teams is quite hard as a lot of devices are needed and have to be maintained. Moreover, the teams are split across several locations, which makes it difficult to lend and borrow devices. The next aspect is the fact that there are so many different mobile devices on the market (especially for Android), which makes it almost impossible to buy them all for every team. By now we have more than 40 different Android devices at XING that are representative of the devices most used by our users. This number is growing on a monthly basis as we buy new phones and tablets that show up in our usage statistics.
To simplify this process and to ensure smooth coordination between all of the teams, we took advantage of our internal HackWeek to research the Smartphone Test Farm for Android based on openstf.io.
What is the HackWeek all about?
First of all we’d like to provide a short introduction to the XING HackWeek. The HackWeek takes place every 8 weeks. For 5 days, everybody involved in the development process (UX, PO, DEV, QA) can experiment with new things, irrespective of whether or not it’s relevant to XING. What’s important here is that participants learn and share the knowledge they acquire during the HackWeek. Some of the projects are of course XING related and will be implemented on the platform. Some projects simply involve learning a new programming language or using our 3D printer to print a mobile device holder. Either way, it’s a nice way to learn about new approaches in the development process, the product and even new people working in different teams.
We, Alexander Kreul, Daniel Knott, Katharina Gillmann and Sergej Mudruk, decided to use our 5 days to set up the Smartphone Test Farm based on http://openstf.io/ to see how it can improve our daily mobile testing process and to provide other colleagues with access to real devices from their desks.
What is STF?
STF is an Open Source GitHub project called Smartphone Test Farm, which is described by the authors as “a web application for debugging smartphones, smartwatches and other gadgets remotely, from the comfort of your browser”. With this tool it’s possible to access a real Android device via a web browser and control the device and its software remotely. The tool provides a lot of information, starting with general information about the device, the option to run automated tests through Android Studio as well as the remote control. STF supports every Android OS version.
Open STF also supports Android Wear watches up to Android version 5 (Lollipop), with support for Marshmallow on watches possibly being added in the next couple of months. The tool gives the user the opportunity to connect to several devices in parallel and then work on them by just selecting them from a browser.
Our research during the 5 days of the HackWeek
We started our research phase by first trying to set up the whole project. This turned out to be quite time-consuming as we ran into problems and conflicts with the installed software versions of homebrew and git. This turned out to be our mistake because we were using an old Macbook Pro preinstalled with older versions.
After solving these conflicts, the STF installation was actually very easy to perform. In total we needed about 2 hours to get ready to run the software, but were still very happy to see the login screen.
Now ready to interact with the tool, out first task was to just log in with a normal account, which worked surprisingly well despite the fact we weren’t sure which credentials were necessary here.
As we had no idea how the tool actually works, we used our testing curiosity and just went the easy way. We took a USB hub and plugged in several Android devices, which were immediately shown in the tool’s devices list.
Since the previous steps went without a hitch, we decided to go one step further. Our developer Alexander started to run our automated tests based on Espresso, available in Android Studio, on several connected devices via the remote tool. In parallel Daniel checked all of the possible operations and views available in the tool. Katharina and Sergej tried to remote control Android devices with an iPhone, which worked fine with Safari, as promised by the authors.
Of course we also tried an edge case and plugged in an iPhone to see if the tool can handle apple devices as well. We expected this test to fail, and it did as the creators of the tool stated that Apple devices aren’t supported yet.
A quite funny fact we figured out was that an Android device can be delegated from different other devices (laptop, second phone) in parallel as long as the same login is used. This allowed us to remotely navigate through the Android phone by switching from laptop to iPhone (and vice versa) and to see the action being executed on all devices in parallel without touching the Android phone at all.
Last but not least, we also tried to plug in an Android watch. Unfortunately this didn’t work because the watch was already running Android 6 already and the tool only supports up to Android 5 for now.
The following video shows open STF in action.
During our research phase in the first few days, we figured out several advantages and disadvantages concerning the software that we’d like to mention here.
Once open STF is installed, it’s really easy to get a remote connection to an Android phone by just connecting the device to the open STF server. Selecting and using a device is simple, just open a browser and start using the device. The fact that the tool is very easy to use gives us the opportunity to roll out the tool to the entire company, even allowing non-techy people to try stuff out with Android devices from their desks.
However, the tool has one drawback – and this is a lack of back-end administration for the system and connected devices. This feature is important when it comes to rolling out the tool to several departments within the company. People may forget to “release” the devices after using them and keep them locked, even they’re not in use. With the current status of the tool, it’s only possible to “release” a device by physically reconnecting it to the server or wait until the timeout is triggered by openstf. Having a back-end admin would simplify this as you could just disconnect the phone if it’s not been used in a while so someone else can work with it.
Another nice feature would be the option to restrict software updates on the phones. Having a valuable test lab means having different devices with different OS versions available. Of course, the same problem occurs if the device is physically lend, but a feature that restricts users from updating to new versions of the operating system would be nice to have. But since open STF is Open Source software we could perhaps adapt the tool to our needs and contribute the code to the community.
On the other hand, we discovered a really cool feature. If the user is remotely connected to a phone and downloads an app before disconnecting from the phone, the app is automatically deleted, meaning that the phone itself is always in a clean state. Having devices available to several users is a big advantage because the devices don’t have to be cleaned up manually from time to time.
After successfully researching and setting up the tool, we decided to continue with this project. Although we don’t want to go to the trouble of rolling it out to the whole company without first gathering some more experience, we want to introduce this tool to the core team for 1-2 sprints and see if it’s useful to them and has a positive impact on their testing process. If it does indeed turn out to be really helpful, we’ll then proceed by rolling it out to the whole company.
We’ll keep you posted with another blog post once we reach the stage where we plan to roll out the Android Smartphone Test Farm to the entire company. If you’re interested in becoming a beta tester for our XING Android app, just take a look at the instructions here. If you’d like more information about our scaling challenge, read the full story here.