QA, Automation Testing & the birth of the Hackathon
FX Digital has been working on building a test automation framework for a little while now. For what we class as native devices such as Apple TV, Roku and Android-based devices – like the Firestick and Nvidia Shield – our automation is complete and in daily use by our QA team. Though it was still a lot of work to get up and running, plenty of information exists on these devices and we could make use of open-source software to deploy these quickly and effectively.
However, we have two types of devices that we build TV applications for at FX digital, being Native and web-based (which uses our Lightning codebase). Lightning is where automation gets a lot more complex for our QA team. LightningJS makes use of webGL to render all elements of the DOM into a single image. This is great for performance, meaning our apps use far less CPU and memory resources. However, for our QA team, it poses an issue. Normally we are inspecting elements within the DOM to run tests. As lightningJS creates a single image, only one element exists within the DOM, and there aren’t a huge amount of tests we can run on that!
Automation has a lot of advantages for both our clients and FX digital, so we knew we needed to find a way to solve this solution. The team decided we should try to build a visual testing solution. We have many talented people at FX digital, and combined with our culture of working together to solve issues like this one across departments is something that really sets us apart. When FX Digital was a smaller business, it was a lot easier for the whole company to bounce ideas around together and resolve an issue like this. However, as the team has grown, this has become more of a challenge. So, we wanted to find an effective way to continue to do this and drive innovation utilising the entire development and QA teams.
After a few discussions, we decided a great way to come up with a proof of concept utilising a large number of people would be to host a Hackathon. This is an event where teams are pitted against each other to see who can come up with the best solution to a given problem. The brief here was quite simple: to perform a picture in picture search, one element would be of a screenshot from an app and the other would be of an element that the code needs to confirm is or is not in the image. Teams were to be given a recommended solution but allowed to use any tech they wanted to complete the challenge.
The Hackathon Event
We now had the concept, but wanted to make this event really stand out, and to do so, we decided we would build a live leaderboard. This would mean everyone could see which team was winning – in real time – during the event. This was designed in-house by our marketing team and built by one of our developers using react. We built a CI pipeline that would perform around seven hundred tests on the code and upload the result to the live leaderboard. What better way to test a system for automation than using automation!
The event itself took place across two days in our London office. We kicked the day off with a presentation about the tech created for the Hackathon and gave the teams information about the challenge. We had regular check-ins throughout the day where our Head of Engineering gave the teams more information on how he would go about solving the issue. Although we had created lots of tests to ensure that the solutions were successfully solving the challenge, we did not allow our teams to see these, they could just see the percentage of correct tests they had on the leaderboard. The challenge was given to our QA teams to try and find edge cases, so the code could be tested locally and find the images that were causing the teams software to return incorrect results.
Most of the participants were able to attend in person, however, we have a global team, and so some of the team participated remotely. They were able to login to access our leaderboard and attended the meetings via Zoom. Something that really struck me during the event was not only how the teams were working together to find a solution, but how they were also helping each other out. I think this really does underline how good the culture is here at FX Digital, ultimately enabling one of those teams to become the winner.
At the end of the final day, each of the team captains presented their solutions to us all and the official winner was announced. Our Managing Director Matthew Duhig then played a DJ set for the team while we celebrated with the winners.
Looking ahead
Not only have we now come up with a solution that’s set to be a game changer for our QA processes at FX Digital and also the wider TV app industry, we also had an amazing two days doing it, ultimately bringing the team closer together in the process. We are already in the process of integrating the winning solution into our brand new framework and we can use all of the other solutions to help find ways to make this even better.
As this was such a popular day that yielded such great results, we would be crazy not to do it all again. The organisation for the next FX Digital hackathon is already in progress. Stay tuned!
