This is the second blog in a two part series on mobile analytics. The first blog can be found here.
In my last blog, I typed about the challenges involved in the mobile space in creating a valuable analytics loop. It isn’t enough to talk about analytics and analytics systems; you need to effect positive change through. Without action they are just a bunch of numbers. In my first blog I boiled down the problem of creating value to three core issues.
1.How quickly we can change or enhance the analytics we are gathering
2.How quickly we can deliver a new solution(s) based on the analytics
3.How quickly we can accurately verify which solution is best solving the problem.
The third point needs a further explanation. For many people coming from traditional gaming worlds, accurately verifying multiple solutions is something not often done. A solution is decided based on the information available and designer knowledge. Delivering multiple variants of an application or feature to a user’s device can be complex and often slowed by the platforms submission and verification process. Traditionally, features have been delivered as a whole, and then deemed as working or not working by very coarse grain analysis. This is very different in the web world where multivariate testing and funnels are commonly used.
I will use a particular feature to help illustrate how we approached the three core issues and how we broke down the problem to improve our outcomes. As mentioned in the previous blog we were able to achieve some amazing gains in our multiplayer game usage in a relatively short time period. The goal we were concentrating on was having a user complete a single online match*.
In many cases a feature goal is made up of smaller goals or tasks. For us, we wanted people to complete an online match. To complete an online match the user must complete the following smaller tasks:
- Enter the online section of the game
- Select an opponent to play against
- Play a round of the game
- Report the score of the round
- Return to the online section to view the result
When the user has completed the final smaller tasks then we can consider our goal completed. The sub-goals are a series of conversions. For example, converting a user who has started the online section to a user who has selected an opponent is a conversion. The issue is that the percent of users completing each subsequent conversion diminishes. The ultimate goal would be to have all the conversions carry across 100% of the users from the previous conversion and have a 100% completion of the desired results (which fortunately never happens and keeps us all gainfully employed).
When doing analytics driven development, an interesting thing happens: the assumptions you make about what needs to occur (i.e. the smaller tasks) are sometimes wrong, or sometimes in the wrong order. You often find that the small tasks need to be further broken down to provide better understanding or granularity on why that portion of conversion is under performing. You may find that the measurement you originally created is not really reflecting an accurate picture of what is happening.
This leads us to tackle the first challenge:
Change or enhance the analytics gathered
Unfortunately we won’t get everything right the first time. This is why we iterate on the data we receive from our mobile game. There are two main approaches:
1.Boil the ocean and plan for everything
2.Keep things lean and iterate as needed
The first method doesn’t really work in a rapid iteration environment. It’s the same as the waterfall development model versus the agile development model. It can work when nothing surprising occurs but it otherwise limiting. In my experience (particularly in the gaming world when emergent discoveries can make all the difference) method one will fail to provide the data needed. So even in the first model you still need to iterate, and when it comes time to iterate you want to do it as fast as possible.
Additionally, I’ve found that collecting huge amounts of data without purpose can have a negative effect. Besides the obvious data storage cost, maintaining the mechanism to ingest data, and code complexities; the problem of huge data is that it very quickly becomes background noise. Finding the right piece of data can be obscured and confuses what needs to be looked at. The numbers become overwhelming.
A dashboard with 27 flashing red lights is a problem if only one of them really needs to be looked at.
In our example we noticed we had drop off between the start of the online game and the selection of an opponent to play against. While some of the predefined metrics indicated that we had some general asset loading issues there was an additional loss after the loading of assets and before the player was given the option to start a match. It turned out our tutorial was acting as a deterrent to starting the first match. This was only understood through multiple rounds of iteration and changes to the data sent to our analytics server. Fortunately, this was all done within a day.
We need to look at systems that can iterate quickly. This boils down to dynamic code or writing code which can act dynamically based on data. An example of the former is a scripting language engine that can run different scripts at execution time. An example of the latter is a configuration file that can turn on and off a series of command (a variable set to 0 might indicate that only level 0 analytic data should be sent and level 1 might indicate that level 0 and level 1 analytic data should be sent). Clearly the former provides more flexibility so we chose to dig deeper down that road.
The additional considerations for a dynamic scriptable solution are how the scripts are delivered and how they are stored.
A simple system for transfer could be using a standard communication protocol (such as TCP or UDP) to access a network stored asset at a known location. The script could then be stored in local storage and new versions can be checked for periodically.
The advantage of using standard protocols is that they allow you to take advantage of all the existing systems out there that have been built to optimize and accelerate web traffic. You can do so without having to write custom code and their advantages are well known. A good example of this is leveraging a context delivery network, or CDN, which can place your script code as geographically close to the user as possible.
Delivering a new solution quickly
The Grantoo product is a vibrant UI and backend solution, which supplies online and multiplayer services to games. Fortunately the solution is NOT the game itself but a UI that can be used to set up a multiplayer game and allows users to build a community with their friends around the games they play. Being UI-centric versus a twitch portion of the game was a significant criterion in how we solved the second problem of delivering a new solution into the game.
We knew that we wanted to iterate quickly and respond to our analytics information as fast as possible. We knew there were platform limitations (i.e. submission processes) in place that would hamper that delivery and slow the iteration cycle. It also is generally a bad practice (from an SDK standpoint) to rely on your game partners integration of a newer version of your SDK to improve your offering. Finally we also wanted a solution that would be available to all users at the same time, not having to rely on the user accepting a new version of the game, for us to be able to rapidly improve the performance of our subsection of the game.
In some ways this is the same problem as the scriptable code issue, just more complex. It essentially is the scriptable scene and UI problem. There are several solutions around this but each has their complexity and limitations. Most modern game development platforms are engines, which accept a number of assets in order to compose a visual scene. For the sake of performance, however, these scenes are “baked” so they can perform optimally on the target devices. This involves reducing script to machine instruction, asset compression, and optimization indices and layouts (amongst a whole other bunch of voodoo magic). In doing this kind of compression, the dynamic nature of the content creation process is removed, and it becomes difficult to change small subsections of the scene and its layout (of course, human ingenuity shone out with the early mod community which was able to decode bit streams, “patch” portions of the executable, and create incredible change in what were believed to be monolithic systems).
Given our need to build a UI and not a game, our options were increased. We choose to go with an HTML5/Canvas based solution (particularly cocos2d-html5) on the delivery of our UI. We were willing to give up some of the performance and extreme features to gain the flexibility of being able to add code and assets quickly, universally, and modularly. We also recognized that we would be able to leverage the growth advantages of an emergent technology. Mobile browsers (and WebViews) are becoming faster, more robust, and more full featured over time. Middleware providers of the HTML5/Canvas authoring systems are striving for speed, efficiency and platform compatibility. Essentially the platform providers are invested in the same goals that we are looking to get to.
This is not to say that we wouldn’t look at other ways to deliver our solution (particularly inside a WebView … any solution that could be hosted inside that environment is a fairly trivial switch for us as it continues to preserve the aspect of not requiring a new submission to work and can be delivered quickly and universally). Newer solutions such as famo.us (more CSS, DOM, JSS based) would work just as well for us and can be delivered without a hitch to our end users. Other solutions exist but for us the WebView with its built in script handling, solving the first core issue, made it easier for us to cement it as our choice to solve the second core issue.
In our example, the changes and eventual removal of the tutorial system was all done dynamically (i.e. no new integration was required by the game team and changes were immediately and universally available to all game users). Several iterations were required where different parts of the tutorial system were removed and tested to see impact on the sub-goal of “getting the player to select an opponent”. The selection screen itself was tested to see where placement and highlighting of buttons would bring the most people through the funnel.
We had a solution, which could update quickly and universally across multiple platforms. In this way we could generate new data whenever we wanted and dynamically update what the user was seeing with equal speed. This laid the groundwork for us to attack the third core problem. To really iterate quickly and to really understand the impact of our changes, we needed to do multivariate testing.
Selecting and verifying the “best” solution
Multivariate or A/B testing can be done with any coding system. Given the desire to get results on the largest audience possible (for reduction in sampling error), and being able to turn on and off tests and install “winners” as fast as possible, the multivariate test works best when combined with a system that can deploy features rapidly and universally as described above. Without universal deployment multivariate testing can be detrimental to a product. Imagine a product that (at the very same time) has the following installation profile:
- 25% of the audience on the old solution
- 25% of the audience on solution A with multivariate test code
- 25% of the audience on solution B with multivariate test code
- 25% of the audience on the final “winning” solution without the multivariate test code
This is a very real scenario when trying to deploy with a non-universal system (such as going through the platform store). Google Play and iOS7 has made it easier to have universal deployment (via automatic app updates) although each of these platforms has caveats on when it will work. The fracturing of the audience will have a long tail impact that worsens over time. Therefore, one of the key criteria on being able to successfully use a multivariate system is being able to quickly move your audience from a solution to test scenario and back to a solution again.
Going back to our example, one of the multivariate tests performed on opponent selection revolved around how many options were presented to the user. Our choices were to restrict user flow to the single option of playing a quick match, or opening up a choice to opponents as they might see after they had played the first match. The questions revolved around whether giving choice at this time was a benefit or more of a distraction, versus the possibility that we might be just delaying the drop off point. It is not a question with a confirmed answer and in reality we had multiple camps within the team. What we did share, however, was the goal.
The test was set up to test both the funnel in question (did the player select an opponent, as opposed to finding a way to exit the app) and the downstream impact (would there be a reduction in retention and creation of a second match) of both solutions. We set up the following A/B test and ran it for several days (over several games in our case to give us more understanding):
1.A cohort that only had the quick match button available on the first match but had all opponents available on the second match
2.A cohort that had all opponents available from the first match on
Through our testing we not only identified a significant lift in the selection of an initial opponent, but also saw no drop (while accounting for the new users who had moved further through the funnel) in the number of users who went on to play a second match. Once the test results proved conclusive we moved our new solution into place and stopped the multivariate test immediately. All players were moved to the new solution without stragglers.
By concentrating on the three core issues identified and finding a system that fit our criteria and constraints (or lack of constraints in some regard) we have been able to build a system, which allows rapid iteration. The system allows us to deploy both data and visuals quickly and universally across all platforms. In addition, building a lightweight A/B testing system on top, we were able to confidently test multiple solutions with regards to our goals.
The end result?
500 online games a day to nearly 150,000 online games a day within one game, within 3 months.
*Grantoo has created an asynchronous cross platform match up system. Each of the two users in the match plays a round (or rounds) by themself and their results are posted and compared to determine the winner.
At Grantoo we concentrate on bringing happiness to the world through games. We provide multiplayer and social solutions for mobile games. We focus on great user experiences and high levels of customer service.