English text:Control: A/B Testing in Mobile Games Losing
Almost every day I hear
We released a free racing game, where users can earn a soft money by taking part in the game, and they can consume a soft currency to buy a new car or upgrade their own car. You can consume an action point every day and only take part in 10 games, and choose to buy more action points or soft money through IAP. Users of the game is very good, but because the game is relatively narrow range of target users, the game's user acquisition may be more expensive, so the implementation of the need to think of ways to improve ARPU.
During the design of the meeting, it was suggested that the change of UI so that players can be in the main menu to see the upgrade page on the main menu. But after some discussion, the team split into two factions. It's a very good idea to send someone to think it's a good idea to improve the visibility of the upgrade page, which is a slot in the game's internal currency. Others are opposed; they think that reducing the visibility of the car's page will reduce the number of users to participate in the game, and therefore lead to their use less action point, which is another important point of the game's internal currency.
How to solve the division of the team? How can we determine which decision is correct? We both thought of the A/B test. So the programmer accepted the task and will change the menu flow through DLC. Now 50 of the new and the current user can see the highlight of the new menu, and the new page, while the other 50 users will still see the old menu highlight the car page. In the 3 week, the design team will get the answer, it may not be.
2 days prior to the change in DLC, the marketing team changed their user acquisition strategy, which is the combination of the first column in the country to use the higher CPI and higher prices of the. Based on this change, the program team in the test week time to modify a possible impact on the new car speed server vulnerabilities. In the end, it may be because of the holiday, the test results are shown to be able to promote the game in all IAP.
If we have enough users, so the effect of changing the menu will tend to be normal: This is the so-called law of large numbers. However, whether it is in black Friday sales or the first column of a higher proportion of the country's users, this will affect the user's decision to make the IAP and which game internal currency more effective. How much of the new track's download time can improve the functionality of the user stickiness? Before testing it is hard to determine whether the change in the DLC menu really affects ARPU or user behavior. In fact, it's hard to quantify the impact of these changes on the game.
From the core, a new game function of any A/B testing and based on a treatment and control group and the psychological experiment, that is, after all, will try to determine whether the experimental operation has a significant impact. As a researcher in the psychological laboratory, you can make sure that your actions are the only difference between your treatment and the control group. As a game analyst, the experimental group is your elements beyond your control of the unfortunate victims.
So what's left of this test? In my point here is not the A/B test is invalid, it is not that the decision is not affected by the data is not suitable for free mobile game ecosystem. On the contrary, I suggest that analysts and designers should carefully consider the costs of real experimental control (the ability to create other game changes) or to take the risk of making a decision on the use of error data.