Beyond gameplay: testing the mobile gaming experience

The rise of mobile gaming shows no signs of slowing down. 

Not to mention, new demographics of people are coming together to defend cities, crush candy, and even catch and battle creatures using augmented reality. And this sudden surge in mobile gaming popularity isn’t just hype. One year into the pandemic, demand for mobile gaming has remained strong and is trending to continue that way. 

In fact, weekly downloads first surpassed 1B in March 2020 and have remained at that level since.

To take it one step further, in the first half of 2021, consumers spent $1.7 billion per week in mobile games, up 40% from pre-pandemic levels.

This sudden surge in mobile gaming presents the industry with a unique opportunity to capitalize on new and existing game players. To say there’s room for growth would be a massive understatement. 

The time to act is now.

You may feel like you need to rush your app into the app store, but without a deep understanding of this explosive market and its consumers, can you afford to miss the mark? It should come as no surprise that people abandon apps as quickly as they download them, so with all of this money and opportunity sitting on the table, it’s important to ensure you’re meeting the needs of your customers and optimizing your potential for success. 

In this game of survival of the fittest, how are you going to ensure you maintain a competitive advantage and break through the noise of all the other apps out there? To be specific, the ~4M in the Google Play Store and ~2M in the iOS App Store.

In this guide, we’ll walk you through what you need to know about testing the mobile gaming player’s journey (hint: it goes beyond simply gameplay), how to show return on investment (ROI), and ultimately how you can exceed customer expectations with confidence.

Challenges facing the mobile gaming industry

When it comes to building successful mobile games, or really any product for that matter, there are several challenges that emerge:

• Understanding your target audience and total addressable market (TAM)
• Building a solution that’s better than the competition/what’s already available
• Avoiding rework and saving the bottom line by getting it right the first time

Perhaps the most essential ingredient, to start, is an understanding of your target and total addressable markets. With smartphones taking residence in pockets around the world, the potential market for your next gaming app is greater than ever. And that means there’s a significant amount of work to be done attracting players and developing content and experiences that resonate with them enough to keep them coming back for more. 

Moreover, when a potential market has such a high opportunity for growth, it becomes increasingly difficult to break through the clutter. In fact, the average adult in the U.S. spends 11 hours and 27 minutes per day interacting with media across devices. While this gives you a lot of screentime, it also lays bare the total amount of content available for consumption. Innovation in the gaming industry isn’t easy, otherwise innovative ideas would be a dime a dozen. Developers and game designers are under huge pressure to create new experiences faster—those which gamers are willing to pay for. Ensuring that your next mobile gaming concept has product-market fit is essential.

Finally, it can’t be ignored that developing games is expensive. Especially if you don’t get it right the first time. Sure, sometimes games are released with bugs that can be fixed later, but you don’t get many chances to captivate an audience. This means minimal impact on player experience is paramount. Apps lose 77% of users within the first three days after the install, and that shouldn’t come as a surprise.  When a different game can be downloaded in as few as a couple of taps, you may only get one chance to impress.

But it’s not all doom and gloom. People are spending a ton of time playing mobile games and subsequently spending a proportionate amount of money on them too. 

Big risks sometimes reap big rewards. However, wouldn’t you prefer to bet on a game doing well in the market because your customers said it would? That’s what this guide is all about. Getting the fresh perspectives of your players to design, build, and launch your next mobile gaming app with their needs in mind.

Whether your largest hurdle is marketing, concepting, developing, innovating, or something totally different, you don’t have to take a shot in the dark. In this next section, we’ll outline exactly how you can incorporate real human feedback into your processes so you don’t have to wonder if you’ll meet customer expectations at launch. 

There’s never a right or wrong time to test the player journey

It’s widely known that a majority of new mobile gaming apps won’t become a viral phenomenon. The reasons are manifold. Perhaps the new game couldn’t oust a longtime player favorite. Maybe the new game was aesthetically second-to-none but was too difficult to play and not rewarding. Or maybe, despite being a superior game to others in the market, the marketing of it failed to compel.

Despite the best-laid plans and substantial investments of time and resources, failure to successfully launch happens all the time. But a successful launch goes way beyond QA testing. While that’s certainly important, your secret weapon is the unique perspectives of your potential players.

While there’s always some risk in new endeavors, you can hedge your bets by ensuring that you understand player expectations at every stage of your player’s journey. Doing so allows you to identify and remedy small problems along the way before they have the chance to become critical issues down the line.

We tested [our game] once a week for about eight weeks. More user testing is always better because we can identify and fix issues before the game launches to our user base.

Ketaki Shriram

CTO, Krikey

While there are many elements to the player’s journey, both inside and outside of the play experience, there’s never a right or wrong time to test. The rest of this section is organized into a typical player’s journey—from awareness, to conversion, to play experience. Testing the development of all aspects of the journey can happen at any time and in any order. In fact, the more often you test the better.

Let’s dive in to the player journey.

1. Awareness and consideration

When you think about a player’s journey, it’s easy to call to mind elements of gameplay that will capture and keep their attention. While this is certainly important, it’s not the first step in launching a successful game. Before a player even has a chance to play your game, they need to discover it. 

Most people split their tests based on where they sit in the player's funnel. We see tests happening either outside the game or inside: in user acquisition ads, store listings, welcome screens, and more.

Mobile gaming is a core tenant of owning a smartphone. Sure, not everyone considers themself a gamer, but most people have played a game on their phone before. This means your target audience may span from casual gamers to avid gamers, and everything in between. It’s unlikely that everyone will want to play your game, and that’s ok. Start by finding out who your audience is by conducting a needs assessment.

Needs assessment test

Before diving headfirst into the app store, get to know your players including what they’re looking for and their unmet gaming needs. Think of this as an open-minded conversation with your players. Give them the opportunity to share what’s on their mind and listen with open ears. This is often where you’ll find inspiration for your marketing materials and get a better understanding of your audience.

By getting to know potential players early in the process, you can gather the data that helps you make more informed decisions based on what they like or need most. In the example below, various mobile app companies asked potential users how they would react if their app included ads (or not). This is but one example of exploring a user’s needs, but the opportunities are endless.

Remote video URL

A great way to organize your customer needs is by way of an empathy map. While the name may sound complex, an empathy map is really just a tool that aids your active listening when speaking with potential players. By using it to map your conversation, you’re more easily able to organize feedback using a four-quadrant system—categorizing what people say, think, feel, and do.

Once you have a clear idea of who your players are and what they need, you’ll want to test your marketing and advertising materials.

Ad messaging and imagery

Now that you have a better understanding of your target audience, you’ll want to test your ad copy and visuals with them to ensure it piques their interest. You may want to start small, but there’s no shortage of things you can test:

• Get reactions to email copy including both subject line and body copy
• Test different messages on creatives to get a sense of which will perform the best and why without using any impressions
• Understand whether the messaging on your digital properties resonates with customers
• Test social media messages—including tweets or LinkedIn messages to avoid costly mistakes
• Get candid feedback on decks, datasheets, banners, and other marketing collateral

Remote video URL

The trick is to test your advertising and digital marketing to make sure the content you’re developing resonates with your target audience. Relevance is key. 

2. Conversion

At this stage in the player journey, your prospective customer has been intrigued by something and compelled to look into your game more deeply. While your previous goal was to attract your customers’ attention, your goal now is to get them to convert and download your game. Here are some things you should consider when trying to optimize your conversions and boost install rates.

App store optimization

Once you’ve intrigued a potential player, it’s likely they’ll move on to investigate your app-store listing. You should think of your app-store listing as a window-shopping experience for your mobile game. 

App store optimization (ASO) is key. To be successful at this, you’ll want to test your description copy, screenshots and capsule images, promo videos, and virtually anything you can think to include on the app store page.

Remote video URL

You may be surprised by what you discover. Often people can be turned on or off by elements you may have never considered before.

To take the window-shopping example further, you wouldn’t expect a retail store to keep their window display the same year-round. And this is because the needs of customers change and evolve. Let’s be honest, what compels someone to download a new game is likely different from what compels someone to download a game a year after it’s been released. Make sure you’re constantly testing and updating your app-store listing to maximize discoverability and align to the current expectations of players.

Competitive testing

Find out what’s working for competitors and areas where you may be lagging. When you understand what your customers like and dislike about your competition, you can make the necessary adjustments to help you surge ahead and win more downloads.

When doing competitive testing, consider what the install experience is like for your competitors. What do their app store pages look like? How do their calls to action read? What sort of gameplay images do they provide? Whatever you may think to test about your own experience, you may want to test on a competitor’s. 

3. Gameplay

Now that you’ve got your customer to download your game, what comes next in the player’s journey? Well, of course, it’s the gameplay. In this phase of their journey, your primary goal is to get them immersed in the experience. You want to create a gaming experience that they don’t want to put down, and when they do, they can’t wait to pick it back up. 

Gameplay has a ton of elements. From graphics to sound to the actual story of your game—you’ll want to test all of it. You can test them separately or together, but the main concern is that you get the information you need to make adjustments that will elevate your game to meet the desires of your audience. 

To be clear, there’s a lot of stuff you can test as it relates to gameplay. However, we’re focusing more on player preferences than QA testing. In other words, this section will help you uncover the deep needs, desires, frustrations, and perspectives of your players, rather than whether or not your codebase is up to snuff. With that in mind, let’s dive into the various elements you’ll want to test.

Gameplay

There are a lot of different styles of games out there. Some games feature deep gameplay mechanics or have a high difficulty level, others opt for addictive elements like puzzles, and yet others put the emphasis on the competitive factor. 

Whatever you choose to do, the game must entertain the player and meet the level of difficulty they’d expect to find. 

Here are some things to consider testing when it comes to gameplay:

• Level of difficulty and tutorials
• Useful vs. non-useful features
• Gaps in the logic of the game
• Texts like instructions, tips, and in-game messages

Remote video URL

Graphics and sound

A brilliant-looking game that’s enhanced with varied musical and sound effects creates a credible world that gamers want to plunge into. If your players constantly encounter visual errors and strange animations, or they’re irritated by poor voice actors and repetitive music, it detracts from the immersion.

Ask your customers how the music and sound effects you incorporate into the gameplay make them feel. Does it seem like it meshes with the world you created? Is it annoying? Is it inspiring? The list goes on.

Story

Interesting characters, exciting developments, and humorous (etc) dialogues ensure that your players feel emotionally involved and are curious to discover what will happen next. The choices you make—along with the way of playing—should lead to a personal experience.

In our new release of I Expect You To Die 2, we conducted full play-through diary studies with real players. We discovered that some of the crucial story elements needed to be made more obvious to ensure players picked up on them and got the full effect of the story.

Alexis Miller

Director of Product Management, Schell Games

4. Upgrade and upsell opportunities

Now that your players are immersed in the gameplay experience, it’s time to capitalize on the fact that they’re hooked. It’s popular today for mobile gaming apps to be offered on a freemium basis. This means that the app is free to download and play. However, if you test and optimize the moments in your game when a player can make a purchase, your freemium app can certainly still be a moneymaker.

Virtual currencies

Take Candy Crush Saga as an example:

The app lets users purchase gold bars with real money, and use them for in-game boosters such as extra lives. However, the game continues, even if you don’t purchase more gold bars—you just have to wait several hours to get your lives back.

In this scenario, you may want to test how people feel about this model. Do you have to wait too long to get your lives back? Is your pricing fair? The list goes on.

Unlock new levels

Another opportunity for in-app purchases includes paying to unlock new levels in the game.

Settlers of Catan is a strategy game that encourages its players to pay to unlock new maps (expansions) that introduce new elements of play. Unlocking new levels by paying money is considered a non-consumable purchase. You’ll want to test these purchase experiences to make sure they add enough value to the gaming experience that they’re worth the money.

Ad-free subscriptions

Another great way to monetize your freemium app is to offer ad-free subscriptions. Think the games with pop-ups that say “Tired of ads? Upgrade to premium to disable ads.” This element of the game lends itself to testing really well. You don’t want to lose players to annoying ads that disrupt games, but you also need to make money. Make sure you’re striking the right balance by asking your customers if the ads would completely deter them from playing your game.

With so many apps on the market, the freemium business model paves the way for businesses to attract and retain users while still making money.

The strategies listed above are just the start. Get creative and take inspiration from other successful businesses to find out how they monetize their freemium apps.

Proving the ROI for game testing

Effective game testing will make a huge difference to the lifecycle of your games, generating more revenue and keeping players passionate for longer. We know this to be true, but it’s likely your job to prove to relevant stakeholders that the time you spend testing the player journey is well worth it.

Whatever your role, it’s typical that you face the challenge of getting stakeholders excited about your findings and ensuring that the business understands the value of your output. With that said, there are many approaches to telling a story that truly speaks the language of your stakeholders.

From selecting the scope of testing to the format of the reports themselves, creating a story that resonates requires a deep understanding of what your audience cares about and crafting a message that speaks to that focus. Down the road, it also becomes critical to connect tangible outcomes to findings and recommendations. 

Here’s what it takes to measure the impact your testing has on your players.

1. Connect business goals to outcomes

Many teams fall into the trap of operating in silos, working independently from one another rather than collaboratively. As work gets passed from one team to another, it’s easy for testing goals to get lost. This results in teams simply filling orders and delivering reports without ever knowing whether their findings were implemented or what business impact they made.

To help alleviate this, heads of all the teams involved in your game’s development might want to start meeting on a regular basis. For example, meeting weekly with your design team, developers, product managers, and other stakeholders to discuss goals and outcomes for testing as they relate to overall business goals could drive collaboration and get everyone on the same page.

This meeting will have three primary goals:

• Ongoing tracking and benchmarking (described in more detail below)
• Improved communication between teams to keep the focus on business goals
• Foster a player-centric culture 

2. Identify business goals

One of the most compelling arguments for testing is that it can directly impact revenue. However, how you influence this will depend on the improvements that will make the biggest impact. 

To begin, you’ll want to review your company’s key goals for the upcoming quarter or even year. From there, you’ll determine which aspects of your game will most likely be able to influence that goal. This is how you’ll isolate your initial focus for testing. 

Perhaps you want to increase the number of downloads for your app? Out-of-game testing on marketing materials and collateral might be the best place to start. Perhaps you want to increase daily active users? Try testing your push notifications or how game progress or game score is being communicated. Or maybe you want to introduce another game to the market? It might be smart to run preference tests on existing games to see what works well and what doesn’t.

The point is, whatever you set out to improve should be in line with business objectives and be able to be tracked.

3. Benchmark results

Once you’ve identified the business goals, you’ll want to track your progress by comparing your numbers before and after you made improvements to your game. To do this, you’ll need to create a baseline from which you’ll measure your progress.

1. Start by noting the key metrics you typically track when measuring your game’s performance. This can be anything from downloads to in-game purchases.
2. Establish who will be responsible for tracking and communicating the results.
3. Determine how often you’d like to measure results: weekly, monthly, or quarterly. This will depend on what you’re testing and how long your development cycles are. It may take a cycle or two to determine the best intervals.
4. Once you’ve established your timeline, track your results over time, making sure to note what testing was conducted and what changes were introduced between measuring each metric.
5. Be sure to incorporate sample user videos, both before and after changes have been made, to help you illustrate the improvements

Over six months of testing and editing our Google Play Store page, our install conversion grew 45%. Now, we test every few weeks to keep up with new product features and trends in our target market. There are always improvements to be made.

Jhanvi Shriram

CEO, Krikey

Arriving at an experience that meets the hype

If you’ve tested every stage of your player’s journey, uncovered valuable first-person perspectives, and implemented feedback accordingly, then you’re well on your way to launching a mobile game that meets the expectations of your players. And if it’s not perfect,  that’s ok. Sometimes glitches and imperfections can be celebrated and become the next-best viral sensation. Developing a great game is a constant process, so don’t be discouraged.

Whether you accessed this guide to understand how to assess the potential marketability of your game, improve the overall design, or optimize your monetization strategy, it’s imperative for you to market-test your game as early and often as possible. 

By continuously collecting feedback, you’ll be better able to ensure that you’re delivering the gaming experiences your customers actually want. It might not be easy at first. Collecting feedback is a muscle you need to flex over and over until it becomes a natural part of your decision-making process and embedded in your culture. But the results are undeniable.

There are real benefits and ROI for gaming companies that make player-centricity their #1 goal. And the best path to player-centricity is to start talking to your customers and target audiences today.