+6 votes
I'm looking to understand some of the best practices for gathering data by soft launching an app without ruining long term global launches for that same app. I've heard different things like naming apps with different store ids, then launching once you have data or changing icons once you finish a test but keeping the store ids. Do these things matter? What's the best framework to think about testing an app?

There's also things like test flight, though I don't think you're able to truly understand your app in the "wild" using this.
by (1.6k points)

1 Answer

+1 vote

I'd tackle this question by first trying to succinctly establish what the goal of any Soft Launch is:

Can we convince ourselves that we can scale this product into a meaningful business?

That's it -- that's really the goal of a Soft Launch. Now we can break that goal down into components from a marketing standpoint:

Can we convince ourselves that we can scale this product into a meaningful business?

Why "convince ourselves" vs. "prove" or "determine"? Because you can't prove anything -- ever. The trouble I see teams getting into with Soft Launches is that they want to try to "prove" beyond a shadow of a doubt that marketing metrics will hold when geos change, when budgets change, when the app ages, when the channels change, etc. They put together these bulky, cumbersome "models" stuffed full of assumptions to try to convince themselves that they are making data-driven decisions when in reality they are just guessing very precisely. 

You can't prove anything about the app's global performance with sample data from what's essentially a small, controlled experiment. So what you're trying to do with a Soft Launch is establish firm conviction that the performance you see in your test geographies is representative of what it'll look like in global. How do you do that? 

First, you choose geographies that monetize in a way that mimics your largest markets in global launch. For most teams, this means some representative split of iOS and Android in a combination of AU / NZ / CA / Scandinavian countries. The problem with these countries is that they are small, meaning 1) traffic tends to be more expensive on a CPM basis and 2) saturation is reached faster. So really the best data you can get out of these countries is the monetization curve shape: unit economics aren't reliable from these countries because acquisition costs are disproportionately higher relative to the US and other large Western countries, and any sort of organic factor you see might be exaggerated given the small sizes of the markets.

I scratch my head when I see teams running Soft Launches in Southeast Asia because installs are "cheap" there. Unless your app is specific to Southeast Asia or otherwise isn't designed for a Western audience, what are you trying to prove with that traffic? Those countries skew Android, the GDP per capita is different from those in Western markets, credit card penetation is low, etc. Cost control shouldn't jeopardize the quality of data in a Soft Launch: if money is a constraint, then volume of data should be sacrified, not quality / representativeness of data.

Can we convince ourselves that we can scale this product into a meaningful business?

What does scale this product mean here? It means that growth can be achieved through performance marketing. In the Soft Launch, do the biggest and most important channels -- Facebook and Google -- support the growth of the app? You shouldn't be relying on niche or marginal channels in Soft Launch: you want to see effectiveness on the channels that are going to be important to you in global. I see teams experimenting with influencer marketing and radio and whatever else in Soft Launch and declaring success when they see traction without stopping to think if those channels will make up the bulk of their traffic in global.

You can save the experimental, small-potatoes traffic sources for global launch; unless you are planning to build the growth machine of your app around some non-standard or non-mainstream channel in global launch, don't focus on it in Soft Launch. I genuinely believe that the only two channels you need to think about in Soft Launch (as of 2019, this may change in the future) are Facebook and Google, since for 90% of apps, they'll be 50+% of traffic in global launch. Don't worry about proving success metrics on any other channel.

Can we convince ourselves that we can scale this product into a meaningful business?

One of my biggest pet peeves is when a user acquisition person says a channel is "great," but it doesn't provide much volume. What you care about in user acquisition is volume: if you are playing in the big leagues, you are looking for vast amounts of traffic, not marginal installs with low cost. A channel is not "good" for your product unless it is providing an appreciable, business-supporting amount of profitable traffic. I don't care about channels that provide 20-30 installs per day at low cost, profitably: that is small-ball thinking that doesn't scale businesses to meaningful size.

I wrote an entire article about this called The "Quality vs. Volume" Fallacy in mobile user acquisition, so I won't regurtitate those points. But what you're trying to establish in Soft Launch is whether you think the data you're seeing translates into volumes of installs in global launch to support a meaningful business. What meaningful means to you is a personal decision: is it $1,000 in revenue per day? $10,000? $1,000,000? Whatever it is, can you get from "here" (the metrics you're seeing in Soft Launch) to "there" (the volume of new users you'll need to hit your revenue target in global launch)?

The process of getting from "here" to "there" with a model is another area where teams get bogged down. You'll rely on assumptions here (how does a $5 cost per registration in NZ in iOS at $1k / day spend give me information on what the cost per registration will be in the US in global launch at $20k / day spend?) but you have to accept that you can't possibly cobble together enough assumptions to entirely remove risk from the process of globally launching.

So what do you do?

First, I think the Soft Launch / Global Launch paradigm is dated. You don't need to leave the gate in a global launch with tens of millions of dollars of spend in your launch week anymore -- platform featuring has far less of an impact on overall app success than it did in 2013-2014, so there's not as much value in a huge, saturation-level launch now as there once was. You can go global and scale incrementally over time, adjusting bids and budgets and in-app targeting events as you gather more performance data. The opportunity cost of a massive land grab launch is lower now; apps don't lose as much lifetime revenue by not growing as quickly as possible in the case that they can (and so the risk of lighting money on fire with a big launch for an app that can't support it should be lower, too).

by (15.2k points)
Eric addressed the correct way to think of soft launch.  For businesses...soft launch is the period you are testing to see if you have an app that you can scale and turn in to multi year revenue and profit center.   I'd also ask the question:  What am I gaining by not releasing in US? (or other primary market).   Yes you only get one launch to maybe leverage with the stores but you can't build a multi year business off a big launch. Are you trying to hide?  It's pretty hard to hide things in other markets these days.  However if it was so great you needed to hide it I'm not sure why you are in soft launch still.    As Eric highlighted I think there can be benefits to more quickly testing in your primary markets and still being incremental.    On SE Asia soft launch..we have done a release there first as more of a shakeout cruise....basically QA in the wild.  Don't see any point to actual KPIs there.
Yes, soft launching in a cheaper country for UA is a good idea when the goal is to iron out any technical issues. Phillipines is great for this and is also majority English speaking. But for user metrics definitely as Eric said, countries that will reflect your target market.
My opinion is that you do soft launch in phases. Phase 1 is mostly about figuring out if your app/game is not crashing and servers are stable. If technical stability is not there then all the other items are out of question. And as previous commenters mentioned Southeast Asia countries are pretty good locations to test it. Phase 2 is about testing if your retention is good or not. If there is no proper retention metrics then you can't really scale your business. There should be people retaining to monetize them. Countries like Turkey, Romania are good options.  Those countries have a lot of mobile gamers, decent representation of EU but lower than average monetization. Definitely localization is needed but you can get very decent number of installs with decent CPI to test retention metrics. Phase 3 is the monetization phase and you can use North European countries and Canada, Australia. The ultimate metrics are ARPDAU, ARPPU, Conversion and repeat conversion. These countries have higer CPI but definitely a final test for your game.
@Sertouch thanks, never thought about Turkey or Romania. What are your thoughts on using a country like India for gathering retention metrics? I can understand players there won't spend, or it might be worth reducing IAP costs significantly to test spend. India is the fastest growing mobile market now from what I've read, and there is a good degree of English penetration.
@meltdowninteractive India's network infrastructure is not really good based on my experience. If you have a game / app that relies on fast connections and/or real time multiplayer features i wouldn't recommend India. It results in 1 star ratings on Google Play and hurts your overall ratings.