How Google Cloud helped Phoenix Labs meet demand spikes with ease for its hit multiplayer game Dauntless

In the role-playing video game Dauntless, players work in groups to battle monsters and protect the city-island of Ramsgate. Commitment reaps big rewards: with every beast slayed, you earn new weapons and armor made of the same materials as the Behemoth you took down, strengthening your arsenal for the next battle. 

And when creating Dauntless, game studio Phoenix Labs channeled these same values of resourcefulness, teamwork, and persistence. But instead of using war pikes and swords, it wielded the power of the cloud to achieve its goals.  

Preparing for unknown battles with containers and the cloud

For the gaming industry, launches bring unique technological challenges. It’s impossible to predict if a game will go viral, and developers like Phoenix Labs need to plan for a number of scenarios without knowing exactly how many players will show up and how much server capacity will ultimately be needed. In addition, since Dauntless was the first game in the industry to launch cross-platform—available on PlayStation 4, Xbox One, and PCs—it would be critical for all the underlying cloud-based services to work together flawlessly and provide an uninterrupted, real-time and consistent experience for players around the globe.

As part of staying agile to meet player needs, Phoenix Labs runs all its game servers in containers on Google Cloud Platform (GCP). The studio has a custom Google Kubernetes Engine (GKE) cluster in each region where Dauntless is available, across five continents (North America, Australia, Europe and Asia). When a player loads the game, Dauntless matches him or her with up to three other players, forming a virtual team that is taken to a neighboring island to hunt a Behemoth monster together. Each “group hunt” runs on an ephemeral pod on GKE, lasting for about 15 minutes before the players complete their assignment and return to Ramsgate to polish their weapons and prepare for the next battle. 

“Containerizing servers isn’t very common in the gaming industry, especially for larger games,” said Simon Beaumont, VP Technology at Phoenix Labs. “Google Cloud spearheaded this effort with their leadership and unique technology expertise, and their platform gave us the flexibility to use Kubernetes-as-a-service in production.”

dauntless-screenshot_stormchasers_login-fullsize.jpg

Addressing player and customer needs at launch and beyond

When Dauntless launched out of beta earlier this year, the required amount of server capacity turned out to be a lot. Within the first week, player count quickly climbed to 4 million—rapid growth that was no small feat to accommodate.

Continuously addressing Reddit and Twitter feedback from players, Phoenix Labs’ lean team worked side by side with Google Cloud Professional Services to execute over 1,700 deployments to its production platform during the week of the launch alone. 

“Google Cloud’s laser focus on customers reaches a level I’ve never seen before,” said Jesse Houston, CEO and co-founder at Phoenix Labs. “They care just as much about our experience as a GCP customer as they do about our players. Without their ‘let’s go’ attitude, Dauntless would have been a giant game over.”

dauntless-artwork_switch_launch_malkarion_box_art-fullsize.jpg

“Behemoth” growth, one platform at a time 

Now that Dauntless has surpassed 16 million unique players and launched on Nintendo Switch, Phoenix Labs is preparing to expand to new regions such as Russia and Poland (they recently launched in Japan) and take advantage of other capabilities across Google. For example, by leveraging Google Ads and YouTube as part of its digital strategy for Dauntless, 5 million new gamers were onboarded in the first week of launch; using YouTube Masthead ads also increased exposure to its audience. Phoenix Labs has migrated to Google Cloud’s data warehouse BigQuery for its ease of use and speed, returning queries in seconds based on trillions of rows of data. They’re even beginning to use the Google Sheets data connector for BigQuery to simplify reporting and ensure every decision is data informed. 

At Google Cloud, we’re undaunted by behemoth monsters—and the task of making our platform a great place to launch and run your multiplayer game. Learn more about how game developers of all sizes work with Google Cloud to take their games to the next level here.

How Google Cloud’s AI has boosted Netmarble’s team collaboration, game development and consumer reach

In less than two decades, Netmarble has become one of the world’s largest mobile-gaming companies, with more than 35 titles available in 120 countries, and hit MMORPG games like Blade & Soul RevolutionLineage 2: Revolution and most recently, BTS World. We began collaborating with Netmarble in 2017, at first to aid them in migrating to Google Cloud Platform (GCP), but more recently to help them leverage cloud tools and services to solve business challenges faced by many companies in the gaming industry. 

Most recently, we’ve worked with Netmarble’s AI Center, which manages all of the company’s AI initiatives. By applying AI to their infrastructure and operations, they’ve seen a wide variety of benefits, from faster team collaboration, to more intuitive game development, to increased reach in various regions.

In this blog post, we’ll share three examples of how Netmarble and its AI Center team have worked to infuse Cloud AI into all aspects of their business, improving development, game services and operations, marketing, and player experiences overall. 

ML for game services operation: churn factor analysis, churn prediction and in-game anomaly detection
For gaming companies like Netmarble with a substantial online and mobile audience, it’s not just enough to attract players, they need to retain them as well. This makes understanding why players stop playing a game—what’s known as churn—critically important.

Taking it a step further, Netmarble makes a churn prediction report, categorizing players by those who are likely to leave, who are likely to remain, and who should be managed. Based on this report, the Netmarble team can then decide each day what actions to take for each respective user group. 

“The churn report has been an invaluable resource, because we hadn’t previously had access to that type of information,” said Kim. “Our next goal will be to get even more nuanced with the report, in hopes to answer tough questions like how likely are we to lose a particular player? We’ll also want to look into what preventative measures we can take to retain users who have been categorized as ‘very likely’ to abandon a game.”

One way to retain users is to continuously add new content, but this can have unintended consequences, such as increasing the number of bugs which the QA team must address. By applying machine learning for automated testing, Netmarble can quickly find any bugs—even after a high volume launch day.

As games are successfully launched, millions of users will access the game, including many fraudulent users (such as hackers and bots, for example). That’s why Netmarble uses Google Cloud AI Platform to build ML models for fraudulent user detection. Learning the growth and consumption patterns of in-game users means anomalous behavior can be quickly identified, analyzed, and aggregated into a report for further assessment.

ML in marketing: from multi-market promotions to managing ad fraud
Game marketing can be complex, with many functions to think about, such as lead generation, digital communications and game launch promotions. Additionally, Netmarble must craft strategies and battle ad fraud not in one market alone, but in both Western and Asian markets. 

To address these challenges, Netmarble turned to BigQuery—a serverless, highly-scalable, and cost-effective cloud data warehouse—to build its Return On Advertising Spend (ROAS) prediction. This tool helps Netmarble predict when marketing expenses can be collected after spends in various regions. Its LTV prediction solution can evaluate the quality of users entered by cohort whom the marketer wants. 

In order to cope with a variety of ad fraud challenges, Netmarble set its AD Fraud Detection system to classify heterogeneous traffic through machine learning as well as through general rule-basis as part of its detection and mediation process. This way, the company can test new media and channels preventing invalid clicks and conversions. 

“In order to effectively reach the right audiences and make sure they have the best touchpoints with our games, we need to have very nuanced marketing procedures in place,” Duke Kim, SVP, Head of Netmarble’s AI Center, recently shared with us. “Google Cloud AI Platform gives us the agility and technological prowess to quickly and cost-efficiently build out internal solutions that best met these requirements.”

ML in game development: AI agents, balance checks and animation 
Just as the types of games that Netmarble makes has evolved in the past 19 years, so has the way it approaches game development. 

Now, the team’s next big focus is to create an AI agent that will help provide the best game experience for each player. Through this agent, which will be released soon, Netmarble intends to deliver a customized experience with specific levels, tasks or challenges tailored to a player’s skill level, that will ultimately help increase retention. This agent will offer players personalized user experiences and check how the user perceives a particular game. It will even be able to play on the user’s behalf in the event of an issue like sudden internet disconnection.

Netmarble is also looking to AI for voice and animation, which can be applied to in-game cut-scenes as well as using AI to animate the faces of its in-game non-player characters (NPCs). ML scripts prompt the NPC’s voice which then mimics mouse movements. 

“I never even dreamed of some of the functionalities that AI can now bring into a game,” Kim said. “We’ve only scratched the surface of AI’s benefits for games; I’m beyond excited about what lies ahead one, five and even ten years in the future. The best part is that Google’s Cloud AI technologies have been so easy to infuse into our games, typically only taking one month. I have no doubt we’ll be able to integrate the latest AI quickly, moving forward, with Google as a collaborative partner.”

It’s been fantastic to have such a close-knit relationship with Netmarble and the entire AI Center team for the past three years. We look forward to helping them continue to reach business goals and customers in the years to come. 

To  learn more about game development on Google Cloud, visit our website, and to find our more about Deployed AI business use cases, read this latest blog.

Firebase Unity Solutions: Update game behavior without deploying with Remote Config

Last June we announced Firebase Unity Solutions, an open-source Github repository with sample projects and scripts to help you add cloud-based features to your games being built on Unity. Our debut project,Firebase_Leaderboard, utilized Firebase Realtime Database to create and manage a cross-platform high score leaderboard. Today, we’re introducing the second solution incorporating Firebase services into Unity: Auto-Sync with Remote Config.

Auto-Sync uses Firebase’s Remote Config to allow Unity developers to modify their game’s behavior in real time, without requiring a fresh build and release cycle. This means you can tweak the behavior of entities in your game without waiting for builds to be processed through the app stores. 

In days past, making incremental changes to game behavior in real time required a good bit of setup. Firebase Remote Config is an outstanding tool for updating behavior without requiring a new publish cycle, but the Unity library requires hard-coding every single variable a developer wants to control.

With this package, you can pop the provided Component onto any GameObject or Prefab, tweak how to identify the fields you want remote access to, and click “Upload to Remote Config.” When the game is live, you can modify those values right in a custom Unity Editor built using Remote Config Management REST API, click “Upload” again, and voilà: the live game updates right before your eyes!

Think your platformer hero’s jump should be a little higher? Want your patrolling enemies to move a bit faster? How about modifying the density of powerups, ammo, or obstacles? Or disabling features in a marketplace until translations are ready, or toggling a promotion during a very specific time window? The possibilities are limitless; with Auto-Sync, this can all be done right from the Editor at a moment’s notice. No code push required.

gcp_firebase_unmapped_parameters.png

To learn more, visit theFirebase Unity Solutions repository on Github. You can incorporate the scripts into your project directly, or import the unitypackage file from the current-builds subdirectory to get started.

Massive Entertainment hosts Tom Clancy’s The Division 2 on Google Cloud Platform

As multiplayer games continue to increase in popularity, game developers need a reliable cloud provider with a flexible global infrastructure to support real time AAA gaming experiences. At Google Cloud we’ve spent many years building a world class infrastructure and easy to use solutions so that gaming companies and development studios can focus on what they’re most passionate about—building great games.

With the recent release of Tom Clancy’s The Division 2 by Massive Entertainment, a Ubisoft studio, we’re excited to share that Google Cloud was selected as the public cloud provider to host game servers globally for the highly anticipated sequel. Massive and Google Cloud worked together to deliver a smooth online experience and services for all players at launch.

“Google Cloud performed beautifully in our early tests and private beta, and we are thrilled with its ability to scale in the early days of our launch,” said Fredrik Brönjemark, Online & Live Operations Director at Massive. “But more importantly, we were looking for a partner to trust with our game. Google Cloud’s team of engineers and gaming experts get it; they’ve played our games, and were always available to us with deep technical expertise, from when we initially designed the game infrastructure to private beta and now launch.”

Massive Entertainment studio was looking for reliable and scalable cloud services that could keep pace with global player demand. Google Cloud provides Massive with the ease, flexibility and scalability to ensure consistently high game performance.

Google Cloud’s secure, global high speed fiber network allows for consistent high performance experiences for players across regions. The scalable infrastructure also supports game data and core services required for game play including matchmaking, high scores, stats, and inventory.

You can learn more about how game developers are using Google Cloud for game server hosting, platform services, and machine learning and analytics here. And for more information about game development on Google Cloud, visit our website.

How Google Cloud helped Multiplay power a record-breaking Apex Legends Launch

Can you take a wild guess how many players a new multiplayer game typically attracts in its first day of availability? Would you say thousands, tens of thousands or even hundreds of thousands?

Without any pre-launch marketing or promotional pushes, the free-to-play battle royale game Apex Legends, from Respawn Entertainment, reached a whopping one million unique players during the first eight hours of its debut on Monday, February 4, 2019. In the first 72 hours after its initial launch, Apex Legends reached 10-million players and has now reached 50 million unique players after just one month.

Managing such high levels of engagement can be nerve-racking and intense. If players experience connectivity issues at launch, the game may never recover. So much rides on a game’s launch, including its reputation, revenue, and longevity; it’s no surprise that it requires a robust infrastructure for an optimal multiplayer experience.

Apex Legends was developed by Respawn Entertainment and published by Electronic Arts, using the game server hosting specialists on Unity’s Multiplay team to facilitate the game’s availability across most major platforms. With a state-of-the-art cloud server orchestration framework and 24/7/365 professional services team, Multiplay is able to fully support ongoing game growth. The orchestration layer leverages Google Cloud to help deliver seamless global-scale game server hosting for Apex Legends in ten regions spanning the Americas, Europe, and Asia.

Predicting the capacity required for a free-play title, from such a prominent studio, is impossible. Multiplay’s Hybrid Scaling technology scaled the majority of the demand for Apex Legends with Google Cloud while utilizing its global network of bare metal data centers.  Google Compute Engine, an Infrastructure-as-a-Service that delivers virtual machines running in Google’s data centers and global network, provides the core computing services. Compute Engine enables Multiplay to effortlessly ramp up to match user spikes — a critical requirement for many games, especially since Apex Legends received 1M downloads in eight hours after its initial debut. Compute Engine virtual machines can also spin down quickly, correlated to player demand, helping to optimize costs when fewer game servers are needed.

Google Cloud’s global private network is also an important infrastructure component for Multiplay. Fast connections, low latency and the ability for game servers to crunch through updates as quickly as possible together ensure the best experience for players.

Multiplay, a division of Unity Technologies, creator of the world’s most widely used real-time 3D development platform, has had a long-standing relationship with Google Cloud.

“After working with Google Cloud on Respawn’s Titanfall 2, Google Cloud was the logical option for Apex Legends. With its reliable cloud infrastructure and impressive performance during our testing phase, it was clear we made the right choice,” Paul Manuel, Managing Director for Multiplay, recently shared. “Throughout launch, Google Cloud has been a great partner. We greatly appreciated the level of dedication the team demonstrated during the simulated game launch, and for making sure we had the necessary number of cores worldwide to support this launch.”

You can learn more about how game developers and platforms turn to Google Cloud for game server hosting, platform services, and global scale and reach in this blog post. And for more information about game development on Google Cloud, visit our website.

Making game development more flexible and open with Google Cloud

The gaming industry is entering a period of tremendous growth.There are more than two billion players across the world, from competitive gamers to casual enthusiasts, and they enjoy games across a variety of platforms. Whether it’s mobile, console, PC, AR or VR—anyone can play, from anywhere, on any device. 

But they are not playing alone. Advances in global connectivity have powered the rise of real-time multiplayer games that offer shared experiences to players from all over the world.

As a result, these global smash hits are more than just games—increasingly they are becoming platforms themselves, with complex game economies and growing live viewing and esports communities.

For game developers of all sizes, these trends have incredible implications for the underlying cloud infrastructure powering their games. To operate a global game, it’s critical to have reliable, scalable infrastructure. Game services, such as matchmaking, need to be flexible enough to support cross platform gaming. And finally, data, analytics, and machine learning are essential tools for optimizing player engagement, segmentation and monetization, especially with the prevalence of free-to-play models.

Google Cloud is already powering many of the world’s top AAA and mobile games and developers, helping build better player experiences. Our infrastructure has 18 regions and a presence in over 200 countries and territories, connected by our private fiber optic network, to ensure that game servers and players are as close to each other as possible. 

If your game requires working with bare metal or multi-cloud deployments, we provide that flexibility as well. Through Kubernetes, we empower you to simply run your backend services wherever it makes sense, and open source Kubernetes services like Agones—co-founded with Ubisoft—helping to make hosting and scaling dedicated game servers easy and flexible. To make it even easier for developers to take advantage of Agones, we’ve now made it available in the Cloud Marketplace, which makes installation and management available in just a few clicks.

We want to give game developers the freedom to build without being constrained by inflexible off-the-shelf solutions that put constraints on their vision—and that starts with building a stronger open source community for games. 

Open Match, our open source matchmaking framework co-founded with Unity, lets developers re-use their matchmakers instead of building them from scratch for every game. It’s designed for flexibility, allowing you to bring your own match logic, so you can build your game your way, across all platforms. Open Match was used to help create Google’s first multiplayer Doodle, which scaled to a peak of 500,000 concurrent players.

Finally, Google Cloud’s leading analytics and machine learning capabilities can help developers store, manage, and analyze the petabytes of data generated by hit games, and generate insights and predictions that can help grow your game. King, makers of the Candy Crush Saga, transitioned their data warehouse from Hadoop to leverage the scalability, flexibility and reliability of BigQuery in 2018, and created hundreds of virtual players, trained using our Cloud Machine Learning Engine (CMLE), to quickly gather insights that were used to optimize the game design.

If you’re attending GDC March 18-22 at San Francisco’s Moscone Center, please stop by our booth and say hello. Don’t miss our Cloud Developer Day on Wednesday, March 20 or our ongoing booth sessions at the conference to hear from Google Cloud experts as well as companies we collaborate with like DeNA, FACEIT, Improbable, Multiplay, Pocket Gems, Square Enix, SuperSolid, Ubisoft, Unity and others. They’ll share how they’re using Google Cloud to make great games. Can’t make it? No worries. Our sessions will also be live streamed and recorded, viewable here.  

If you attend GDC, you’ll also hear from other Google teams such as Google Play, Google Maps Platform, Assistant, and Android on how we’re working with developers to create great games, connect with players, and scale their business.

Let’s take your game to the next level, together.