GCP Archives - KenkoGeek

Want to use AutoML Tables from a Jupyter Notebook? Here’s how

While there’s no doubt that machine learning (ML) can be a great tool for businesses of all shapes and sizes, actually building ML models can seem daunting at first. Cloud AutoML—Google Cloud’s suite of products—provides tools and functionality to help you build ML models that are tailored to your specific needs, without needing deep ML expertise.

AutoML solutions provide a user interface that walks you through each step of model building, including importing data, training your model on the data, evaluating model performance, and predicting values with the model. But, what if you want to use AutoML products outside of the user interface? If you’re working with structured data, one way to do it is by using the AutoML Tables SDK, which lets you trigger—or even automate—each step of the process through code.

There is a wide variety of ways that the SDK can help embed AutoML capabilities into applications. In this post, we’ll use an example to show how you can use the SDK from end-to-end within your Jupyter Notebook. Jupyter Notebooks are one of the most popular development tools for data scientists. They enable you to create interactive, shareable notebooks with code snippets and markdown for explanations. Without leaving Google Cloud’s hosted notebook environment, AI Platform Notebooks, you can leverage the power of AutoML technology.

There are several benefits of using AutoML technology from a notebook. Each step and setting can be codified so that it runs the same every time by everyone. Also, it’s common, even with AutoML, to need to manipulate the source data before training the model with it. By using a notebook, you can use common tools like pandas and numpy to preprocess the data in the same workflow. Finally, you have the option of creating a model with another framework, and ensemble that together with the AutoML model, for potentially better results. Let’s get started!

Understanding the data

The business problem we’ll investigate in this blog is how to identify fraudulent credit card transactions. The technical challenge we’ll face is how to deal with imbalanced datasets: only 0.17% of the transactions in the dataset we’re using are marked as fraud. More details on this problem are available in the research paper Calibrating Probability with Undersampling for Unbalanced Classification.

To get started, you’ll need a Google Cloud Platform project with billing enabled. To create a project, follow the instructions here. For a smooth experience, check that the necessary storage and ML APIs are enabled. Then, follow this link to access BigQuery public datasets in the Google Cloud console.

In the Resources tree in the bottom-left corner, navigate through the list of datasets until you find ml-datasets, and then select the ulb-fraud-detection table within it.

Click the Preview tab to preview sample records from the dataset. Each record has the following columns:

Time is the number of seconds between the first transaction in the dataset and the time of the selected transaction.
V1-V28 are columns that have been transformed via a dimensionality reduction technique called PCA that has anonymized the data.
Amount is the transaction amount.

Set up your Notebook Environment

Now that we’ve looked at the data, let’s set up our development environment. The notebook we’ll use can be found in AI Hub. Select the “Open in GCP” button, then choose to either deploy the notebook in a new or existing notebook server.

Configure the AutoML Tables SDK

Next, let’s highlight key sections of the notebook. Some details, such as setting the project ID, are omitted for brevity, but we highly recommend running the notebook end-to-end when you have an opportunity.

We’ve recently released a new and improved AutoML Tables client library. You will first need to install the library and initialize the Tables client.

By the way, we recently announced that AutoML Tables can now be used in Kaggle kernels. You can learn more in this tutorial notebook, but the setup is similar to what you see here.

Import the Data

The first step is to create a BigQuery dataset, which is essentially a container for the data. Next, import the data from the BigQuery fraud detection dataset. You can also import from a CSV file in Google Cloud Storage or directly from a pandas dataframe.

Train the Model

First, we have to specify which column we would like to predict, or our target column, with set_target_column(). The target column for our example will be “Class”—either 1 or 0, if the transaction is fraudulent or not.

Then, we’ll specify which columns to exclude from the model. We’ll only exclude the target column, but you could also exclude IDs or other information you don’t want to include in the model.

There are a few other things you might want to do that aren’t necessary needed in this example:

Set weights on individual columns
Create your own custom test/train/validation split and specify the column to use for the split
Specify which timestamp column to use for time-series problems
Override the data types and nullable status that AutoML Tables inferred during data import

The one slightly unusual thing that we did in this example is override the default optimization objective. Since this is a very imbalanced dataset, it’s recommended that you optimize for AU-PRC, or the area under the Precision/Recall curve, rather than the default AU-ROC.

Evaluate the Model

After training has been completed, you can review various performance statistics on the model, such as the accuracy, precision, recall, and so on. The metrics are returned in a nested data structure, and here we are pulling out the AU-PRC and AU-ROC from that data structure.

Deploy and Predict with the Model

To enable online predictions, the model must first be deployed. (You can perform batch predictions without deploying the model).

We’ll create a hypothetical transaction record with similar characteristics and predict on it. After invoking the predict() API with this record, we receive a data structure with each class and its score. The code below finds the class with the maximum score.

Conclusion

Now that we’ve seen how you can use AutoML Tables straight from your notebook to produce an accurate model of a complex problem, all with a minimal amount of code, what’s next?

To find out more, the AutoML Tables documentation is a great place to start. When you’re ready to use AutoML in a notebook, the SDK guide has detailed descriptions of each operation and parameter. You might also find our samples on Github helpful.

After you feel comfortable with AutoML Tables, you might want to look at other AutoML products. You can apply what you’ve learned to solve problems in Natural Language, Translation, Video Intelligence, and Video domains.

Find me on Twitter at @kweinmeister, and good luck with your next AutoML experiment!

New Anthos training: a masterclass in hybrid cloud architecture and management

You’re moving faster than ever to build new applications, innovate, and bring value to your customers. Anthos, Google Cloud’s open application modernization platform, can help you modernize your existing applications, making them more portable, maintainable, scalable and secure. And now, our newest learning specialization, Architecting Hybrid Cloud Infrastructure with Anthos, is live, showing how you can use its technologies to transform your IT environments.

Designed for infrastructure operators, architects, and DevOps professionals, Architecting Hybrid Cloud Infrastructure with Anthos teaches you how to modernize, observe, secure, and manage your applications using Istio-powered service mesh and Kubernetes, whether you’re on-premises, on Google Cloud, or distributed across both. With a mix of lectures and hands-on labs, you’ll learn about compute, networking, service mesh, config management, and their underlying control-planes, so you can begin to understand the full scope of the platform’s capabilities. The training also unpacks the complexities of modern environments, and equips you with the foundational knowledge needed to address challenges such as migrating and scaling among environments hosted in multiple regions and by multiple providers.

This specialization builds on the Architecting with Google Kubernetes Engine (GKE) learning specialization, and assumes that students have extensive hands-on experience with Kubernetes. Architecting Hybrid Cloud Infrastructure with Anthos is delivered as three courses, which are available on demand and in a classroom setting:

Hybrid Cloud Infrastructure Foundations with Anthos – This course lays the groundwork for assembling hybrid infrastructure by presenting the Anthos platform architecture including Anthos GKE and Anthos Service Mesh.

Hybrid Cloud Service Mesh with Anthos – Gain the practical skills you need to deploy a service mesh to overcome challenges in multi-service application management, operation, and delivery.

Hybrid Cloud Multi-Cluster with Anthos – The final course will help you to understand configuration and get hands-on practice to manage a multi-cluster Anthos GKE deployment, including on-premises and in-cloud clusters.

Interested in hearing more? Register today for our webinar, Architecting Hybrid Cloud Infrastructure with Anthos, on Jan 31 at 9:00 am PST to get hands-on Anthos experience and receive a special discount on additional Anthos training.

Code to Inspire brings developer skills to girls in Afghanistan with the help of G Suite

Born in Iran as a refugee during the Soviet invasion of Afghanistan, I understand the challenges many people face there to get access to formal education. I was one of eight children in a progressive, yet financially limited, family. We left everything behind in Herat to move to a new country. To make ends meet, my mother sold handmade clothing. She invested what little she earned in my education, which made it possible for me to finish high school.

While opportunities for education in Afghanistan have increased over the past few decades, there are still many barriers that stand in the way of education for Afghan women—familial expectations, socioeconomic circumstances, cultural stigmas, societal norms and even safety issues. These circumstances make it challenging for women to find work and explain why only 19 percent participate in the workforce, 84 percent lack formal education and are often illiterate, and just 2 percent have access to higher education.

2001 marked the fall of the Taliban in Afghanistan, and many Afghan families, including my own, found hope in their motherland again. The next year, I returned to Herat. Seeing my peers —women just like me with so much potential—I felt compelled to do something. But I knew that in order to help others, I first needed to further my own education. I earned a bachelor’s degree in Computer Science, and later a Master’s degree from the Technical University of Berlin in Germany.

When I returned to Afghanistan after school, I hoped to share my newly-minted tech skills with Herat women by teaching at the local university. At the time, few women were participating in the public workforce and there were still many extremist, conservative views in the country. I was vocal about inequalities and faced backlash from the community; because of these threats, I came to the United States as an asylum seeker in 2012.

After arriving in the United States, I was inspired to start Code to Inspire, the first coding school for girls in Afghanistan between the ages of 15 and 25 that provides free after school education in gaming, web development, graphic design, mobile applications and full stack development. Our goal is to empower women with the skills they need to program so that they can drive change in their communities and gain equal access to opportunities and financial independence.

Determined to make this coding school happen, I found myself faced with an interesting obstacle: how could I build an education center in Afghanistan without ever leaving the US?

To make this possible, I turned to technology—just as I had hoped my future students would. I took a side job teaching Farsi to pay my bills, and built the blueprint for Code to Inspire from a laptop in Brooklyn. I managed everything from my computer working from cafes in New York: fundraising, shipping equipment, recruiting mentors, registering applicants, and developing the curriculum. To move things forward day-to-day, I use G Suite to communicate and collaborate with my team. I use Hangouts Meet to connect with students remotely during weekly calls and monthly check-ins. We use Google Docs, Sheets and Slides to create essential documents and keep track of our operations. We even used Slides for our first pitch deck, which I shared with my board to get their feedback. This power and connectivity enabled a refugee to make her dream come true, and built up the digital literacy of the women I was working with back home!

Since opening its doors, 150 girls have studied with us, and the students have created so many interesting projects. One inspiring example is the popular mobile game, Afghan Hero Girl, which has taken off in Afghanistan and abroad, and was even recognized by the local government. The girls developed the game to show a female protagonist dressed in traditional Afghan garb, who undergoes obstacles that are relatable to them.

Over 50 women have graduated from Code to Inspire, and 20 have secured remote full-time employment and freelance projects. Many of these women are even out-earning their male relatives who have become huge supporters of the program.

By 2030, we hope to open two additional schools in Kabul and Mazar that can serve up to 500 girls and provide employment opportunities within six months of graduation. From the ruins of a shattered nation and shattered lives of refugees can come treasure, if we know where to find it. For me, the girls in Afghanistan are the treasure and investing in their education is the future of a peaceful Afghanistan.

Making Gmail’s tabbed inbox work better for you

In 2013 we introduced tabbed inboxes in Gmail, which sorts your email into helpful categories in a simple, organized way. Over the years, we’ve heard from many Gmail users that these categories help keep their inboxes free of clutter so they can focus on getting things done. Recently, there have been some discussions around how Gmail classifies and sorts messages. We wanted to provide an overview of how classification works and share some best practices for both senders and receivers to make sure their emails land in the right tabs.

Gmail tabs use a classification system that applies machine learning to determine where to put email based on a variety of signals. Signals include (but aren’t limited to) who the email comes from, what type of content is in the message and how Gmail users have interacted with similar content.

Based on these signals, Gmail sorts messages into the following categories:

Primary: Emails from people you know (and messages that don’t appear in other tabs)
Social: Messages from social networks and media-sharing sites
Promotions: Deals, offers, newsletters and other “call to action” emails
Updates: Notifications, confirmations, receipts, bills and statements
Forums: Messages from online groups, discussion boards and mailing lists

As a user, you’re free to select one, a few or all of these categories. Gmail automatically adjusts to match your preferences and actions.

While Gmail takes all of these signals into account, the most important one is your direct input. Your actions teach Gmail how best to sort your email based on your preferences. Here are four things you can do to teach Gmail to sort email from certain senders into specific tabs, so you stay in control of where your email goes.

Move a message from one tab to another: Just drag and drop it, or use the right click menu. Gmail will prompt you to remember this preference in the future from email from this sender.
Create a filter: Use this option to create a filter that marks email from specific senders as important and/or directs it to a category of your choice.
Add senders to your contact list: This option tells Gmail you’re getting mail from a person you know.
Reply to the email: This is another way to indicate you and the sender are familiar with each other.

And if you don’t find tabs helpful, you can remove them altogether.

Sending large amounts of email? Take a look at our sender guidelines for tips on how to ensure that your mail gets delivered to the right place.

4 tips for managing information overload at work

Modern workplaces use many different communication tools to drive work forward—email, instant messaging, face-to-face video conferencing, and more. While multiple channels can make it easier to communicate with coworkers, they can also increase the amount of noise we experience every day. According to research from the University of California, Irvine, it can take up to 23 minutes to get back to a task once you’ve been interrupted.

So before you reply to that one email or seemingly important chat message, consider these strategies to manage communication more effectively.

1. Set aside time to read and respond to emails and chats

Alerts and beeps from incoming messages can interrupt your focus, so try to designate time each day to respond to email and chat messages. Doing so can safeguard the time you need for focused work and help keep unread messages from piling up.

Pro tip: Both Gmail and Hangouts Chat offer snooze and mute functions to help you manage time and protect your focus. High-priority notificationsare a handy way to quickly note what needs attention, too. You can also change your notifications in Chat to customize what you see.

2. Categorize communications by the action you need to take

One of my first bosses used to say, “When everything’s important, nothing’s important.” The immediacy of modern communications can make it feel like everything is urgent. When you’re buried under a stack of messages, it can help to categorize them by what they mean to you.

To get started, try asking yourself these questions:

Am I the best person to respond to this? (If you’re not, add that person to the conversation.)
Does this require a response, or is it just an FYI?
What’s the deadline for a response or action item?
How long will a response take? If it can be done in less than two minutes, do it. If not, set aside time to respond later.

As another boss liked to say, “The best fire drill is NO fire drill.” If something feels urgent but you need time to work on it, say so.

Pro tip: In Gmail, you can use labels and filters to organize your inbox and make it easier to categorize and respond to emails. For tasks you can address right away, try Smart Reply to respond even quicker in Gmail and Hangouts Chat. If the task requires others’ input or will take more than 30 minutes, you can always schedule a meeting instead.

3. Managers: don’t be part of the problem

If you’re a people manager, the way you communicate can have a significant impact on your employees’ emotional wellbeing. Here are a few of the best practices I try to follow:

Be clear about why you’re communicating. If you’re just passing on information, spend the 1.5 seconds it takes to add a “just FYI” note at the top of a forwarded email. If you need a response ASAP, clarify exactly what you need and assign a clear owner.
Don’t send email over the weekend. In Gmail, you can schedule messages to be sent later, so your employees don’t feel the pressure to respond over the weekend (even if that’s when you’re catching up).

Give context whenever you add people. For example, when you add people to an email thread or Chat room, explain why they’re there (“+Alexa will schedule this meeting, +Matt will do a first draft of the presentation”) so roles are clear.

Pro tip: It also helps to check in with your team and see what’s working (or not). Google Forms is an easy way to do an informal pulse check, and you can make the survey anonymous so team members feel more comfortable sharing honest feedback.

4. Take advantage of modern technology

One of the advantages of using modern tools like G Suite is that we’re constantly adding features to help you be more productive—like the ability to search your Chat history or enable nudges in Gmail so you don’t forget to follow up on an important email. (Take a look at our 2019 recap for more of the latest and greatest.)

Want more tips? Check out these. And for even more email tips, check out this post from Google Productivity Advisor, Laura Mae Martin.

Using deemed SLIs to measure customer reliability

Do you own and operate a software service? If so, is your service a ”platform”? In other words, does it run and manage applications of a wide range of users and/or companies? There are both simple and complex types of platforms, all of which serve customers. One example could be Google Cloud, which provides, among other things, relatively low-level infrastructure for starting and running VM images. A higher-level example of a platform might be a blogging service that allows any customer to create and contribute to a blog, design and sell merchandise featuring pithy blog quotes, and allow readers to send tips to the blog author.

If you do run a platform, it’s going to break sooner or later. Some breakages are large and easy to understand, such as no one being able to reach websites hosted on your platform while your company’s failure is frequently mentioned on social media. However, other kinds of breakage may be less obvious to you—but not to your customers. What if you’ve accidentally dropped all inbound network traffic from Kansas, for example?

At Google Cloud, we follow SRE principles to ensure reliability for our systems and also customers partnered with the Customer Reliability Engineering (CRE) team. A core SRE operating principle is the use of service-level indicators (SLIs) to detect when your users start having a bad time. In this blog post, we’ll look at how to measure your platform customers’ approximate reliability using approximate SLIs, which we term “deemed SLIs.” We use these to detect low-level outages and drive the operational response.

Why use deemed SLIs?

CRE founder Dave Rensin noted in his SRECon 2017 talk, Reliability When Everything Is A Platform, that as a platform operator, your monitoring doesn’t decide your reliability—your customers do! The best way to get direct visibility into your customers’ reliability experience is to get them to define their own SLIs, and share those signals directly with you. That level of transparency is wonderful, but it requires active and ongoing participation from your customers. What if your customers can’t currently prioritize the time to do this?

As a platform provider, you might use any number of internal monitoring metrics related to what’s happening with customer traffic. For instance, say you’re providing an API to a storage service:

You may be measuring the total number of queries and number of successful responses as cumulative numeric metrics, grouped by each API function.
You may also be recording the 95th percentile response latency with the same grouping, and get a good idea of how your service is doing overall by looking at the ratio of successful queries and the response latency values. If your success ratio suddenly drops from its normal value of 99% to 75%, you likely have many customers experiencing errors. Similarly, if the 95th percentile latency rises from 600ms to 1400ms, your customers are waiting much longer than normal for responses.

The key insight to motivate the use of “deemed SLIs” is that metrics aggregated across all customers will miss edge cases—and your top customers are very likely to depend on those edge cases. Your top customers need to know about outages as soon as, or even before, they happen. Therefore, you most likely want to know when any of your top customers is likely to experience a problem, even if most of your customers are fine.

Suppose FooCorp, one of your biggest customers, uses your storage service API to store virtual machine images:

They build and write three different images every 15 minutes.
The VM images are much larger than most blobs in your service.
Every time one of their 10,000 virtual machines is restarted, it reads an image from the API.
Therefore, their traffic rate is one write per five minutes and assuming a daily VM restart, one read per 8.6 seconds.
Your overall API traffic rate is one write per second and 100 reads per second.

Let’s say you roll out to your service a change that has a bug, causing very large image reads and writes, which are likely to time out and not complete. You initially don’t see any noticeable effect on your API’s overall success rate and think your platform is running just fine. FooCorp, however, is furious. Wouldn’t you like to know what just happened?

Implementation of deemed SLIs

The first and foremost step is to see key metrics at the granularity of a single customer. This requires careful assessment and trade-offs.

For our storage API, assuming we were originally storing two cumulative measures (success, total) and one gauge (latency) at one-minute intervals, we can measure and store three data points per minute with no problem at all. However, if we have 20,000 customers, then storing 60,000 points per minute is a very different problem. Therefore, we need to be careful in the selection of metrics for which we provide the per-customer breakdown. In some cases, it may be sensible to have per-customer breakdowns only for a subset of customers, such as those contracting for a certain level of paid support.

Next, identify your top customers. “Top” could mean:

invests the most money on your platform;
is expected to invest the most money on your platform in the next two years;
is strategic from the point of view of partnerships or publicity; or even
raises the most support cases and hence causes the greatest operational load on your team.

As we mentioned, customers use your platform in different ways and as a result, have different expectations of it. To find out what your customer might regard as an outage, you need to understand in some depth what their workload really does. In some cases, the customer’s clients might automatically read data from your API every 30 minutes, and update their state if new information is available. However, even if the API is completely broken for an hour, very few customers might actually notice.

To determine your deemed SLIs, consider applying your understanding of the customer’s workload from the limited selection of metrics per customer. Think about your observation of the volatility of the metrics over time, and if possible, observation of the metrics during a known customer outage. From this, pick the subset of metrics which you think best represent customer happiness. Identify the normal ranges of those metrics, and aggregate them into a dashboard view for that customer.

This is why we call these metrics “deemed SLIs”—you deem them to be representative of your particular customer’s happiness, in the absence of better information.

Some of the metrics you look at for your deemed SLIs of the storage service might include:

Overall API success rate and latency
Read and write success rate for large objects (i.e., FooCorp’s main use case)
Read latency for objects below a certain size (i.e., excluding large image read bursts so there’s a clear view of API performance for its more common read use case).

The main challenges are:

Lack of technical transparency into the customer’s key considerations. For instance, if you only provide TCP load balancing to your customer, you can’t observe HTTP response codes.
Lack of organizational transparency—you don’t have enough understanding of the customer’s workload to be able to identify what SLIs are meaningful to them.
Missing per-customer metrics. You might find that you need to know whether an API call is made internally or externally because the latter is the key representative of availability. However, this distinction isn’t captured in the existing metrics.

It’s important to remember that we don’t expect these metrics to be perfect at first— these metrics are often quite inconsistent with the customer’s experience in the beginning. So how do we fix this? Simple—we iterate.

Iteration when choosing deemed SLIs

Now sit back and wait for a significant outage of your platform. There’s a good chance that you won’t have to wait too long, particularly if you deploy configuration changes or binary releases often.

When your outage happens:

Do an initial impact analysis. Look at each of your deemed SLIs, see if they indicate an outage for that customer, and feed that information to your platform leadership.
Feed quantitative data into the postmortem being written for the incident. For example, “Top customer X first showed impact at 10:30 EST, reached a maximum of 30% outage at 10:50 EST, and had effectively recovered by 11:10 EST.”
Reach out to those customers via your account management teams, to discover what their actual impact was.

Here’s a quick reference table for what you need to do for each customer:

As you gain confidence in some of the deemed SLIs, you may start to set alerts for your platform’s on-call engineers based on those SLIs going out of bounds. For each such alert, see whether it represents a material customer outage, and adjust the bounds accordingly.

It’s important to note that customers can also shoot themselves in the foot and cause SLIs to go out of bounds. For example, they might cause themselves a high error rate in the API by providing an out-of-date decryption key for the blob. In this case, it’s a real outage, and your on-caller might want to know about it. There’s nothing for the on-caller to do, however—the customer has to fix it themselves. At a higher level, your product team may also be interested in these signals because there may be opportunities to design the product to guard against customers making such mistakes—or at least advise the customer when they are about to do so.

If a top customer has too many “it was us, not the platform” alerts, that’s a signal to turn off the alerts until things improve. This may also indicate that your engineers should collaborate with the customer to improve their reliability on your platform.

When your on-call engineer gets deemed SLI alerts from multiple customers, on the other hand, they can have a high confidence that the proximate cause is likely on the platform side.

Getting started with your own deemed SLIs

In Google Cloud, some of these metrics are exposed to customers directly through project-related, Transparent SLIs.

If you run a platform, you need to know what your customers are experiencing.

Knowing that a top customer has started having a problem before they phone your support hotline shrinks incident detection by many minutes, reduces the overall impact of the outage, and improves relationships with that customer.
Knowing that several top customers have started to have problems can even be used to signal that a recent deployment should presumptively be rolled back, just in case.
Knowing roughly how many customers are affected by an outage is a very helpful signal for incident triage—is this outage minor, significant, or huge?

Whatever your business, you know who your most important customers are. This week, go and look at the monitoring of your top three customers. Identify a “deemed SLI” for each of them, measure it in your monitoring system, and set up an automated alert for when those SLIs go squirrelly. You can tune your SLI selection and alert thresholds over the next few weeks, but right now, you are in tune with your top three customers’ experience on your platform. Isn’t that great?

Learn more about SLIs and other SRE practices from previous blog posts and the online Site Reliability Workbook.

Thanks to additional contributions from Anna Emmerson, Matt Brown, Christine Cignoli and Jessie Yang.

Exploring container security: Announcing the CIS Google Kubernetes Engine Benchmark

If you’re serious about the security of your Kubernetes operating environment, you need to build on a strong foundation. The Center for Internet Security’s (CIS) Kubernetes Benchmark give you just that: a set of Kubernetes security best practices that will help you build an operating environment that meets the approval of both regulators and customers.

The CIS Kubernetes Benchmark v1.5.0 was recently released, covering environments up to Kubernetes v1.15. Written as a series of recommendations rather than as a must-do checklist, the Benchmarks follows the upstream version of Kubernetes. But for users running managed distributions such as our own Google Kubernetes Engine (GKE), not all of its recommendations are applicable. To help, we’ve released in conjunction with CIS, a new CIS Google Kubernetes Engine (GKE) Benchmark, available under the CIS Kubernetes Benchmark, which takes the guesswork out of figuring out which CIS Benchmark recommendations you need to implement, and which ones Google Cloud handles as part of the GKE shared responsibility model.

Read on to find out what’s new in the v1.5.0 CIS Kubernetes Benchmark, how to use the CIS GKE Benchmark, and how you can test if you’re following recommended best practices.

Exploring the CIS Kubernetes Benchmark v1.5.0

The CIS Kubernetes Benchmark v1.5.0 was published in mid October, and has a significantly different structure than the previous version. Whereas the previous version split up master and worker node configurations at a high level, the new version separates controls by the components to which they apply: control plane components, etcd, control plane configuration, worker nodes, and policies. This should help make it easier for you to apply the guidance to a particular distribution, as you may not be able to control some components, nor is it your responsibility.

In terms of specific controls, you’ll see additional recommendations for:

Secret management. New recommendations include Minimize access to secrets (5.1.2), Prefer using secrets as files over secrets as environment variables (5.4.1), and Consider external secret storage (5.4.2).
Audit logging. In addition to an existing recommendation on how to ensure audit logging is configured properly with the control plane’s audit log flags, there are new recommendations to Ensure that a minimal audit policy is created (3.2.1), and Ensure that the audit policy covers key security concerns (3.2.2).
Preventing unnecessary access, by locking down permissions in Kubernetes following the principle of least privilege. Specifically, you should Minimize wildcard use in Roles and ClusterRoles (5.1.3).

Introducing the new CIS GKE Benchmark

What does this mean if you’re using a managed distribution like GKE? As we mentioned earlier, the CIS Kubernetes Benchmark is written for the open-source Kubernetes distribution. And while it’s intended to be as universally applicable as possible, it doesn’t fully apply to hosted distributions like GKE.

The new CIS GKE Benchmark is a child of the CIS Kubernetes Benchmark specifically designed for the GKE distribution. This is the first distribution-specific CIS Benchmark to draw from the existing benchmark, but removing items that can’t be configured or managed by the user. The CIS GKE Benchmark also includes additional controls that are Google Cloud-specific, and that we recommend you apply to your clusters, for example, as defined in the GKE hardening guide. Altogether, it means that you have a single set of controls for security best practice on GKE.

There are two kinds of recommendations in the GKE CIS Benchmark. Level 1 recommendations are meant to be widely applicable—you should really be following these, for example enabling Stackdriver Kubernetes Logging and Monitoring. Level 2 recommendations, meanwhile, result in a more stringent security environment, but are not necessarily applicable to all cases. These recommendations should be implemented with more care to avoid potential conflicts in more complicated environments. For example, Level 2 recommendations may be more relevant to multi-tenant workloads than single-tenant, like using GKE Sandbox to run untrusted workloads.

The CIS GKE Benchmark recommendations are listed as “Scored” when they can be easily tested using an automated method (like an API call or the gcloud CLI), and the setting has a value that can be definitively evaluated, for example, ensuring node auto-upgrade is enabled. Recommendations are listed as “Not Scored” when a setting cannot be easily assessed using automation or the exact implementation is specific to your workload—for example, using firewall rules to restrict ingress and egress traffic to your nodes—or they use a beta feature that you might not want to use in production.

If you want to suggest a new recommendation or a change to an existing one, please contribute directly to the CIS Benchmark in the CIS Workbench community.

Applying and testing the CIS Benchmarks

There are actually several CIS Benchmarks that are relevant to GKE, and there are tools available to help you test whether you’re following their recommendations. For the CIS Kubernetes Benchmark, you can use a tool like kube-bench to test your existing configuration; for the CIS GKE Benchmark, there’s Security Health Analytics, a security product that integrates into Security Command Center and that has built-in checks for several CIS GCP and GKE Benchmark items. By enabling Security Health Analytics, you’ll be able to discover, review, and remediate any cluster configurations you have that aren’t up to par with best practices from the CIS Benchmarks in the Security Command Center vulnerabilities dashboard.

Security Health Analytics scan results for CIS Benchmarks.png — *Security Health Analytics scan results for CIS Benchmarks*

Documenting GKE control plane configurations

The new CIS GKE Benchmark should help make it easier for you to implement and adhere to Kubernetes security best practices. And for components that they don’t cover, we’ve documented where the GKE control plane implements the new Kubernetes CIS Benchmark, where we are working to improve our posture, and the existing mitigating controls we have in place. We hope this helps you make an informed decision on what controls to put in place yourself, and better understand your existing threat model.

Check out the new CIS GKE Benchmark, the updated CIS Kubernetes Benchmark, and understand how GKE performs according to the CIS Kubernetes Benchmark. If you’re already using the GKE hardening guide, we’ve added references to the corresponding CIS Benchmark recommendations so you can easily demonstrate that your hardened clusters meets your requirements.

The CIS GKE Benchmark were developed in concert with Control Plane and the Center for Internet Security (CIS) Kubernetes community.

How Google Cloud helped Phoenix Labs meet demand spikes with ease for its hit multiplayer game Dauntless

In the role-playing video game Dauntless, players work in groups to battle monsters and protect the city-island of Ramsgate. Commitment reaps big rewards: with every beast slayed, you earn new weapons and armor made of the same materials as the Behemoth you took down, strengthening your arsenal for the next battle.

And when creating Dauntless, game studio Phoenix Labs channeled these same values of resourcefulness, teamwork, and persistence. But instead of using war pikes and swords, it wielded the power of the cloud to achieve its goals.

Preparing for unknown battles with containers and the cloud

For the gaming industry, launches bring unique technological challenges. It’s impossible to predict if a game will go viral, and developers like Phoenix Labs need to plan for a number of scenarios without knowing exactly how many players will show up and how much server capacity will ultimately be needed. In addition, since Dauntless was the first game in the industry to launch cross-platform—available on PlayStation 4, Xbox One, and PCs—it would be critical for all the underlying cloud-based services to work together flawlessly and provide an uninterrupted, real-time and consistent experience for players around the globe.

As part of staying agile to meet player needs, Phoenix Labs runs all its game servers in containers on Google Cloud Platform (GCP). The studio has a custom Google Kubernetes Engine (GKE) cluster in each region where Dauntless is available, across five continents (North America, Australia, Europe and Asia). When a player loads the game, Dauntless matches him or her with up to three other players, forming a virtual team that is taken to a neighboring island to hunt a Behemoth monster together. Each “group hunt” runs on an ephemeral pod on GKE, lasting for about 15 minutes before the players complete their assignment and return to Ramsgate to polish their weapons and prepare for the next battle.

“Containerizing servers isn’t very common in the gaming industry, especially for larger games,” said Simon Beaumont, VP Technology at Phoenix Labs. “Google Cloud spearheaded this effort with their leadership and unique technology expertise, and their platform gave us the flexibility to use Kubernetes-as-a-service in production.”

Addressing player and customer needs at launch and beyond

When Dauntless launched out of beta earlier this year, the required amount of server capacity turned out to be a lot. Within the first week, player count quickly climbed to 4 million—rapid growth that was no small feat to accommodate.

Continuously addressing Reddit and Twitter feedback from players, Phoenix Labs’ lean team worked side by side with Google Cloud Professional Services to execute over 1,700 deployments to its production platform during the week of the launch alone.

“Google Cloud’s laser focus on customers reaches a level I’ve never seen before,” said Jesse Houston, CEO and co-founder at Phoenix Labs. “They care just as much about our experience as a GCP customer as they do about our players. Without their ‘let’s go’ attitude, Dauntless would have been a giant game over.”

dauntless-artwork_switch_launch_malkarion_box_art-fullsize.jpg

“Behemoth” growth, one platform at a time

Now that Dauntless has surpassed 16 million unique players and launched on Nintendo Switch, Phoenix Labs is preparing to expand to new regions such as Russia and Poland (they recently launched in Japan) and take advantage of other capabilities across Google. For example, by leveraging Google Ads and YouTube as part of its digital strategy for Dauntless, 5 million new gamers were onboarded in the first week of launch; using YouTube Masthead ads also increased exposure to its audience. Phoenix Labs has migrated to Google Cloud’s data warehouse BigQuery for its ease of use and speed, returning queries in seconds based on trillions of rows of data. They’re even beginning to use the Google Sheets data connector for BigQuery to simplify reporting and ensure every decision is data informed.

At Google Cloud, we’re undaunted by behemoth monsters—and the task of making our platform a great place to launch and run your multiplayer game. Learn more about how game developers of all sizes work with Google Cloud to take their games to the next level here.

Google acquires AppSheet to help businesses create and extend applications—without coding

Today, Google is excited to announce that it has acquired AppSheet, a leading no-code application development platform used by a number of enterprises across a variety of industries.

The demand for faster processes and automation in today’s competitive landscape requires more business applications to be built with greater speed and efficiency. However, many companies lack the resources to address these challenges. This acquisition helps enterprises empower millions of citizen developers to more easily create and extend applications without the need for professional coding skills.

According to “The Forrester Wave™: Low-Code Platforms For Business Developers,” Q2 2019, “AppSheet has the most aggressive strategy and roadmap for empowering business people as developers. The platform had the highest score possible in the commercial model criterion and it shows in a stellar experience along with strong features for mobile app development, data design, application scaling, and documentation generation.”

AppSheet complements Google Cloud’s strategy to reimagine the application development space with a platform that helps enterprises innovate with no-code development, workflow automation, application integration and API management as they modernize their business processes in the cloud. AppSheet’s ability to power a range of applications—from CRM to field inspections and personalized reporting—combined with Google Cloud’s deep expertise in key verticals, will further enable digital transformation across industries like financial services, manufacturing, retail, healthcare, communication and media & entertainment.

With this acquisition, customers will be able to develop richer applications at scale that leverage not only Google Sheets and Forms which are already popular with customers, but other top Google technologies like Android, Maps and Google Analytics. In addition, AppSheet customers can continue to integrate with a number of cloud-hosted data sources including Salesforce, Dropbox, AWS DynamoDB and MySQL.

For more information, you can read AppSheet CEO, Praveen Seshadri’s blog post. We look forward to sharing more with you soon!

Krungsri Consumer: Preparing for the cardless future of finance with APIs

Editor’s note:Today’s post comes from Surin Asawachaisittigul, head of open APIs at Krungsri Consumer a subsidiary of Krungsri Bank, the fifth largest bank in Thailand, and focuses on personal loan and credit card services. Using APIs, Krungsri Consumer is taking an established bank into the digital future of finance.

The finance industry is changing quickly in the digital age. Customers no longer want to pay for purchases in cash. Even credit cards are starting to lose ground as customers turn to cardless mobile payment systems.

At Krungsri Consumer, we believe that innovation is key to succeeding in a rapidly evolving financial industry. Our strategy to stay ahead of competitors and enhance customer experiences is to offer more digital services and connect to more partners. That means switching from a traditional monolithic architecture to flexible microservices generated using OpenAPI.

We knew that we needed a robust API management system to serve as a strong foundation for our microservices architecture. It also needed to be secure and easy to use. We looked at many platforms, and the Apigee API Management Platform stood out for its developer portal and strong monetization features. Apigee gives us all of the tools that we need to grow our business and keep pace with changing customer demands and new competitive pressures.

Connecting with partners twice as fast

Our developers weren’t very familiar with APIs, so we teamed up with Apigee partner Tangerine to help us develop our first APIs. Working with Apigee turned out to be very straightforward, and our developers quickly learned how to structure and build APIs through Apigee.

The security features were extremely important for us as a financial institution. Apigee makes it very easy for us to apply API security standards such as OAuth2 to secure our APIs. The add-on Apigee Sense module will further help us protect our APIs from unwanted and malicious traffic.

We started by releasing five APIs related to our points program. Customers earn points when they make purchases with their credit card under Krungsri Consumer group. Our new APIs allow partners to authenticate accounts, look up point balances, and redeem points. Customers can easily pay with points just by shopping online.

Connecting with more partners benefits our customers, as they have a wider selection of products and services that they can easily purchase online using their points. For the partners, it means more customers who shop on their website or app.

The Apigee developer portal simplifies how partners connect with us through APIs. We can store all of our documentation in the portal, so partners can quickly learn how APIs work, what APIs are available, and whether they are free or if there are associated costs. We’ve had a great response to our developer portal so far.

Most importantly, the developer portal makes it much faster to onboard partners. It could take six months to integrate with partners’ systems before Apigee. Now partners can connect with our services in weeks. Some partners are new to APIs, so they need a lot of extra support to get up and running. But even these customers are getting onboarded in less than three months—about half the time as before.

Next, we’re planning to focus on APIs related to payment services. Once these APIs are live, e-commerce partners will be able to offer full or installment payment options through all credit cards of Krungsri Consumer (Krungsri Credit Card, Central The1 Credit Card, Tesco Lotus Credit Card and Krungsri First Choice Card).

Setting the stage for monetization

As a company offering credit cards and personal loans, we have access to a great deal of unique customer data. We’re developing new business models that will turn this data into revenue streams while maintaining customer privacy. For example, one new business that we’re working on is a credit check service, using a model that our data scientists built in-house. Once complete, partners will have the option to run a credit check on a customer using our APIs.

We plan to take advantage of the monetization features in Apigee to drive revenue from these types of data services. In fact, these monetization features were one of the top reasons that we decided to work with Apigee. Apigee not only streamlines setting up API monetization, but it enables partners to subscribe to an API by themselves through the developer portal.

Apigee is opening many doors for other types of future business opportunities. We can connect with new fintech services and startups that also use APIs. We can lead the market with new cardless payment services. And we can even expand services abroad to customers outside of Thailand. With our new API ecosystem, we’re ready for any challenge that may come so that we can give customers and partners the most innovative financial experiences possible.

IBM Power Systems now available on Google Cloud

Enterprises looking to the cloud to modernize their existing infrastructure and streamline their business processes have many options. At one end of the spectrum, some organizations are replatforming entire legacy systems to adopt the cloud. Many others, however, want to continue leveraging their existing infrastructure while still benefiting from the cloud’s flexible consumption model, scalability, and new advancements in areas like artificial intelligence, machine learning, and analytics.

To help you meet your cloud goals, whatever they may be, Google Cloud now offers IBM Power Systems as part of our cloud solutions. Today, customers can run IBM Power Systems as a service on Google Cloud—whether you’re using AIX, IBM i, or Linux on IBM Power.

For organizations using a hybrid cloud strategy, especially, IBM Power Systems are an important tool. Because of their performance and ability to support mission critical workloads—such as SAP applications and Oracle databases—enterprise customers have been consistently looking for options to run IBM Power Systems in the cloud. IBM Power Systems for Google Cloud offers a path to do just that, providing the best of both the cloud and on-premise worlds. You can run enterprise workloads like SAP and Oracle on the IBM Power servers that you’ve come to trust, while starting to take advantage of all the technical capabilities and favorable economics that Google Cloud offers.

IBM Power Systems on Google Cloud offers many other benefits, as well, including:

Integrated billing: You can deploy the solution through the Google Cloud Marketplace and take advantage of integrated Google Cloud billing. This means you can take advantage of this offering just like any other Google Cloud service and get a unified bill from Google Cloud.
Private API access: Google Cloud’s Private API Access technology lets you access Google Cloud resources privately, while enabling all IBM Power Systems resources (LPARs) to use private IP spaces that you choose. It’s secure by design and enables ultra low latency between the IBM Power servers and Google Compute Engine virtual machines.
Integrated customer support: Google Cloud manages customer support, giving you one point of contact for any issues.
Rapid deployment: An intuitive new management console enables quick ramp and rapid deployment of the solution.

Many enterprise customers, including leaders in energy and retail, have already begun modernizing their infrastructure with this new offering. To learn more about how you can use IBM Power Systems on Google Cloud contact your Google Cloud sales representative, or email us at [email protected].

Performance art: Making cloud network performance benchmarking faster and easier

Before you migrate workloads to the cloud, you need to benchmark network performance in order to understand how that performance affects your business applications. Unfortunately, the cloud hasn’t offered the standards, tools, and methods to do the benchmark testing you need. As a result, you’re forced to make deployment decisions without comprehensively understanding the implications of network performance for your use case.

Today, we’re excited to make a few announcements that will help you understand cloud network performance more quickly and easily:

We are investing in performance benchmarking tools. To begin with, we merged new contributions to PerfKit Benchmarker, an open-source tool created inside Google that makes network performance benchmarking faster and easier by automating network setup, provisioning of VMs, and test runs. With the updates, PerfKit Benchmarker now supports a broader range of network performance tests for VM-to-VM latency, throughput, packets-per-second for multiple clouds (inter-region, inter-zone, intra-zone, and on-prem to cloud), and lets you view the results in Google Data Studio (free to use). With this information, you can more accurately predict the performance impact of moving workloads to/across different clouds.
The publication of a new benchmarking methodology for using PerfKit Benchmarker continuously and consistently. This methodology, co-developed with performance engineering researchers at Southern Methodist University’s AT&T Center for Virtualization, is based on Google’s own internal best practices. “Continuous performance measurement and benchmarking are essential for understanding trends and patterns in large-scale cloud deployments,” said Suku Nair, director of SMU AT&T Center for Virtualization. “PerfKit Benchmarker, which wraps over 100 industry standard benchmark testing tools in an easy-to-use and extensible manner, is a key enabler in automating this process.”

Read on for an overview of how to use PerfKit Benchmarker to take advantage of its new features, such as support for additional performance metrics (e.g., packets per second) and deployment use cases (e.g., VPN).

Using PerfKit Benchmarker

PerfKit Benchmarker automates the setup and teardown of all the resources you need to run tests on (or between) most major public cloud providers, as well as on-premises deployments like Docker and OpenStack. Specifically, it automates the setup and provisioning of networks, subnets, firewalls and firewall rules, virtual machines, and drives required to run a large variety of benchmarks, as well as running the benchmarks themselves and tearing down the infrastructure afterwards.

Along with installing and running the actual benchmark tests, PerfKit Benchmarker packages the test results in an easy-to-consume JSON format and offers hooks into backend storage providers like Google BigQuery, Elasticsearch, and InfluxDB to automate publishing results, making reporting and analytics a breeze.

When performing network tests, the critical metrics you need to understand include throughput, latency, jitter, and packets per second. To find the values of these metrics across various configurations, you can use PerfKit Benchmarker to draw upon a number of testing tools, including iperf2, iperf3, ping, netperf, nuttcp, nttcp and NTttcp, just to name a few.

Once PerfKit Benchmarker has been installed, running a single benchmark is simple: Specify the test you want to run and where you want to run it. As a basic example, here is a ping benchmark between two VMs that are located in zone us-east1-b of Google’s cloud:

./pkb.py --benchmarks=ping --zones=us-east1-b --cloud=GCP

This command creates a new Virtual Private Cloud (VPC) and two new VMs in zone us-east1-b of Google Cloud, configures them for a ping test (including setting the appropriate firewall rules), runs the test, and then deletes the VMs and the VPC. Finally, it outputs the results to the console and stores them in a file in the /tmp directory. You can also store results in BigQuery or Elasticsearch when appropriate flags have been set.

Measuring Google Cloud inter-region latency with PerfKit Benchmarker

When designing your environment, it’s important to understand the latency between components in different Google Cloud regions. As an example, here are the results of our own all-region to all-region round trip latency tests using n1-standard-2 machine types and internal IP addresses. The daily benchmark tests ran over the course of the last month. The statistics were all collected using PerfKit Benchmarker to run ping benchmarks between VMs in each pair of regions.

To reproduce this chart, you can run the following command with the following config file. To run a smaller subset of regions, just remove the regions you don’t want included from the zones and extra_zones lists.

You can also add the --run_processes=<# of processes> flag to tell it to run multiple benchmarks in parallel. Furthermore, you can add the --gce_network_name=<network name> flag to have each benchmark use a Cloud VPC you have already created so each benchmark doesn’t make its own VPC.

./pkb.py --benchmarks=ping --benchmark_config_file=/pat /to/all_region_latency.yaml

More benchmarks using PerfKit Benchmarker

Other examples of network performance benchmark tests you can run using PerfKit Benchmarker include:

Inter-region, inter-zone, and intra-zone network performance tests
On-premises to cloud and cross-cloud performance benchmarks between a VM in one cloud, and a VM on-premises or in another cloud
Performance benchmarks using various network tiers
Benchmarking across various guest OSes (e.g., Linux vs Windows) and machine types (e.g., general purpose, compute-optimized)

For complete details about the methodology for running more of these benchmarks, read the “Measuring cloud network performance with PerfKit Benchmarker” methodology white paper.

More good stuff on the way

By using PerfKit Benchmarker, you can make better decisions about where to put workloads, and improve the experience of your end-users. As time goes on, we’ll continue to add coverage for new performance benchmarking use cases, publish additional guidelines for cloud performance benchmarking, and report on the experiences of cloud adopters. In the meantime, we welcome and encourage new contributions to the PerfKit Benchmarker codebase, and look forward to seeing the community grow!