Document understanding is the practice of using AI and machine learning to extract data and insights from text and paper sources such as emails, PDFs, scanned documents, and more.In the past, capturing this unstructured or “dark data” has been an expensive, time-consuming, and error-prone process requiring manual data entry. Today, AI and machine learning have made great advances towards automating this process, enabling businesses to derive insights from and take advantage of this data that had been previously untapped.
In a nutshell, document understanding allows you to:
Organize documents
Extract knowledge from documents
Increase processing speed
At Google Cloud we provide a solution, Document AI, which enterprises can leverage in collaboration with partners to implement document understanding. However, many developers have both the desire and the technical expertise to build their own document understanding pipelines on Google Cloud Platform (GCP)—without working with a partner—using the individual Document AI products.
If that sounds like you, this post will take you step-by-step through a complete document understanding pipeline. The Overview section explains how the pipeline works, and the step-by-step directions below walk you through running the code.
Overview
In order to automate an entire document understanding process, multiple machine learning models need to be trained and then daisy-chained together alongside processing steps into an end-to-end pipeline. This can be a daunting process, so we have provided sample code for a complete document understanding system mirroring a data entry workflow capturing structured data from documents.
Our example end-to-end document understanding pipeline consists of two components:
A prediction pipeline which takes PDF documents from a specified Cloud Storage Bucket, uses the AutoML models to extract the relevant data from the documents, and stores the extracted data in BigQuery for further analysis.
Training Data The training data for this example pipeline is from a public dataset containing PDFs of U.S. and European patent title pages with a corresponding BigQuery table of manually entered data from the title pages. The dataset is hosted by the Google Public Datasets Project.
Part 1: The Training Pipeline
The training pipeline consists of the following steps:
Training data is pulled from the BigQuery public dataset. The training BigQuery table includes links to PDF files in Google Cloud Storage of patents from the United States and European Union.
The PDF files are converted to PNG files and uploaded to a new Cloud Storage bucket in your own project. The PNG files will be used to train the AutoML Vision models.
The PNG files are run through the Cloud Vision API to create TXT files containing the raw text from the converted PDFs. These TXT files are used to train the AutoML Natural Language models.
The links to the PNG or TXT files are combined with the labels and features from the BigQuery table into a CSV file in the training data format required by AutoML. This CSV is then uploaded to a Cloud Storage bucket. Note: This format is different for each type of AutoML model.
This CSV is used to create an AutoML dataset and model. Both are named in the format patent_demo_data_%m%d%Y_%H%M%S. Note that some AutoML models can sometimes take hours to train.
Part 2: The Prediction Pipeline
This pipeline uses the AutoML models previously trained by the pipeline above. For predictions, the following steps occur:
The patent PDFs are collected from the prescribed bucket and converted to PNG and TXT files with the Cloud Vision API (just as in the training pipeline).
The AutoML Image Classification model is called on the PNG files to classify each patent as either a US or EU patent. The results are uploaded to a BigQuery table.
The AutoML Object Detection model is called on the PNG files to determine the location of any figures on the patent document. The resulting relative x, y coordinates of the bounding box are then uploaded to a BigQuery table.
The AutoML Text Classification model is called on the TXT files to classify the topic of the patent content as medical technology, computer vision, cryptocurrency or other. The results are then uploaded to a BigQuery table.
The AutoML Entity Extraction model is called to extract predetermined entities from the patent. The extracted entities are applicant, application number, international classification, filing date, inventor, number, publication date and title. These entities are then uploaded to a BigQuery table.
Finally, the BigQuery tables above are joined to produce a final results table with all the properties above.
Step-by-Step Directions
For the developers out there, here’s how you can build the document understanding pipeline of your dreams. You can find all the code in our GitHub Repository.
Before you begin:
You’ll need a Google Cloud project to run this demo. We recommend creating a new project.
We recommend running these instructions in Google Cloud Shell(Quickstart). Other environments will work but you may have to debug issues specific to your environment. A Compute Engine VM would also be a suitable environment.
1. Git clone the repo and navigate to the patents example.
2. Install the necessary system dependencies. Note that in Cloud Shell system-wide changes do not persist between sessions, so if you step away while working through these step-by-step instructions you will need to rerun these commands after restarting Cloud Shell.
3. Create a virtual environment and activate it.
When your virtual environment is active you’ll see patents-demo-env in your command prompt. Note: If your Cloud Shell session ends, you’ll need to reactivate the virtual environment by running the second command again.
For the remaining steps, make sure you’re in the correct directory in the professional-services repo: /examples/cloudml-document-ai-patents/.
4. Install the necessary libraries into the virtual environment.
5. Activate the necessary APIs.
Note: We use the gcloud SDK to interact with various GCP services. If you are not working in Cloud Shell you’ll have to set your GCP project and authenticate by running gcloud init.
6. Edit the config.yaml file. Look for the configuration parameter project_id (under pipeline_project) and set the value to the project id where you want to build and run the pipeline. Make sure to enclose the value in single quotes, e.g. project_id: ‘my-cool-project’.
Note: if you’re not used to working in a shell, run nano to open a simple text editor.
7. Also in config.yaml, look for the configuration parameter creator_user_id (under service_acct) and set the value to the email account you use to log in to Google Cloud. Make sure to enclose the value in single quotes, e.g. creator_user_id: ‘[email protected]’.
8. Create and download a service account key with the necessary permissions to be used by the training and prediction pipelines.
9. Run the training pipeline. This may take 3–4 hours, though some models will finish more quickly.
Note: If Cloud Shell Closes while the script is still downloading, converting, or uploading the PDFs, you will need to reactivate the virtual environment, navigate to the directory, and rerun the pipeline script. The image processing should take about 15-20 minutes, so make sure Cloud Shell doesn’t close during that time.
10. After training the models (wait about 4 hours), your Cloud Shell session has probably disconnected. Reconnect to Cloud Shell and run the following commands to reinstall dependencies, reactivate the environment, and navigate to the document understanding example code.
11. Next, you need to use the AutoML UIs to deploy the object detection and entity extraction models, and to find the ids of your models (which you will enter into config.yaml). In the UIs, you can also view relevant evaluation metrics about the model and see explicit examples where your model got it right (and wrong).
Note: Some AutoML products are currently in beta; the look and function of the UIs may change in the future.
Go to the AutoML Image Classification models UI, and make sure you are in same project you ran the training pipeline in (top right dropdown). Note the ID of your trained image classification model. Also note the green check mark to the right of the model—if this is not a check mark, or if there is nothing listed, it means model training is still in progress and you need to wait.
In Cloud Shell, edit config.yaml. Look for the line model_imgclassifier: and on the line below you’ll see model_id:. Put the image classification model id after model_id: in single quotes, so the line looks something like model_id: 'ABC1234567890'
Go to the AutoML Natural Language models UI, and similarly, make sure you are in the correct project, make sure your model is trained (green check mark) and note the model id of the trained text classification model.
In Cloud Shell, edit config.yaml. Look for the line model_textclassifier: and on the line below you’ll see model_id:. Put the text classification model id after model_id: in single quotes, so the line looks something like ” model_id: 'ABC1234567890'“
Go to the AutoML Object Detection models UI, again making sure your project is correct and your model is trained. In this UI, the model id is below the model name.
You also need to deploy the model, by opening the menu under the three dots at the far right of the model entry and selecting “Deploy model”. A UI popup will ask how many nodes to deploy to, 1 Node is fine for this demo. You need to wait for the model to deploy before running prediction (about 10-15 minutes), the deployment status is in the UI under “Deployed” and it will be “Yes” when the model is deployed.
In Cloud Shell, edit config.yaml. Look for the line model_objdetect: and on the line below you’ll see model_id:. Put the object detection model id after model_id: in single quotes, so the line looks something like model_id: 'ABC1234567890'
Go to the AutoML Entity Extraction models UI, and make sure the project and region are correct. Verify the model training is complete by checking to make sure the Precision and Recall metrics are listed. Note the id as well.
To deploy the model, click the model name and a new UI view will open. Click “DEPLOY MODEL” near the top right and confirm. You need to wait for the model to deploy before running prediction (about 10-15 minutes), the UI will update when deployment is complete.
In Cloud Shell, edit config.yaml. Look for the line model_ner: and on the line below you’ll see model_id:. Put the entity extraction model id after model_id: in single quotes, so the line looks something like model_id: 'ABC1234567890'.
Before going further, make sure the model deployments are complete.
12. To run the prediction pipeline, you’ll need some PDFs of patent first pages in a folder in Cloud Storage.
You can provide your own PDFs in your own Cloud Storage location, or for demonstration purposes, you can run the following commands, which create a bucket with the same name as your project id (if it doesn’t already exist) and copy a small set of five patent first pages into a folder in the bucket called patent_sample.
13. In the config.yaml file, fill in the configuration parameter labeled demo_sample_data (under pipeline_project) with the Cloud Storage location of the patents you want to process (in single quotes). If you’re following the example above, the parameter value is 'gs:///patent_sample', substituting your project id.
Also, fill in the parameter labeled demo_dataset_id, just under demo_sample_data. With the BigQuery dataset (in single quotes) in your project where the predicted entities will be collected. For example, you can use 'patent_demo'. Note that this dataset must not exist already; the code will attempt to create the dataset and will stop running if the dataset already exists.
Now, you are ready to make predictions!
14. Run the prediction pipeline. This will process all the patent PDF first pages in the Cloud Storage folder specified in the demo_sample_data parameter, and upload predictions to (and create) BigQuery tables in the dataset specified by the demo_dataset_id parameter.
python3 run_predict.py
15. Finally, go check out the dataset in the BigQuery UI and examine all the tables that have been created. There is a table called "final_view" which collects all the results in a single table. You should see something like this:
Note, for this, or any of the other intermediate tables that were created during the prediction pipeline, you may need to query the table to see the results since it is so new. For example:
If you are new to BigQuery, you can open a querying interface in the UI prepopulated with the table name by clicking the “QUERY TABLE” button visible when you select a table to view.
Conclusion
Congratulations! You now have a fully functional document understanding pipeline that can be modified for use with any PDF documents. To modify this example to work with your own documents, the data collection and training stages will need to be modified to pull documents from a local machine or another Cloud Storage bucket rather than a public BigQuery dataset. The training data will also need to be created manually for the specific type of documents you will be using.
Now, if only we could create a robot that could scan paper documents to PDFs…
For more information about the general process of document understanding, see this blog post from Nitin Aggarwal at Google Cloud. And for more information about the business use cases of Document Understanding on Google Cloud, check out this session from Google Cloud Next ’19.
Ricardo Costeira is a Software Engineer from Coimbra, Portugal. For the first time last year, he attended DevFest, the largest developer community-led movement hosted by Google Developer Groups across the world. To celebrate DevFest 2019, we want to share with you Ricardo’s story and how he went from writing code to finding community.
Ricardo (left) and a fellow GDG Lead plan DevFest Coimbra.
1. How did you first hear about DevFest? What inspired you to join?
In 2018, after living in Coimbra for 3 years, I didn’t have any friends outside of work that were software developers. I longed to fill my life with more people that understood my passion and decided it was time to make a change. So I took to social media to see how I could connect with more like minded thinkers. Eventually, DevFest showed up on my feed. Out of nowhere, I saw this crazy event in Coimbra packed with bold leaders, energizing speakers, and profoundly creative exercises. I never expected that being with a community would get me so excited. I got a ticket on the spot.
2. Can you tell us about your first experience at DevFest?
It was exhilarating – out of this world. When I first walked in, everyone talked to me as if we had known each other for years. Big smiles, loud laughs, and deep kindness were all around me. As someone who is relatively shy and a loner by nature, I was stunned when I felt myself saying, “I belong here, I’m home.” That very same night, I looked up the next event I could attend. Since then, I have attended 2 other events, signed up for 6 more, and have become a GDG Lead. In other words, I’m hooked.
So how did this all change me? To be honest, DevFest brought forward a shift in my personality. I now want to be part of a community – that is a new feeling in my life.
“As someone who is relatively shy and a loner by nature, I was stunned when I felt myself saying, “I belong here, I’m home.”
3. What from DevFest 2018 are you looking forward to seeing again this year?
The booths. DevFest Coimbra has booths where you can talk with different companies. It’s exciting to learn about all the opportunities to grow so close to home. In my case, it was thrilling to see just how quickly Portugal is scaling and how so many companies come to DevFest eager to talk with the best talent. Forming these relationships is what can make the difference when finding the right opportunity for you.
4. Leading up to DevFest 2019, what are you most excited for?
Lighting a spark in new attendees. I recently joined the organizational staff and I’m excited to give new attendees that feeling I had when I first walked into DevFest. I’ve found such meaning in working with my fellow GDG Leads to bring together attendees in a sense of shared awe.
5. Any advice for those attending DevFest 2019?
Just say hi. You will be surprised with how far it will take you. DevFest is not only about the talks or workshops, it’s also about the people. This community knows the extrovert and understands the introvert, and warmly welcomes both. That is to say, no matter who you are or how you code, there is a place for you here.
“DevFest is not only about the talks or workshops, it’s also about the people.”
Want to find a DevFest event near you? Check out devfest.withgoogle.com to join our community, meet other developers and learn about Cloud, Android, Flutter, Machine Learning and more.
Supporting Windows workloads and helping you modernize your apps using containers and Kubernetes is one of our top priorities at Google Cloud. Soon after the Microsoft and Kubernetes announcements, we added support for Windows Server 2019 in Compute Engine and Windows containers in Google Kubernetes Engine (GKE).
Given this expanded landscape for Windows containers on Google Cloud, let’s take a fresh look at how best to deploy and manage them. In this first post, we’ll show you how to deploy an app to a Windows container on Windows Server 2019 on Compute Engine. Then stay tuned for the next post, where we’ll deploy and manage the same Windows container via Kubernetes 1.14 on GKE.
Let’s get started!
Create a Windows Server instance on Compute Engine
First, you need a Windows Server instance on which to run a Windows container. Compute Engine comes with many flavors of Windows Server (Server vs. Server Core), and many versions (2008 to 2019). There are also container-optimized versions that come with Docker and some base images already installed.
For this exercise, let’s choose the latest container-optimized version of Windows Server. In Google Cloud console, create a VM with the Windows Server 2019 Datacenter for Containers image:
Make sure that HTTP/HTTPS traffic is enabled:
And also make sure to select “Allow full access to all Cloud APIs”:
Allowing HTTP/HTTPs and Cloud API traffic will be useful later when we want to push/pull Docker images.
Once the VM is up and running, you can set a Windows password and connect into it using Remote Desktop (RDP). Inside the VM, open a Command Prompt in Admin mode and type the following:
As you can see, Docker is already installed and the Windows Server Core image for 2019 is already on the VM (“pulled,” in Docker-speak). We will use this as a base image for our Windows container app.
Create a Windows container app
For the app inside the Windows container, let’s use an IIS Web Server. IIS has an image for Windows Server 2019. We could use the image as is and it will serve the default IIS page. But let’s do something more interesting and have IIS serve a page we define.
Create a folder called my-windows-app with the following folder and file structure:
Replace index.html with the following content:
This is the page IIS will serve.
Build a Docker image
Next, let’s create a Dockerfile for the Docker image. Notice that we’re using the IIS Container image version that is compatible with Windows Server 2019:
Build the Docker image and tag it with Google Container Registry and your project id. This will be useful when we push the image to Container Registry later (replace dotnet-atamel with your project id):
Once the Docker image is built, you will be able to see it along with its IIS dependency:
Run your Windows container
We’re now ready to run the Windows container. From inside the command prompt, run the container and expose it on port 80:
You can check that the container is running:
To see the web page, go to the External IP column of the Compute Engine instance and simply open it with HTTP in the browser:
We’re now running an IIS site inside a Windows container!
If you want to try out these steps on your own, we also published a codelab on this topic:
Note that this setup is not ideal for production. It does not survive server restarts or crashes. In a production system, you’ll want a static IP for your VM and create a startup script to start the container. This will take care of server restarts but doesn’t help so much for server crashes.
To make the app resilient against server crashes, you can run the container inside a pod managed by Kubernetes. The process for doing this will be the topic of our next blog post.
Last year, we announced container-native load balancing, a feature that allows you to create services using network endpoint groups (NEGs) so that requests to your service get load balanced directly to the containers serving the requests. Since announcing the beta, we have worked hard to improve the performance, scalability and user experience of container-native load balancing and are excited to announce that it is now generally available.
Container-native load balancing removes the second hop between virtual machines running containers in your Google Kubernetes Engine (GKE) cluster and the containers serving the requests, improving efficiency, traffic visibility and container support for advanced load balancer capabilities. The NEG abstraction layer that enables this container-native load balancing is integrated with the Kubernetes Ingress controller running on Google Cloud Platform (GCP). If you have a multi-tiered deployment where you want to expose one or more services to the internet using GKE, you can also create an Ingress object, which provisions an HTTP(S) load balancer and allows you to configure path-based or host-based routing to your backend services.
Figure 1. Ingress support with instance groups vs. with network endpoint groups.
Improvements in container-native load balancing
Thanks to your feedback during the beta period, we’ve made several improvements to container-native load balancing with NEGs. In addition to having several advantages over the previous approach (based on IPTables), container-native load balancing now also includes:
Latency improvements: The latency of scaling down your load-balanced application to zero pod backends and then subsequently scaling back up is now faster by over 90%. This significantly improves response times for low-traffic services, which can now quickly scale back up from zero pods when there’s traffic.
Improved Kubernetes integration: Using the Kubernetes pod readiness gate feature, a load-balancer backend pod is considered ‘Ready’ once the Load balancer health check for the pod is successful and the pod is healthy. This ensures that rolling updates will proceed only after the pods are ready and fully configured to serve traffic. Now, you can manage the load balancer and backend pods with native Kubernetes APIs without injecting any unnecessary latency.
Standalone NEGs (beta): You can now manage your own load balancer (without having to create an HTTP/S based Ingress on GKE) using standalone NEGs, allowing you to configure and manage several flavors of Google Cloud Load Balancing. These include TCP proxy or SSL proxy based load balancing for external traffic, HTTP(S) based load balancing for internal traffic (beta) and global load balancing using Traffic Director for internal traffic. You can also create a load balancer with hybrid backends (GKE pods and Compute Engine VMs) or a load balancer with backends spread across multiple GKE clusters.
Getting started with container-native load balancing
Google Cloud’s technology powers both our customers as well as our internal teams. Recently, the Solutions Architect team decided to move an internal process to use BigQuery to streamline and better focus efforts across the team.
The Solutions Architect team publishes reference guides for customers to use as they build applications on Google Cloud. Our publishing process has many steps, including outline approval, draft, peer review, technical editing, legal review, PR approval and finally, publishing on our site. This process involves collaboration across the technical editing, legal, and PR teams.
With so many steps and people involved, it’s important that we effectively collaborate. Our team uses a collaboration tool running on Google Cloud Platform (GCP) as a central repository and workflow for our reference guides.
Increased data needs required more sophisticated tools
As our team of solution architects grew and our reporting needs became more sophisticated, we realized that we couldn’t effectively provide the insights that we needed directly in our existing collaboration tool. For example, we needed to build and share status dashboards of our reference guides, build a roadmap for upcoming work, and analyze how long our solutions take to publish, from outline approval to publication. We also needed to share this information outside our team, but didn’t want to share unnecessary information by broadly granting access to our entire collaboration instance.
Building a script with BigQuery on the back end
Since our collaboration tool provides a robust and flexible REST API, we decided to write an export script which stored the results in BigQuery. We chose BigQuery because we knew that we could write advanced queries against the data and then use Data Studio to build our dashboards. Using BigQuery for analysis provided a scalable solution that is well-integrated into other GCP tools and has support for both batch and real-time inserts using the streaming API.
We used a simple Python script to read the issues from the API and then insert the entries into BigQuery using the streaming API. We chose the streaming API, rather than Cloud Pub/Sub or Cloud Dataflow, because we wanted to repopulate the BigQuery content with the latest data several times a day. The Google API Python client library was an obvious choice, because it provides an idiomatic way to interact with the Google APIs, including the BigQuery streaming API.
Since this data would only be used for reporting purposes, we opted to keep only the most recent version of the data as extracted. There were two reasons for this decision:
Master data: There would never be any question about which data was the master version of the data.
Historical data: We had no use cases that required capturing any historical data that wasn’t already captured in the data extract.
Following common extract, transform, load (ETL) best practices, we used a staging table and a separate production table so that we could load data into the staging table without impacting users of the data. The design we created based on ETL best practices called for first deleting all the records from the staging table, loading the staging table, and then replacing the production table with the contents.
When using the streaming API, the BigQuery streaming buffer remains active for about 30 to 60 minutes or more after use, which means that you can’t delete or change data during that time. Since we used the streaming API, we scheduled the load every three hours to balance getting data into BigQuery quickly and being able to subsequently delete the data from the staging table during the load process.
Once our data was in BigQuery, we could write SQL queries directly against the data or use any of the wide range of integrated tools available to analyze the data. We chose Data Studio for visualization because it’s well-integrated with BigQuery, offers customizable dashboard capabilities, provides the ability to collaborate, and of course, is free.
Because BigQuery datasets can be shared with users, this opened up the usability of the data for whomever was granted access and also had appropriate authorization. This also meant that we could combine this data in BigQuery with other datasets. For example, we track the online engagement metrics for our reference guides and load them into BigQuery. With both datasets in BigQuery, it made it easy to factor in the online engagement numbers to build dashboards.
Creating a sample dashboard
One of the biggest reasons that we wanted to create reporting against our publishing process is to track the publishing process over time. Data Studio made it easy to build a dashboard with charts, similar to the two charts below. Building the dashboard in Data Studio allowed us to easily analyze our publication metrics over time and then share the specific dashboards with teams outside ours.
Monitoring the load process
Monitoring is an important part of any ETL pipeline. Stackdriver Monitoring provides monitoring, alerting and dashboards for GCP environments. We opted to use the Google Cloud Logging module in the Python load script, because this would generate logs for errors in Stackdriver Logging that we could use for error alerting in Stackdriver Monitoring. We set up a Stackdriver Monitoring Workspace specifically for the project with the load process. We then created a management dashboard to track any application errors. We set up alerts to send an SMS notification whenever errors appeared in the load process log files. Here’s a look at the dashboards in the Stackdriver Workspace:
And this shows the details of the alerts we set up:
BigQuery provides the flexibility for you to meet your business or analytical needs, whether they’re petabyte-sized or not. BigQuery’s streaming API means that you can stream data directly into BigQuery and provide end users with rapid access to data. Data Studio provides an easy-to-use integration with BigQuery that makes it simple to develop advanced dashboards. The cost-per-query approach means that you’ll pay for what you store and analyze, though BigQuery also offers flat-rate pricing if you have a high number of large queries. For our team, we’ve been able to gain considerable new insights into our publishing process using BigQuery, which have helped us both refine our publishing process and focus more effort on the most popular technical topics.
Today, we are honored to share that Cloud Build, Google Cloud’s continuous integration (CI) and continuous delivery (CD) platform, was named a Leader in The Forrester Wave™: Cloud-Native Continuous Integration Tools, Q3 2019. The report identifies the 10 CI providers that matter most for continuous integration (CI) and how they stack up on 27 criterias. Cloud Build received the highest score in both the current offering and strategy categories.
“Google Cloud Build comes out swinging, matching up well with other cloud giants. Google Cloud Build is relatively new when compared to the other public cloud CI offerings; this vendor had a lot to prove, and it did…. Customer references are happy to trade the cost of paying for the operation and management of build servers for moving their operations to Google’s pay-per-compute system. One reference explained that it’s in the middle of a cloud shift and is currently executing up to 650 builds per day. With that proved out, this customer plans to move an additional 250 repositories to the cloud” – Forrester Wave™ report
Top score in the Current Offering category: Among all 10 CI providers evaluated in the Wave, Cloud Build got the highest score in the Current Offering category. The Current Offering score is based on Cloud Build’s strength in the developer experience, build speed and scale, enterprise security and compliance, and enterprise support, amongst other criteria.
Top score in the Strategy category: Along with top score in the current offering category, Cloud Build also received the highest score in the Strategy category. In particular, Cloud Build’s scores in the partner ecosystem, commercial model, enterprise strategy and vision, and product roadmap criteria contributed to the score.
CI plays an increasingly important role in DevOps, allowing enterprises to drive quality from the start of their development cycle. The report mentions “that cloud-native CI is the secret development sauce that enterprises need to be fast, responsive, and ready to take on incumbents and would-be digital disruptors.”
At Google, we’ve seen first-hand how cloud-based serverless CI tool can help drive quality and security at scale, and the lessons we’ve learned in that process manifest directly in Cloud Build.
Today, organizations of all sizes use Cloud Build to drive productivity improvements via automated, repeatable CI processes. Some customers include Zendesk, Shopify, Snap, Lyft, and Vendasta, who chose Cloud Build for its:
Fully serverless platform: Cloud Build scales up and scale down in response to load with no need to pre-provision servers or pay in advance for additional capacity.
Flexibility: With custom build steps and pre-created extensions to third party apps, enterprises can easily tie their legacy or home-grown tools as a part of their build process.
Security and compliance features: Developers have an ability to perform deep security scans within the CI/CD pipeline and ensure only trusted container images are deployed to production.
We’re thrilled by Forrester’s recognition for Cloud Build. You can download a copy of the report here.
Whether they’re lifting-and-shifting workloads or taking cloud-native approaches to application development, enterprises are increasingly looking to the cloud to build and grow their businesses. As a result, over the past 12 months we’ve seen extraordinary momentum in Germany, Austria, and Switzerland—also known as the DACH region—as more and more businesses take advantage of Google Cloud. In March, we launched a new Google Cloud region in Zurich to support businesses in the region. We continue to expand our support of regional and global certifications and standards, now including FINMA-regulated customers in Switzerland. And we continue to work closely with many enterprises in the region, such as METRO AG, and many more.
Yesterday’s Cloud Summit in Munich is a great example of the continuing momentum we’re seeing. With 2200 attendees and 45 customer speakers, the summit was an important opportunity for us to connect with—and learn from—businesses in the region. Customers across every industry shared with us how they’re transforming with the power of the cloud. Here are a few highlights.
DLR: Groundbreaking robotics research with the help of Google Cloud infrastructure Part of Deutsches Zentrum für Luft- und Raumfahrt(DLR), Germany’s national aerospace center, the Institute of Robotics and Mechatronics is one of the largest pure robotics research institutes in the world. To advance its work, the institute relies on deep learning techniques, but this requires a substantial amount of compute resources for training and running simulations. Thanks to Google Cloud, it can now easily configure powerful instances with Compute Engine, even using hundreds of CPU cores if necessary. And it now has the flexibility to spin up specialized configurations when needed—something it couldn’t do with in-house servers. You can learn more in DLR’s case study.
“With Google Cloud, we can train our robots between five to ten times faster than before and really explore things like deep reinforcement learning,” says Berthold Bäuml, Head of Autonomous Learning Robots Lab, DLR. “We can do cutting edge research and we’re no longer bound by our computing resources. It’s providing totally new opportunities for us.”
ARD: From monolithic architecture to modern microservices A joint organization of Germany’s regional public-service broadcasters, the ARD network produces key national and regional television and radio programs, and faces challenges shared by many media organizations aiming to serve customers with the content they want, where and when they want it. Starting from a monolithic, rigid structure, ARD wanted a technology stack that fit its own culture—open and scalable. With Google Cloud, ARD is able to focus on five digital, scalable core products and take advantage of fully-managed services which has allowed it to increase its number of releases from once a year to 50 releases a day. In addition to using GCP as a scalable underlying platform, ARD is also using an API-as-a-service approach with Apigee to rethink the way they are delivering content to German citizens by making content delivery universally accessible to other broadcasting entities.
Deutsche Börse Group: Embracing a multi-cloud approach to infrastructure As an international exchange organization and market infrastructure provider, Deutsche Börse Group offers its customers a wide range of products, services and technologies covering the entire value chain of financial markets. As a result, Deutsche Bӧrse Group has frequently been an early adopter of new technologies that drive its industry forward, whether that’s using distributed ledger/blockchain technology in new ways or offering new advanced analytics and artificial intelligence (AI) services to clients. Most recently, Deutsche Bӧrse Group has ramped up its cloud-first strategy, led by its inaugural Chief Cloud Officer Michael Girg, choosing Google Cloud to help them modernize, develop and operate its enterprise workloads in a more efficient and secure way that also supports compliance.
“As a key technology, cloud lays the foundation for enabling some of Deutsche Börse’s major initiatives focussing on new technologies,” says Michael Girg, Chief of Cloud Officer at Deutsche Börse. “Together with Google Cloud as a strong partner, we are very much looking forward to accelerating cloud adoption and to jointly define innovative solutions that further push ahead data security for the financial industry”
You can read more on their progress in our recent blog post.
MediaMarktSaturn Retail Group: Transforming retail, online and offline One of the world’s largest consumer electronics retailers, the MediaMarktSaturn Retail Group operates more than 1,000 stores across 15 countries in Europe, including a sizeable ecommerce presence. But the company was increasingly finding that its on-premises infrastructure, used to first built its online stores, was not able to keep up with evolving business and customer demands. So MediaMarktSaturn turned to the cloud, upgrading its infrastructure but also to adopting a new way of working that places technology at the heart of its strategy. For MediaMarktSaturn, this meant combining data streams into a new data lake from which it can query data with and derive insights with ease, as well as adopting Google Kubernetes Engine (GKE), to manage the Kubernetes clusters that form the backbone of its new online system.
“Google Cloud has paved the way for us to break new ground not only technically, but also in our way of working,” says Dr. Johannes Wechsler, Managing Director, MediaMarktSaturn Technology. “Thanks to this strong partnership, we are well equipped to offer our customers a great user experience even in high-load situations and at the same time deliver new features more quickly.”
Looking ahead We’re excited to see what enterprises in Germany, Austria, and Switzerland can do as they embrace Google Cloud. We remain committed to working with them to help them grow and digitally transform their businesses.
Editor’s Note:This is the third blog in our six-part series on how to use Cloud Security Command Center. There are links to the first two blogs in the series at the end of this post.
When a threat is detected, every second counts. But, sometimes it can be difficult to know if a threat is present or how to respond. Cloud Anomaly Detection is a built-in Cloud Security Command Center (Cloud SCC) feature that uses behavioral signals to detect security abnormalities, such as leaked credentials or unusual activity, in your GCP projects and virtual machines. In this blog, and the accompanying video, we’ll look at how to enable Cloud Anomaly Detection and quickly respond to threats.
1. Enable Cloud Anomaly Detection from Cloud Security Command Center Cloud Anomaly Detection is not turned on by default. You need to go to Security Sources from the Cloud SCC dashboard and activate it. Keep in mind, to enable a security source, you need to have the Organization Administrator Cloud IAM role. Once it’s turned on, findings will automatically be surfaced and displayed in the Cloud Anomaly Detection card on the Cloud Security Command Center dashboard.
2. View findings in Cloud Security Command Center
Cloud Anomaly Detection can surface a variety of anomalous findings, including:
Leaked service account credentials: GCP service account credentials that are accidentally leaked online or compromised.
Resource used for outbound intrusion: One of the resources or GCP services in your organization is being used for intrusion activities, like an attempt to break in to or compromise a target system. These include SSH brute force attacks, Port scans, and FTP brute force attacks.
Potential compromised machine: A potential compromise of a resource in your organization.
Resource used for crypto mining: Behavioral signals around a VM in your organization indicate that it might have been compromised and could be getting used for crypto mining.
Unusual Activity/Connection: Unusual activity from a resource in your organization.
Resource used for phishing: One of the resources or GCP services in your organization is being used for phishing.
3. Remediate findings from Cloud Security Command Center After Cloud Anomaly Detection generates a finding, you can click on the finding for more information about what happened and use that information to fix the security issue.
To learn more about Cloud Anomaly Detection, including how to turn it on and how it can help your organization, check out the video below.
Posted by Nafis Zebarjadi, Product Manager and Adam Dawes, Senior Product Manager
Project Strobe was started to help users have control over their data while giving developers more explicit rules of the road to ensure everyone is confident that their data is secure. One result of this effort has been to expand our app verification program to cover more apps and more types of data access. It is important to understand how the process works so that you can optimally build your app and streamline the verification process. Here we walk you through the process of preparing your app for OAuth verification.
Getting prepared for verification
The first thing you should do is confirm whether your app needs verification. App verification is only required if you want to launch your app widely to consumer or enterprise users and the app requests sensitive or restricted scopes. Apps that use non-sensitive scopes, are under development, or are built just for your own G Suite users are not required to go through verification. If the app is just for users within your own organization, choose the ‘Internal’ application type to restrict the app to use within your own organization and skip verification.
Once you initiate app verification, it is not easy to make updates to your app’s Google API configuration. If you make any changes while in the process, you will need to start over again, so it’s critical that you get your app ready before initiating verification to avoid delays.
Determining if your app is using sensitive or restricted scopes
The first thing you need to do is look at your code on each platform to determine which OAuth scopes (Google APIs) your service needs. Be sure to do this on every client; we often see that apps will request different scopes on different platforms, and then initiate app verification on a subset of scopes than your clients actually use. Often, you can find the scopes by searching your code for the string “www.googleapis.com/auth”. Not all legacy scopes contain that string so you may also want to find the code related to the Google API library you’re using (on the specific platform) to see what scopes are being requested, or look at our directory of scopes.
Once you have identified all of the scopes that your apps use, you can check to see whether they are sensitive or restricted by going to the Cloud Console (APIs & Services -> Credentials -> OAuth consent screen -> Scopes for Google APIs) and pressing the ‘Add scope’ button. This will bring up the following window:
If the scope has a lock icon, it means that the scope is either sensitive or restricted and that you’ll need to go through app verification before you can widely launch to Google users.
[Note that the tool only lists scopes for APIs that you’ve enabled for your project. If you don’t see a scope listed, you’ll first need to enable the corresponding API for your project from the API Library. The fact that you’re not seeing the scope used in your code may mean that you have clients set up in different projects.]
Setting up the right project structure
Apps are reviewed and approved at the project level so you’ll want to make sure that you’ve configured your clients properly before starting app verification. If you have multiple projects, each one will have to independently go through app verification.
When to add multiple clients to a project: You may have multiple clients for your app to support different platforms like Android, Web and iOS. Ideally, all of these clients should be in the same project because it will smooth out the cross-client consent experience. When clients are in the same project, users only need to provide consent to one of the clients. Other clients can automatically get tokens without forcing the user to go through the consent flow for the same requested scopes again. The user is agreeing to share data with your service regardless of which platform they happen to be using and your service terms should be the same across platforms.
When to separate clients into separate projects: Your company may also have multiple apps that you publish to users. You may or may not want to host the clients related to your different apps in the same project. Generally, if the different apps use the same login system, have the same privacy policy and users recognize the brand of the publisher of all the apps, then it makes sense to have all the clients in the same project. For example, if PersonalFinance Corp has accounting, budgeting and tax apps that all share the same login, privacy policy and users recognize the PersonalFinance Corp brand, then it is best to structure those all in the same project. However, if CoolGames publisher has lots of titles that have different login systems and different privacy policies, or users are more familiar with the individual game titles than the CoolGames brand, then you should use separate projects.
Reorganizing projects: It is not possible to move or reorganize clients once they are created. If you want to make changes, you can either choose to create new clients in a centralized project or get each app verified independently. If you create new clients in a centralized project, you’ll update your apps to use the new client and abandon the old clients. The issue you may encounter with this approach is that your app may have to obtain user consent all over again (if the user hasn’t also consented to your other client). Alternately, you can leave your clients in separate projects; however, each project will have to go through app verification independently and users will have to consent to each of your clients individually.
Setting up test vs production projects: For many developers, it is also helpful to have a parallel test project to your production project. This allows you to easily change scopes or other app properties and test behavior without having to go through app verification.
Configuring your project
If your app does need to be verified, you’ll want to make sure the information about your project is up-to-date to avoid delay.
Project Owners
As we roll out changes across our API ecosystem, it is important to make sure your projects have up-to-date contact information. We often need to send notifications about changes, and have had developers miss important updates because of incorrect contact information which has resulted in their app being unexpectedly disabled. One way to help ensure your team gets notifications is to create a Google Group that aliases to a stable group within your company (and be sure to configure the group to receive emails from non-members). Another option is to create an Organizational Resource in the Cloud Console so that your client assets can be centrally administered and recovered when owners leave the company. It’s also very good practice to ensure the owners of the Android/iOS/Web clients are also owners or editors of the project. Domain verification is also required for every app, so you will also want to add your DNS administrator to the project so that person can easily go through the process.
To update project owners, use Cloud IAM in the Cloud Console (Cloud Console -> IAM and admin -> IAM).
Branding Info and Domain Verification
Branding info includes your app’s name and logo. It is critical that these are accurate because users use these to decide whether they know and trust your app. In the verification process, we will validate that you own the brand and logo and that it matches the information on your web site. If you make changes, your previously approved branding will continue to be shown until the new information can be verified.
You will also need to verify the domain associated with your brand. This is true even if you only have Android/iOS versions of your app because you must have a website to publicly host your privacy policy. You start the domain verification process by linking your domain to your project in the Cloud Console (APIs & Services -> Domain verification). You’ll then need to go to the Search Console to prove that you own and control the domain.
Domain verification is a key security feature for your web clients. If you have web clients in your project, each of those must have their Authorized Redirect URIs or Authorized JavaScript Origins match an already verified domain. This enables us to guarantee that OAuth tokens are only returned to your application.
Scopes
Since you’ve already identified the scopes that your app uses, you should now check to see if you can change scopes to minimize your data access. Our API User Data Policy requires that you only request information that your app needs and that you’re clear to the user about how you will use it. It’s inappropriate to gain access to Google user data for alternate purposes such as advertising and market research.
In particular, you’ll want to try to avoid the use of restricted scopes. The verification process for restricted scopes can take several weeks longer than sensitive scopes. It also requires significant documentation and may involve a third-party security assessment that you must pay for. Currently, only specific Gmail scopes are restricted, but we have announced that most Drive scopes are also becoming restricted in early 2020.
If your app does need to access a restricted scope, consider architecting your app such that the Google user data is only ever stored client-side on the user’s device (like a contact manager app). Storing data in the cloud or on your own servers will require you to obtain a third-party security assessment (at your expense), and could also result in significant work to resolve any security issues found during the assessment.
Once you’ve decided on the scopes your app will need, make sure that they are registered with your project and reflected in your app’s code. We’ve seen many cases where a developer’s code calls a different set of scopes than those that have been registered in the Cloud Console. If your app does this, your users will see an unverified app error. Many developers request troubleshooting help because their users are unexpectedly seeing these errors even though their app was approved. Inevitably, it is because their code does not match what was verified. Similarly, if you need to add new scopes to your application, you’ll need to get those scopes approved before you launch the functionality into your production app (a test client is going to be essential here).
While you’re thinking about scopes, you should also consider how and when you are asking your users for consent. The best practice is to not request scopes at sign-in, but to use incremental authorization to allow a user to access a particular feature when they want it. This is a great way to build trust because the user interacts in a particular feature, can see the benefit of the feature, and understands why granting a particular permission will make the feature more useful.
Privacy Policy
Our goal in verifying apps is to ensure that any data users choose to share with third-parties is well-managed and meets users’ expectations about how it will be used. Your privacy policy is your public contract to your users and a critical proof to us that users’ expectations will be met.
You must include a link to your privacy policy on your website. If the domain where you host that policy isn’t verified, we won’t verify your app. If your app is purely mobile, with no server-side component, you will still need a privacy policy, but it may be very simple and describe that your app only stores data on a user’s device.
Google can not provide guidance on your privacy policy, but if your app requests restricted scopes, we will scrutinize your policy to understand how you plan to use that data and ensure that it conforms to our requirements. Make sure you understand the Limited Use requirements, and consult with your legal counsel to ensure that your privacy policy is consistent with the requirements. To ensure clarity in how your app handles email content, we also recommend adding the following statement to your application’s home page: “App’s use of information received from Gmail APIs will adhere to Google’s Limited Use Requirements.” This is needed when your privacy policy is not specific in how email content is used.
Submitting your app for verification
Once you have your project(s) configured with all the appropriate information, you can submit your app for verification. We have three different types of app verification depending on the scopes you request, each taking a different amount of time to complete. If you start your verification with one set of scopes and later decide you need different scopes, you usually need to finish your existing verification before you can start the process again. This could cause frustration and lengthen your overall verification process.
Brand Verification (2-3 days)
Brand Verification is our simplest process and validates that your brand name and logo belong to you. It is an optional step if your app is requesting non-sensitive scopes like Google Sign-In and typically takes just 2 to 3 business days. If your app doesn’t go through brand verification, users will only see your domain name listed on the consent page.
Sensitive Scope Verification (3-5 days)
Starting in June 2019, we greatly expanded the classification of sensitive scopes and started requiring more extensive verification for new apps that are accessing those scopes. Existing apps that are already accessing sensitive scopes need to go through this verification process in the latter half of 2019.
Sensitive scope verification involves brand and domain validation, checking that the privacy policy is prominently available from your application home page. We also review your app and privacy policy against our API Services: User Data Policy and check for deceptive practices. The privacy policy must disclose the manner in which your application accesses, uses, stores, or shares Google user data. Your use of Google user data must be limited to the practices disclosed in your published privacy policy.
A YouTube or accessible Drive video will also be required to understand how users will experience your request for scopes, showing specifically how they’ll benefit from granting you access. The identity of your app needs to be clear from the video (including the app’s client ID), and you’ll need to highlight the value proposition you communicate to the user before requesting the scopes.
Until verification is completed, users will see an unverified app page when your app requests a scope requiring verification. Up to 100 users may choose to grant access while your app is unverified. After that, users will be blocked from granting access to your app until verification is complete.
Sensitive scope verification usually takes 3 to 5 business days if there aren’t any issues with your app.
Restricted Scope Verification (4-6 weeks)
Restricted scope verification is a much more involved process. In addition to going through all the steps for a sensitive scope verification, your app will also have a much more rigorous privacy policy review to ensure that your use of Google user data conforms to our Limited Use requirements. Only permitted application types will be considered for access to restricted scopes. Finally, if your app stores data on a server, you will need to pass an annual security assessment.
We do error validation before allowing you to click the ‘Submit for Verification’ button. Here are some common reasons why the button is not clickable:
No verified domain
Privacy policy URL, authorized redirect URIs or origins for your client do not match an authorized domain
No new scopes added to the project that require verification
When you submit your app for verification, you will need to provide a written explanation for why your app needs the requested scopes. This explanation should include the nature of the feature and how the user will benefit from using it. It’s also best to include a link to your YouTube video in the original submission to save some back and forth with the review team.
You’ll also be asked again what email should receive questions and notifications about the verification process. Make sure you provide an address that you pay attention to and can receive emails from outside your domain. Questions will go to the person who initiated verification (not necessarily project owners) and the contact email address provided in the verification form. We’ve seen many requests delayed because the developer hasn’t responded to questions from the verification team.
Responding to verification questions
Apps with sensitive and restricted scopes often need to answer questions from the verification team. If you believe it has taken a long time to get a response from the verification team, you should search your inbox for messages from ‘api-oauth-dev-verification-reply’ to ensure that you haven’t missed anything.
By following these guidelines for submitting your app for verification, you can greatly streamline the process of getting your app approved and released to the Google user community. If you have any follow-up questions, be sure to scan the OAuth API Verification FAQ.
Editor’s note: Today’s post comes from Sarah Potenza, executive director of Worldwide Opportunities on Organic Farms, USA (WWOOF-USA), a nonprofit whose map-based website helps travelers find and work on organic farms.
Organic farming allows farmers to grow food in a more sustainable manner and takes care of the planet. To help build a community of people who are interested in ecological farming practices, we connect travelers who want to visit and work on organic farms with the farms themselves. Visitors work on farms for a day or more in return for room and board. We work with 2,134 organic farms and gardens in all 50 U.S. states, the Virgin Islands and Puerto Rico. We’re part of a worldwide WWOOF association founded in the 1970s.
When WWOOF started, we printed a book of the participating organic farms which we mailed to people who paid an annual fee to become WWOOF members. Members used the book to decide which farms they wanted to visit and work. Printing and mailing the book was expensive, and because of production limitations and mailing times, the books didn’t have the most up-to-date listing of participating farms.
As the Internet became popular, we realized we could use it to solve those problems because it could eliminate production and mailing delays, and offer instant information to people in ways never before possible. So we built a map-based website that would allow members to more easily find the farms they wanted to visit and get more information in a timely way about each farm. It would also be an effective means of outreach, allowing those who might be considering WWOOF membership to see the range of farms they could visit if they joined.
We chose Google Maps Platform to add maps and location data to our website because its APIs could easily link to our backend systems, it has the most comprehensive and accurate mapping data, and its interface has become the standard for maps. We used the Google Maps JavaScript API, the MarkerClusterer library and the Google Maps Geocoding API to build an interactive website that allows subscribers to browse or search for WWOOF-associated farms anywhere in the United States.
Site visitors can go to the Our Farms page and zoom in on any area of the country they want to visit and see markers for each host organic farm. They can use filters to search for farms, including the length of time they want to stay, the type of farm (vegetable, orchard, vineyard, dairy, among others), which farms are immediately available to visit, and whether children and pets are allowed.
Clicking a marker brings up a description of the farm and shows full details, including photos and reviews from people who have stayed there. Only those who’ve paid to become WWOOF members can see the details; non-members can’t open the profiles. We’ve found that allowing only members to see the breadth of farms available is a great way to get website visitors interested in joining WWOOF. It encourages people who haven’t yet signed up to become members.
Using Google Maps Platform has significantly increased our site’s usability. If we weren’t using Google Maps Platform APIs, we’d have only a static list of farms that is far less inviting to users than being able to browse and search on a map-based interface. As a result of using Google Maps Platform, the site received more than 8 million pageviews in 2018.
Since integrating Google Maps Platform, we’ve seen a dramatic increase in the number of people who pay to become members. Just after we started using Google Maps, we had 6,400 paying members. We now have more than 18,000.
Getting access to Google Maps Platform credits has been a key to WWOOF’s growth. Google for Nonprofits gave us the ability to apply for credits for Google Maps Platform API usage, allowing us to provide subscribers with information about farms in an affordable way. We would not have been able to handle the number of people who view the maps without the grant. I’d recommend Google for Nonprofits to any nonprofit organization.
Today, we’re excited to announce the general availability (GA) of virtual display devices for Compute Engine virtual machines (VMs), letting you add a virtual display device to any VM on Google Cloud. This gives your VM Video Graphics Array (VGA) capabilities without having to use GPUs, which can be powerful but also expensive.
Many solutions such as system management tools, remote desktop software, and graphical applications require you to connect to a display device on a remote server. Compute Engine virtual displays allow you to add a virtual display to a VM at startup, as well as to existing, running VMs. For Windows VMs, the drivers are already included in the Windows public images; and for Linux VMs, this feature works with the default VGA driver. Plus, this feature is offered at no extra cost.
We’ve been hard at work with partners Itopia, Nutanix, Teradici and others to help them integrate their remote desktop solutions with Compute Engine virtual displays to allow our mutual customers to leverage Google Cloud Platform (GCP) for their remote desktop and management needs.
“We needed a cloud provider that could effectively support both our 3D modelling and our artificial intelligence requirements with remote workstations,” said Michael Diener, Engineering Manager for StrucInspect. “Google Cloud was well able to handle both of these applications, and with Teradici Cloud Access Software, our modelling teams saw a vast improvement in virtual workstation performance over our previous solution. The expansion of GCP virtual display devices to support a wider range of use cases and operating systems is a welcome development that ensures customers like us can continue to use any application required for our client projects.”
Our partners are equally excited about the general availability of virtual display devices.
“We’re excited that the GCP Virtual Display feature is now GA because it enables our mutual customers to quickly leverage Itopia CAS with Google Cloud to power their Virtual Desktop Infrastructure (VDI) initiatives,” said Jonathan Lieberman, itopia Co-Founder & CEO.
“With the new Virtual Display feature, our customers get a much wider variety of cost-effective virtual machines (versus GPU VMs) to choose from in GCP,” said Carsten Puls, Sr. Director, Frame at Nutanix. “The feature is now available to our joint customers worldwide in our Early Access of Xi Frame for GCP.”
Now that virtual display devices is GA, we welcome you to start using the feature in your production environment. For simple steps on how you can use a virtual display device when you create a VM instance or add it to a running VM, please refer to the documentation.
Google Cloud Platform (GCP) firewall rules are a great tool for securing applications. Firewall rules are customizable software-defined networking constructs that let you allow or deny traffic to and from your virtual machine (VM) instances.. To secure applications and respond to modern threats, firewall rules require monitoring and adjustment over time. GCP Firewall Rules Logging,Google made generally available in February 2019, is a feature that allows the network administrators to monitor, verify and analyze the effects of firewall rules in Google Cloud. In this blog (the first of many on this topic), we’ll discuss the basics of Firewall Rule Logging, then look at an example of how to use it to identify mislabeled VMs and refine firewall rules with minimal traffic interruption.
GCP Firewall Rules Logging: The Basics
Firewall Rules Logging provides visibility to help you better understand the effectiveness of rules in troubleshooting scenarios. It helps answer common questions, like:
How can I ensure the firewall rules are doing (or not doing) what they were created for?
How many connections match the firewall rules I just implemented?
Are firewall rules the root cause of some application failures?
Unlike VPC flow logs, firewall rules logs are not sampled. Every connection is logged (subject to some limits. Please refer to the Appendix for details). The Firewall Rule log format can be found here.
Additionally, network administrators have the options to export firewall logs toGoogle Cloud Storage for long term log retention,to BigQuery for in-depth analysis using standard SQL, or to Pub/Sub to integrate with popular security information and event management software (SIEM), such assplunk for detecting/alerting traffic abnormalities and threats at near real time.
For reference, GCP firewall rules are software-defined constructs with the following properties:
GCP firewalls are VM-centric. Unlike traditional firewall devices, which are applied at the network edge, GCP firewall rules are implemented at VM level. This means the firewall rules can exist between your instances and other networks, and also between individual instances within the same VPC.
GCP firewall rules always have targets. The targets are considered source VMs when defining egress firewall rules, and destination VMs when defining ingress firewall rules. Do not confuse “target” with the “destination” in the traditional firewall concept.
GCP firewall rules are defined within the scope of aVPC network. There is no concept of subnets when defining firewall rules. However, you can specify source CIDR ranges, which give you better flexibility than subnets.
Every VM has two immutable implied firewall rules: implied allow of egress, and implied deny of ingress at lowest priority. However, Firewall Rule Logging does not generate any entries for these implied firewall rules.
While GCP firewall rules support many protocols—including TCP, UDP, ICMP, ESP, AH, SCTP, and IPIP—Firewall Rule Logging only logs entries for TCP and UDP connections.
Firewall Best Practices
Follow the least privilege principle — make firewall rules as tight as possible. Only allow well-documented and required traffic (ingress and egress), and deny all others. Use a good naming convention to indicate each firewall rule’s purpose.
Use fewer and broader firewall rule sets when possible. Observe the standard quota of 200 firewall rules per project. The complexity of the firewall also matters. A good rule of thumb is to not throw too many atoms (tags/service accounts, protocols/ports, source/destination ranges) at the firewall rules. Please refer to the Appendix for more on firewall quota/limits.
Progressively define and refine rules; start with the broader rules first, and then use rules to narrow down to a smaller set of VMs.
Isolate VMs using service accounts when possible. If you can’t do that, use network tags instead. But do not use both. Service account access is tightly controlled by IAM. While network tags are more flexible, anyone with the instanceAdmin role can change them. More on filtering using service accounts versus network tags can be foundin our firewall rules overview.
Conserve network space by planning proper CIDR blocks (segmentations) for your VPC network to group related applications in the same subnet.
Use firewall rule logging to analyze traffic, detect misconfigurations, and report real-time abnormalities.
In practice, there are many uses for Firewall Rule Logging. One common use case is to help identify mislabeled VM instances. Let’s walk through this scenario in more detail.
Scenario: Mislabeled VM instances
ACME, Inc. is migrating on-prem applications to Google Cloud. Their network admins implemented a shared VPC to centrally manage the entire company’s networking infrastructure. There are dozens of applications, run by multiple engineering teams, deployed in each GCP region with multi-tiered applications. User-facing proxies in the DMZ talk to the web servers, which communicate with the application servers, which, talk to database layers.
This diagram represents one of many regions, each of which has multiple subnets. This region, US-EAST1, includes:
acme-dmz-use1: 172.16.255.0/27
acme-web-use1: 10.2.0.0/22
acme-app-use1: 10.2.4.0/22
acme-db-use1: 10.2.8.0/22
Here are the traffic directions and firewall rules in place for this region:
Proxy can access web servers in acme-web-use1: firewall rule: acme-web-use1-allow-ingress-acme-dmz-use1-proxy
Web servers can access app servers in acme-app-use1: firewall rule: acme-app-use1-allow-ingress-acme-web-use1-webserver
App servers can access database servers in acme-db-use1: firewall rule: acme-db-use1-allow-ingress-acme-app-use1-appsvr
This setup has granular partitions of network space that categorize compute resources by application function. This makes it possible for firewall rules to control the network space and lock the network infrastructure to comply with the least-privilege principle.
For large organizations with thousands of VMs provisioned by dozens of application teams, we use service accounts or network tags to group the VMs, in conjunction with subnet CIDR ranges to define firewall rules. For simplicity, we use the network tags to demonstrate each use case.
The problem
It’s not unusual for large enterprises to have hundreds of firewall rules due to the scale of the infrastructure and the complexity of network traffic patterns.With so much going on, it’s understandable, that application teams sometimes mislabel the VMs when they migrate them from on-prem to cloud and scale-up applications after migration. The consequences of mislabeling range from an application outage to a security breach. The same problem can also arise if we (mis)use service accounts.
Going back to our example, an ACME application team mislabeled one of their new VMs as “appsrv” when it should actually be “appsvr”. As a result, the web server’s requests to access app servers are denied.
Solution 1
In order to identify the mislabeling and mitigate the impact quickly, we can enable the firewall rule logging for all the firewall rules using the following gcloud command.
Next, we export the logs to BigQuery for further analysis with following steps:
Create a BigQuery dataset to store the Firewall Rule Log entries:
Create the BigQuery sink to export the firewall rules logs:
You can also export individual firewall rules by adding filters on the jsonPayload.rule_details.reference. Here is a sample filter:
When the BigQuery sink is created, a service account is generated automatically by the GCP. The service account follows the naming convention of “p-[0–9]*@gcp-sa-logging.iam.gserviceaccount.com” and is used to write the log entries to the BigQuery table. We need to grant this service account the Data Editor role on the BigQuery dataset as following three steps:
Once the logs are populated to the BigQuery sink, you can run queries to determine which rule denies what as following:
The BigQuery table will be loaded with the log entries from the logFileName filter, which, in this case, contains all the firewall rules logs. The BigQuery table schema is directly mapped to the log entry’s json format. To keep it simple, for our example we’ll use the log viewer to inspect the log entries.
Since we have implicit ingress and the denial rule is not being logged, we create a “deny all” rule with priority 65534 to capture anything that gets denied.
The firewall rules for this scenario are:
As we can see in the viewer, “acme-deny-all-ingress-internal” is taking effect, and “acme-allow-all-ingress-internal” is disabled, so we can ignore it.
Below, we can see that the connection from websvr01 to the new appsvr02 (with the incorrect “appsrv” label) is denied.
While this approach works for this example, it presents two potential problems:
If we have a large amount of traffic, it will generate too much data for real-time analysis. In fact, one of our clients implemented this approach and ended up generating 5TB of firewall logs per day.
Mislabeled VMs can cause traffic interruptions. The firewall is doing what it is designed to do, but nobody likes outages.
So, we need a better approach to address both of the issues above.
Solution 2
To resolve the potential issues mentioned above, we can create another ingress rule to allow all traffic at priority 65533 and turn it on for a short period of time whenever there are new deployments.
In this scenario, we don’t need to turn on all the Firewall Rules Logging. In fact, we could turn off most of it to save space.
Any allowed evaluation logged by this firewall rule is a violator, and we do not expect many of them. The suspects are identified in real time.
Now, we fix the label.
As we can see, the connection from websvr01 to appsvr02 now works fine.
After all the mislabels are fixed, we can turn off the allow and capture rule. Everyone is happy… until the next time new resources are added to the network.
Conclusion
With Firewall Rules Logging, we can refine our firewall rules by following a few best-practices and identify undesired network traffic in near real-time. In addition to firewall rules logging, we’re always working on more tools and features to make managing firewall rules and network security in general easier.
Appendix Firewall quotas and limits
The default quota is 200 firewall rules per project, but can be bumped up to 500 through quota requests. If you need more than 200 firewall rules, we recommend that you review the firewall design to see whether there is a way to consolidate the firewall rules. The upcoming hierarchical firewall rules can define firewall rules at folders/org level, and are not counted toward the per-project limit.
The maximum number of source/target tags per firewall rule is 30/70, and the maximum number of source/target per service account is 10/10. A firewall rule can use network tags or service accounts, but not both.
As mentioned, the complexity of the firewall rules also matters. Anything that is defined in firewall rules, such as source ranges, protocol/ports, network tags, and service accounts, counts towards an aggregated per-network hard limit. This number is at the scale of tens of thousands, so it doesn’t concern most customers, except in rare cases where a large enterprise may reach this limit.
There are per-VM maximum number of logged connections in a 5-second interval depending on machine types: f1-macro (100), g1-small (250), and other machine types (500 per vCPU up to 4,000 in total).
Posted by Nafis Zebarjadi, Product Manager and Adam Dawes, Senior Product Manager
