Last year, we announced container-native load balancing, a feature that allows you to create services using network endpoint groups (NEGs) so that requests to your service get load balanced directly to the containers serving the requests. Since announcing the beta, we have worked hard to improve the performance, scalability and user experience of container-native load balancing and are excited to announce that it is now generally available.
Container-native load balancing removes the second hop between virtual machines running containers in your Google Kubernetes Engine (GKE) cluster and the containers serving the requests, improving efficiency, traffic visibility and container support for advanced load balancer capabilities. The NEG abstraction layer that enables this container-native load balancing is integrated with the Kubernetes Ingress controller running on Google Cloud Platform (GCP). If you have a multi-tiered deployment where you want to expose one or more services to the internet using GKE, you can also create an Ingress object, which provisions an HTTP(S) load balancer and allows you to configure path-based or host-based routing to your backend services.
Figure 1. Ingress support with instance groups vs. with network endpoint groups.
Improvements in container-native load balancing
Thanks to your feedback during the beta period, we’ve made several improvements to container-native load balancing with NEGs. In addition to having several advantages over the previous approach (based on IPTables), container-native load balancing now also includes:
Latency improvements: The latency of scaling down your load-balanced application to zero pod backends and then subsequently scaling back up is now faster by over 90%. This significantly improves response times for low-traffic services, which can now quickly scale back up from zero pods when there’s traffic.
Improved Kubernetes integration: Using the Kubernetes pod readiness gate feature, a load-balancer backend pod is considered ‘Ready’ once the Load balancer health check for the pod is successful and the pod is healthy. This ensures that rolling updates will proceed only after the pods are ready and fully configured to serve traffic. Now, you can manage the load balancer and backend pods with native Kubernetes APIs without injecting any unnecessary latency.
Standalone NEGs (beta): You can now manage your own load balancer (without having to create an HTTP/S based Ingress on GKE) using standalone NEGs, allowing you to configure and manage several flavors of Google Cloud Load Balancing. These include TCP proxy or SSL proxy based load balancing for external traffic, HTTP(S) based load balancing for internal traffic (beta) and global load balancing using Traffic Director for internal traffic. You can also create a load balancer with hybrid backends (GKE pods and Compute Engine VMs) or a load balancer with backends spread across multiple GKE clusters.
Getting started with container-native load balancing
Google Cloud’s technology powers both our customers as well as our internal teams. Recently, the Solutions Architect team decided to move an internal process to use BigQuery to streamline and better focus efforts across the team.
The Solutions Architect team publishes reference guides for customers to use as they build applications on Google Cloud. Our publishing process has many steps, including outline approval, draft, peer review, technical editing, legal review, PR approval and finally, publishing on our site. This process involves collaboration across the technical editing, legal, and PR teams.
With so many steps and people involved, it’s important that we effectively collaborate. Our team uses a collaboration tool running on Google Cloud Platform (GCP) as a central repository and workflow for our reference guides.
Increased data needs required more sophisticated tools
As our team of solution architects grew and our reporting needs became more sophisticated, we realized that we couldn’t effectively provide the insights that we needed directly in our existing collaboration tool. For example, we needed to build and share status dashboards of our reference guides, build a roadmap for upcoming work, and analyze how long our solutions take to publish, from outline approval to publication. We also needed to share this information outside our team, but didn’t want to share unnecessary information by broadly granting access to our entire collaboration instance.
Building a script with BigQuery on the back end
Since our collaboration tool provides a robust and flexible REST API, we decided to write an export script which stored the results in BigQuery. We chose BigQuery because we knew that we could write advanced queries against the data and then use Data Studio to build our dashboards. Using BigQuery for analysis provided a scalable solution that is well-integrated into other GCP tools and has support for both batch and real-time inserts using the streaming API.
We used a simple Python script to read the issues from the API and then insert the entries into BigQuery using the streaming API. We chose the streaming API, rather than Cloud Pub/Sub or Cloud Dataflow, because we wanted to repopulate the BigQuery content with the latest data several times a day. The Google API Python client library was an obvious choice, because it provides an idiomatic way to interact with the Google APIs, including the BigQuery streaming API.
Since this data would only be used for reporting purposes, we opted to keep only the most recent version of the data as extracted. There were two reasons for this decision:
Master data: There would never be any question about which data was the master version of the data.
Historical data: We had no use cases that required capturing any historical data that wasn’t already captured in the data extract.
Following common extract, transform, load (ETL) best practices, we used a staging table and a separate production table so that we could load data into the staging table without impacting users of the data. The design we created based on ETL best practices called for first deleting all the records from the staging table, loading the staging table, and then replacing the production table with the contents.
When using the streaming API, the BigQuery streaming buffer remains active for about 30 to 60 minutes or more after use, which means that you can’t delete or change data during that time. Since we used the streaming API, we scheduled the load every three hours to balance getting data into BigQuery quickly and being able to subsequently delete the data from the staging table during the load process.
Once our data was in BigQuery, we could write SQL queries directly against the data or use any of the wide range of integrated tools available to analyze the data. We chose Data Studio for visualization because it’s well-integrated with BigQuery, offers customizable dashboard capabilities, provides the ability to collaborate, and of course, is free.
Because BigQuery datasets can be shared with users, this opened up the usability of the data for whomever was granted access and also had appropriate authorization. This also meant that we could combine this data in BigQuery with other datasets. For example, we track the online engagement metrics for our reference guides and load them into BigQuery. With both datasets in BigQuery, it made it easy to factor in the online engagement numbers to build dashboards.
Creating a sample dashboard
One of the biggest reasons that we wanted to create reporting against our publishing process is to track the publishing process over time. Data Studio made it easy to build a dashboard with charts, similar to the two charts below. Building the dashboard in Data Studio allowed us to easily analyze our publication metrics over time and then share the specific dashboards with teams outside ours.
Monitoring the load process
Monitoring is an important part of any ETL pipeline. Stackdriver Monitoring provides monitoring, alerting and dashboards for GCP environments. We opted to use the Google Cloud Logging module in the Python load script, because this would generate logs for errors in Stackdriver Logging that we could use for error alerting in Stackdriver Monitoring. We set up a Stackdriver Monitoring Workspace specifically for the project with the load process. We then created a management dashboard to track any application errors. We set up alerts to send an SMS notification whenever errors appeared in the load process log files. Here’s a look at the dashboards in the Stackdriver Workspace:
And this shows the details of the alerts we set up:
BigQuery provides the flexibility for you to meet your business or analytical needs, whether they’re petabyte-sized or not. BigQuery’s streaming API means that you can stream data directly into BigQuery and provide end users with rapid access to data. Data Studio provides an easy-to-use integration with BigQuery that makes it simple to develop advanced dashboards. The cost-per-query approach means that you’ll pay for what you store and analyze, though BigQuery also offers flat-rate pricing if you have a high number of large queries. For our team, we’ve been able to gain considerable new insights into our publishing process using BigQuery, which have helped us both refine our publishing process and focus more effort on the most popular technical topics.
Today, we are honored to share that Cloud Build, Google Cloud’s continuous integration (CI) and continuous delivery (CD) platform, was named a Leader in The Forrester Wave™: Cloud-Native Continuous Integration Tools, Q3 2019. The report identifies the 10 CI providers that matter most for continuous integration (CI) and how they stack up on 27 criterias. Cloud Build received the highest score in both the current offering and strategy categories.
“Google Cloud Build comes out swinging, matching up well with other cloud giants. Google Cloud Build is relatively new when compared to the other public cloud CI offerings; this vendor had a lot to prove, and it did…. Customer references are happy to trade the cost of paying for the operation and management of build servers for moving their operations to Google’s pay-per-compute system. One reference explained that it’s in the middle of a cloud shift and is currently executing up to 650 builds per day. With that proved out, this customer plans to move an additional 250 repositories to the cloud” – Forrester Wave™ report
Top score in the Current Offering category: Among all 10 CI providers evaluated in the Wave, Cloud Build got the highest score in the Current Offering category. The Current Offering score is based on Cloud Build’s strength in the developer experience, build speed and scale, enterprise security and compliance, and enterprise support, amongst other criteria.
Top score in the Strategy category: Along with top score in the current offering category, Cloud Build also received the highest score in the Strategy category. In particular, Cloud Build’s scores in the partner ecosystem, commercial model, enterprise strategy and vision, and product roadmap criteria contributed to the score.
CI plays an increasingly important role in DevOps, allowing enterprises to drive quality from the start of their development cycle. The report mentions “that cloud-native CI is the secret development sauce that enterprises need to be fast, responsive, and ready to take on incumbents and would-be digital disruptors.”
At Google, we’ve seen first-hand how cloud-based serverless CI tool can help drive quality and security at scale, and the lessons we’ve learned in that process manifest directly in Cloud Build.
Today, organizations of all sizes use Cloud Build to drive productivity improvements via automated, repeatable CI processes. Some customers include Zendesk, Shopify, Snap, Lyft, and Vendasta, who chose Cloud Build for its:
Fully serverless platform: Cloud Build scales up and scale down in response to load with no need to pre-provision servers or pay in advance for additional capacity.
Flexibility: With custom build steps and pre-created extensions to third party apps, enterprises can easily tie their legacy or home-grown tools as a part of their build process.
Security and compliance features: Developers have an ability to perform deep security scans within the CI/CD pipeline and ensure only trusted container images are deployed to production.
We’re thrilled by Forrester’s recognition for Cloud Build. You can download a copy of the report here.
Details: Amazon CloudFront announces the launch of a new CloudFront Edge (POP) location in Shenzhen, China. With this new POP operated by Ningxia Western Cloud Data Co. Ltd. (NWCD), CloudFront has 4 POPs in 4 cities across China. With this launch, viewers in Shenzhen would see an improvement of 62% in average latency when accessing content through CloudFront.
To view pricing for CloudFront China delivery, please refer to here. For developer guide, please refer here. To get started, log in to the AWS Management Console and start accelerating your content.
Whether they’re lifting-and-shifting workloads or taking cloud-native approaches to application development, enterprises are increasingly looking to the cloud to build and grow their businesses. As a result, over the past 12 months we’ve seen extraordinary momentum in Germany, Austria, and Switzerland—also known as the DACH region—as more and more businesses take advantage of Google Cloud. In March, we launched a new Google Cloud region in Zurich to support businesses in the region. We continue to expand our support of regional and global certifications and standards, now including FINMA-regulated customers in Switzerland. And we continue to work closely with many enterprises in the region, such as METRO AG, and many more.
Yesterday’s Cloud Summit in Munich is a great example of the continuing momentum we’re seeing. With 2200 attendees and 45 customer speakers, the summit was an important opportunity for us to connect with—and learn from—businesses in the region. Customers across every industry shared with us how they’re transforming with the power of the cloud. Here are a few highlights.
DLR: Groundbreaking robotics research with the help of Google Cloud infrastructure Part of Deutsches Zentrum für Luft- und Raumfahrt(DLR), Germany’s national aerospace center, the Institute of Robotics and Mechatronics is one of the largest pure robotics research institutes in the world. To advance its work, the institute relies on deep learning techniques, but this requires a substantial amount of compute resources for training and running simulations. Thanks to Google Cloud, it can now easily configure powerful instances with Compute Engine, even using hundreds of CPU cores if necessary. And it now has the flexibility to spin up specialized configurations when needed—something it couldn’t do with in-house servers. You can learn more in DLR’s case study.
“With Google Cloud, we can train our robots between five to ten times faster than before and really explore things like deep reinforcement learning,” says Berthold Bäuml, Head of Autonomous Learning Robots Lab, DLR. “We can do cutting edge research and we’re no longer bound by our computing resources. It’s providing totally new opportunities for us.”
ARD: From monolithic architecture to modern microservices A joint organization of Germany’s regional public-service broadcasters, the ARD network produces key national and regional television and radio programs, and faces challenges shared by many media organizations aiming to serve customers with the content they want, where and when they want it. Starting from a monolithic, rigid structure, ARD wanted a technology stack that fit its own culture—open and scalable. With Google Cloud, ARD is able to focus on five digital, scalable core products and take advantage of fully-managed services which has allowed it to increase its number of releases from once a year to 50 releases a day. In addition to using GCP as a scalable underlying platform, ARD is also using an API-as-a-service approach with Apigee to rethink the way they are delivering content to German citizens by making content delivery universally accessible to other broadcasting entities.
Deutsche Börse Group: Embracing a multi-cloud approach to infrastructure As an international exchange organization and market infrastructure provider, Deutsche Börse Group offers its customers a wide range of products, services and technologies covering the entire value chain of financial markets. As a result, Deutsche Bӧrse Group has frequently been an early adopter of new technologies that drive its industry forward, whether that’s using distributed ledger/blockchain technology in new ways or offering new advanced analytics and artificial intelligence (AI) services to clients. Most recently, Deutsche Bӧrse Group has ramped up its cloud-first strategy, led by its inaugural Chief Cloud Officer Michael Girg, choosing Google Cloud to help them modernize, develop and operate its enterprise workloads in a more efficient and secure way that also supports compliance.
“As a key technology, cloud lays the foundation for enabling some of Deutsche Börse’s major initiatives focussing on new technologies,” says Michael Girg, Chief of Cloud Officer at Deutsche Börse. “Together with Google Cloud as a strong partner, we are very much looking forward to accelerating cloud adoption and to jointly define innovative solutions that further push ahead data security for the financial industry”
You can read more on their progress in our recent blog post.
MediaMarktSaturn Retail Group: Transforming retail, online and offline One of the world’s largest consumer electronics retailers, the MediaMarktSaturn Retail Group operates more than 1,000 stores across 15 countries in Europe, including a sizeable ecommerce presence. But the company was increasingly finding that its on-premises infrastructure, used to first built its online stores, was not able to keep up with evolving business and customer demands. So MediaMarktSaturn turned to the cloud, upgrading its infrastructure but also to adopting a new way of working that places technology at the heart of its strategy. For MediaMarktSaturn, this meant combining data streams into a new data lake from which it can query data with and derive insights with ease, as well as adopting Google Kubernetes Engine (GKE), to manage the Kubernetes clusters that form the backbone of its new online system.
“Google Cloud has paved the way for us to break new ground not only technically, but also in our way of working,” says Dr. Johannes Wechsler, Managing Director, MediaMarktSaturn Technology. “Thanks to this strong partnership, we are well equipped to offer our customers a great user experience even in high-load situations and at the same time deliver new features more quickly.”
Looking ahead We’re excited to see what enterprises in Germany, Austria, and Switzerland can do as they embrace Google Cloud. We remain committed to working with them to help them grow and digitally transform their businesses.
AWS Glue has updated its Apache Spark infrastructure to support Apache Spark 2.4.3 (in addition to Apache Spark 2.2.1) for Glue scripts submitted on development endpoints. This enables you to take advantage of stability fixes and new features available in this version of Apache Spark.
Composite indexes were introduced in Azure Cosmos DB at Microsoft Build 2019. With our latest service update, additional query types can now leverage composite indexes. In this post, we’ll explore composite indexes and highlight common use cases.
Index types in Azure Cosmos DB
Azure Cosmos DB currently has the following index types that are used for the following types of queries:
Range indexes:
Equality queries
Range queries
ORDER BY queries on a single property
JOIN queries
Spatial indexes:
Geospatial functions
Composite indexes:
ORDER BY queries on multiple properties
Queries with a filter as well as an ORDER BY clause
Queries with a filter on two or more properties
Composite index use cases
By default, Azure Cosmos DB will create a range index on every property. For many workloads, these indexes are enough, and no further optimizations are necessary. Composite indexes can be added in addition to the default range indexes. Composite indexes have both a path and order (ASC or DESC) defined for each property within the composite index.
ORDER BY queries on multiple properties
If a query has an ORDER BY clause with two or more properties, a composite index is required. For example, the following query requires a composite index defined on age and name (age ASC, name ASC):
SELECT * FROM c WHERE c.age ASC, c.name ASC
This query will sort all results in ascending order by the value of the age property. If two documents have the same age value, the query will sort the documents by name.
Queries with a filter as well as an ORDER BY clause
If a query has a filter as well as an ORDER BY clause on different properties, a composite index will improve performance. For example, the following query will require fewer request units (RU’s) if a composite index on name and age is defined and the query is updated to include the name in the ORDER BY clause:
Original query utilizing range index:
SELECT * FROM c WHERE c.name = “Tim” ORDER BY c.age ASC
Revised query utilizing a composite index on name and age:
SELECT * FROM c WHERE c.name = “Tim” ORDER BY c.name ASC, c.age ASC
While a composite index will significantly improve query performance, you can still run the original query successfully without a composite index. When you run the revised query with a composite index, it will sort documents by the age property. Since all documents matching the filter have the same name value, the query will return them in ascending order by age.
Queries with a filter on multiple properties
If a query has a filter with two or more properties, adding a composite index will improve performance.
Consider the following query:
SELECT * FROM c WHERE c.name = “Tim” and c.age > 18
In the absence of a composite index on (name ASC, and age ASC), we will utilize a range index for this query. We can improve the efficiency of this query by creating a composite index for name and age.
Queries with multiple equality filters and a maximum of one range filter (such as >,<, <=, >=, !=) will utilize the composite index. In some cases, if a query can’t fully utilize a composite index, it will use a combination of the defined composite indexes and range indexes. For more information, reference our indexing policy documentation.
Composite index performance benefits
We can run some sample queries to highlight the performance benefits of composite indexes. We will use a nutrition dataset that is used in Azure Cosmos DB labs.
In this example, we will optimize a query that has a filter as well as an ORDER BY clause. We will start with the default indexing policy which indexes all properties with a range index. Executing the following query as referenced in the image below in the Azure Portal, we observe the query metrics:
Query metrics:
This query, with the default indexing policy, required 21.8 RU’s.
Adding a composite index on foodGroup and _ts and updating the query text to include foodGroup in the ORDER BY clause significantly reduced the query’s RU charge.
Query metrics:
After adding a composite index, the query’s RU charge decreased from 21.8 RU’s to only 4.07 RU’s. This query optimization will be particularly impactful as the total data size increases. The benefits of a composite index are significant when the properties in the ORDER BY clause have a high cardinality.
Creating composite indexes
You can learn more about creating composite indexes in this documentation. It’s simple to update the indexing policy directly through the Azure Portal. While creating a composite index for data that’s already in Azure Cosmos DB, the index update will utilize the RU’s leftover from normal operations. After the new indexing policy is defined, Azure Cosmos DB will automatically index properties with a composite index as they’re written.
Explore whether composite indexes will improve RU utilization for your existing workloads on Azure Cosmos DB.
AWS IoT Greengrass Core 1.9.3 is now available. With this release, AWS IoT Greengrass adds support for the ARMv6 architecture and new machine learning inference capabilities.
Editor’s Note:This is the third blog in our six-part series on how to use Cloud Security Command Center. There are links to the first two blogs in the series at the end of this post.
When a threat is detected, every second counts. But, sometimes it can be difficult to know if a threat is present or how to respond. Cloud Anomaly Detection is a built-in Cloud Security Command Center (Cloud SCC) feature that uses behavioral signals to detect security abnormalities, such as leaked credentials or unusual activity, in your GCP projects and virtual machines. In this blog, and the accompanying video, we’ll look at how to enable Cloud Anomaly Detection and quickly respond to threats.
1. Enable Cloud Anomaly Detection from Cloud Security Command Center Cloud Anomaly Detection is not turned on by default. You need to go to Security Sources from the Cloud SCC dashboard and activate it. Keep in mind, to enable a security source, you need to have the Organization Administrator Cloud IAM role. Once it’s turned on, findings will automatically be surfaced and displayed in the Cloud Anomaly Detection card on the Cloud Security Command Center dashboard.
2. View findings in Cloud Security Command Center
Cloud Anomaly Detection can surface a variety of anomalous findings, including:
Leaked service account credentials: GCP service account credentials that are accidentally leaked online or compromised.
Resource used for outbound intrusion: One of the resources or GCP services in your organization is being used for intrusion activities, like an attempt to break in to or compromise a target system. These include SSH brute force attacks, Port scans, and FTP brute force attacks.
Potential compromised machine: A potential compromise of a resource in your organization.
Resource used for crypto mining: Behavioral signals around a VM in your organization indicate that it might have been compromised and could be getting used for crypto mining.
Unusual Activity/Connection: Unusual activity from a resource in your organization.
Resource used for phishing: One of the resources or GCP services in your organization is being used for phishing.
3. Remediate findings from Cloud Security Command Center After Cloud Anomaly Detection generates a finding, you can click on the finding for more information about what happened and use that information to fix the security issue.
To learn more about Cloud Anomaly Detection, including how to turn it on and how it can help your organization, check out the video below.
With AWS Elemental MediaConnect, content owners can now specify the percentage of data transfer costs assigned to themselves and to subscribers. This lets content owners share the cost of content syndication with their subscribers, reducing expenses and simplifying billing. To learn more about entitlements in MediaConnect, please read the user guide.
AWS Marketplace, un seleccionado catálogo digital con más de 4,800 soluciones, anunció una función que le facilitará encontrar soluciones relevantes de terceros directamente en la consola de AWS. Ahora, las soluciones de terceros de AWS Marketplace se filtrarán automáticamente y se enumerarán en el panel de navegación izquierdo basado en la consola de servicio de AWS en la que se encuentra. Esto significa que solo verá modelos y algoritmos de Machine Learning en la consola de Amazon SageMaker. También puede usar la nueva barra de búsqueda o las opciones de filtrado para buscar soluciones específicas sin salir de la consola.
