Google Cloud has some great tools for software developers. Cloud Source Repositories and Stackdriver Debugger are used daily by thousands of developers who value Cloud Source Repositories’ excellent code search and Debugger’s ability to quickly and safely find errors in production services.
But Debugger isn’t a full-fledged code browser, and isn’t tightly integrated with all the most common developer environments. The good news is that these tools are coming together! Starting today, you can debug your production services directly in Cloud Source Repositories, for every service where Stackdriver Debugger is enabled.
What’s new in Cloud Source Repositories
This integration brings two pieces of functionality to Cloud Source Repositories: support for snapshots, and logpoints.
Snapshots Snapshots are point-in-time images of your application’s local variables and stack that are triggered when a code condition is met. Think of snapshots as breakpoints that don’t halt execution. To create one, simply click on a line number as you would with a traditional debugger, and the snapshot will activate the next time that one of your instances executes the selected line. When this happens, you’ll see the local variables captured during the snapshot and the complete call stack—without halting the application or impacting its state and ongoing operations!
You can navigate and view local variables in this snapshot from each frame in the stack, just as with any other debugger. You also have full access to conditions and expressions, and there are safeguards in place to protect against accidental changes to your application’s state.
Logpoints Logpoints allow you to dynamically insert log statements into your running services without redeploying them. Each logpoint operates just like a log statement that you write into your code normally: you can add free text, reference variables, and set the conditions for the log to be saved. Logpoints are written to your standard output path, meaning that you can use them with any logging backend, not just Stackdriver Logging.
Creating a logpoint is a lot like creating a snapshot: simply click on the line number of the line where you wish to set it, and you’re done.
Upon adding a logpoint to your application, it’s pushed out to all instances of the selected service. Logpoints last for 24 hours or until the service is redeployed, whichever comes first. Logpoints have the same performance impact as any other log statement that exists normally in your source code.
Getting started
To use Cloud Source Repositories’ production debugging capabilities, you must first enable your Google Cloud Platform projects for Stackdriver Debugger. You can learn more about these setup steps in the Stackdriver Debugger documentation.
Once this is complete, navigate to the source code you wish to debug in Cloud Source Repositories, then select ‘Debug application’. Today this works best with code stored in Cloud Source Repositories or that is mirrored from supported third-party sources including GitHub, Bitbucket, and GitLab. Once you’ve selected your application you can start placing snapshots and logpoints in your code by clicking on the line numbers in the left gutter.
Production debugging done right
Being able to debug code that’s running in production is a critical capability—and being able to do so from a full-featured code browser is even better! Now, through bringing production debugging to Cloud Source Repositories, you can track down hard-to-find problems deep in your code, while being able to do things like continually sync code from a variety of different sources, cross-reference classes, look at blame layers, view change history, and search by class name, method name, etc. To learn more, check out this getting started guide.
Russell Wolf, Product Manager, also contributed to this blog post.
Microservices enable you to break up large monolithic applications into smaller chunks that are easy to build, maintain and upgrade. With a microservices architecture, individual services can now offload work to the background and be consumed later by another service. This gives users quicker response times as well as smoother interactions across a mesh of services.
At Google Cloud Next 2019, we announced Cloud Tasks, a fully managed, asynchronous task execution service that lets you offload long-running asynchronous operations, facilitating point-to-point collaboration and interaction across these microservices. It is already generally available for App Engine targets (tasks that originate from App Engine) and today, we are announcing new HTTP targets in beta that securely reach Google Kubernetes Engine (GKE), Compute Engine, Cloud Run or on-prem systems using industry-standard OAuth/OpenID Connect authentication.
With Cloud Tasks, you can offload long-running and background activities, decouple services from one another, and make your applications much more resilient to failures. At the same time, Cloud Tasks provides all the benefits of a distributed task queue: flexible routing, task offloading, loose coupling between services, rate limiting and enhanced system reliability:
Flexible routing: You can dispatch tasks that can reach any target within Google Cloud Platform (GCP) and on on-prem systems using HTTP/S securely with OAuth/OpenID Connect authentication. (NEW)
Task offloading: You can dispatch heavyweight and long running processes to a task queue, allowing your application to be more responsive to your users.
Loose coupling: Services don’t talk to one another via direct request/response, but rather asynchronously, allowing them to scale independently.
Rate limiting controls the rate at which tasks are dispatched to ensure the processing microservice doesn’t get overwhelmed.
Higher reliability and resilience comes from the fact that your tasks are persisted in storage and retried automatically, making your infrastructure resilient to intermittent failures. Additionally, you can customize the maximum number of retry attempts, and the minimum wait time between attempts in order to meet the specific needs of your system.
Finally, Cloud Tasks does all this in a fully managed serverless fashion, with no manual intervention needed from the developer or IT administrator. You also pay only for the operations you run; GCP takes care of all the provisioning of resources, replication and scaling required to operate Cloud Tasks. As a developer, simply add tasks to the queue and Cloud Tasks handles the rest.
A1 Comms is one of the UK’s leading business-to-business communications providers and uses Cloud Tasks to simplify its application architecture:
“Cloud Tasks allows us to focus on the core requirements of the application we’re developing, instead of other utility requirements. We’ve been using Cloud Tasks extensively, from handling high volumes of notifications between applications that reside on different platforms, to data ingestion/migration tasks and the delegation, trigger, or control of workloads. After using Cloud Tasks, our development velocity has been given a significant boost and our overall architecture simplified.” – Jonathan Liversidge, IT Director, A1 Comms
How Cloud Tasks works
Cloud Tasks operates in a push queue fashion, dispatching tasks to worker services via HTTP requests according to a routing configuration. The worker then attempts to process them. If a task fails, an HTTP error is sent back to Cloud Tasks which then pauses and retries to execute the task until a maximum number of attempts is reached. Once the task executes successfully, the worker sends a 2xx success status code to Cloud Tasks.
Cloud Tasks handles all of the process management for the task, ensuring tasks are processed in a reliable manner and deleting tasks as they finish executing. Additionally, you can track the status of the tasks in the system through an intuitive UI, CLI or client libraries.
Glue together an end-to-end solution
You can use Cloud Tasks to architect interesting solutions, for example wiring together a deferred task processing system across microservices using products such as Cloud Run, Cloud Functions, GKE, App Engine, BigQuery and Stackdriver. Here’s an example from Braden Bassingthwaite from Vendasta at Cloud Next 2019.
Get started today
You can get started with Cloud Tasks today by using the quickstart guide. Create and configure your own Cloud Tasks queues using the documentation here or start your free trial on GCP!
Yesterday, we announced new second-generation runtimes for Go 1.12 and PHP 7.3. In addition, App Engine standard instances now run with double the memory. Today, we’re happy to announce the availability of the new Java 11 second-generation runtime for App Engine standard in beta. Now, you can take advantage of the latest Long-Term-Support version of the Java programming language to develop and deploy your applications on our fully-managed serverless application platform.
Based on technology from the gVisor container sandbox, second-generation runtimes let you write portable web apps and microservices that take advantage of App Engine’s unique auto-scaling, built-in security and pay-per-use billing model—without some of App Engine’s earlier runtime restrictions. Second generation-runtimes also let you build applications more idiomatically. You’re free to use whichever framework or library you need for your project—there are no limitations in terms of what classes you can use, for instance. You can even use native dependencies if needed. Beyond Java, you can also use alternative JVM (Java Virtual Machine) languages like Apache Groovy, Kotlin or Scala if you wish.
In addition to more developer freedom, you also get all the benefits of a serverless approach. App Engine can transparently scale your app up to n and back down to 0, so your application can handle the load when it’s featured on primetime TV or goes viral on social networks. Likewise, it scales to zero if no traffic comes. Your bill will also be proportional to your usage, so if nobody uses your app, you won’t pay a dime (there is also a free tier available).
App Engine second-generation runtimes also mean you don’t need to worry about security tasks like applying OS security patches and updates. Your code runs securely in a gVisor-based sandbox, and we update the underlying layers for you. No need to provision or manage servers yourself—just focus on your code and your ideas!
What’s new? When you migrate to Java 11, you gain access to all the goodies of the most recent Java versions: you can now use advanced type inference with the new var keyword, create lists or maps easily and concisely with the new immutable collections, and simplify calling remote hosts thanks to the graduated HttpClient support. Last but not least, you can also use the JPMS module system introduced in Java 9.
You’ll also find some changes in the Java 11 runtime. For example, the Java 11 runtime does not provide a Servlet-based runtime anymore. Instead, you need to bundle a server with your application in the form of an executable JAR. This means that you are free to choose whichever library or framework you want, be it based on the Servlet API or other networking stacks like the Netty library. In other words, feel free to use Spring Boot, Vert.x, SparkJava, Ktor, Helidon or Micronaut if you wish!
Last but not least, second-generation runtimes don’t come with the built-in APIs like Datastore or memcache from the App Engine SDK. Instead, you can use the standalone services with their Google Cloud client libraries, or use other similar services of your choice. Be sure to look into our migration guide for more help on these moves.
Getting started To deploy to App Engine Java 11, all you need is an app.yaml file where you specify runtime: java11, signifying that your application should use Java 11. That’s enough to tell App Engine to use the Java 11 runtime, regardless of whether you’re using an executable JAR, or a WAR file with a provided servlet-container. However, the new runtime also gives you more control on how your application starts: by specifying an extra entrypoint parameter in app.yaml, you can then customize the java command flags, like the -X memory settings.
With Java 11, the java command now includes the ability to run single independent *.java files without compiling them with javac! For this short getting started section, we are going to use it to run the simplest hello world example with the JDK’s built-in HTTP server:
Notice how our Main class uses the var keyword introduced in Java 10, and how we re-used the keyword again in the try-with-resources block, as Java 11 makes possible.
Now it’s time to prepare our app.yaml file. First, specify the java11 runtime. In addition, entrypoint define the actual java command with which to we’ll be running to launch the server. The java command points at our single Java source file:
Finally, don’t forget to deploy your application with the gcloud app deploy app.yaml command. Of course, you can also take advantage of dedicated Maven and Gradle plugins for your deployments.
Try Java 11 on App Engine standard today You can write your App Engine applications with Java 11 today, thanks to the newly released runtime in beta. Please read the documentation to get started and learn more about it, have a look at the many samples that are available, and check out the migration guide on moving from Java 8 to 11. And don’t forget you can take advantage of the App Engine free tier while you experiment with our platform.
At Google Cloud, we often throw around the term ‘cloud-native architecture’ as the desired end goal for applications that you migrate or build on Google Cloud Platform (GCP). But what exactly do we mean by cloud-native? More to the point, how do you go about designing such a system?
At a high level, cloud-native architecture means adapting to the many new possibilities—but very different set of architectural constraints—offered by the cloud compared to traditional on-premises infrastructure. Consider the high level elements that we as software architects are trained to consider:
The functional requirements of a system (what it should do, e.g ‘process orders in this format…’)
The non-functional requirements (how it should perform e.g. ‘process at least 200 orders a minute’)
Constraints (what is out-of-scope to change e.g. ‘orders must be updated on our existing mainframe system’).
While the functional aspects don’t change too much, the cloud offers, and sometimes requires, very different ways to meet non-functional requirements, and imposes very different architectural constraints. If architects fail to adapt their approach to these different constraints, the systems they architect are often fragile, expensive, and hard to maintain. A well-architected cloud native system, on the other hand, should be largely self-healing, cost efficient, and easily updated and maintained through Continuous Integration/Continuous Delivery (CI/CD).
The good news is that cloud is made of the same fabric of servers, disks and networks that makes up traditional infrastructure. This means that almost all of the principles of good architectural design still apply for cloud-native architecture. However, some of the fundamental assumptions about how that fabric performs change when you’re in the cloud. For instance, provisioning a replacement server can take weeks in traditional environments, whereas in the cloud, it takes seconds—your application architecture needs to take that into account.
In this post we set out five principles of cloud-native architecture that will help to ensure your designs take full advantage of the cloud while avoiding the pitfalls of shoe-horning old approaches into a new platform.
Principles for cloud-native architecture The principle of architecting for the cloud, a.k.a. cloud-native architecture, focuses on how to optimize system architectures for the unique capabilities of the cloud. Traditional architecture tends to optimize for a fixed, high-cost infrastructure, which requires considerable manual effort to modify. Traditional architecture therefore focuses on the resilience and performance of a relatively small fixed number of components. In the cloud however, such a fixed infrastructure makes much less sense because cloud is charged based on usage (so you save money when you can reduce your footprint) and it’s also much easier to automate (so automatically scaling-up and down is much easier). Therefore, cloud-native architecture focuses on achieving resilience and scale though horizontal scaling, distributed processing, and automating the replacement of failed components. Let’s take a look.
Principle 1: Design for automation Automation has always been a best practice for software systems, but cloud makes it easier than ever to automate the infrastructure as well as components that sit above it. Although the upfront investment is often higher, favouring an automated solution will almost always pay off in the medium term in terms of effort, but also in terms of the resilience and performance of your system. Automated processes can repair, scale, deploy your system far faster than people can. As we discuss later on, architecture in the cloud is not a one-shot deal, and automation is no exception—as you find new ways that your system needs to take action, so you will find new things to automate.
Some common areas for automating cloud-native systems are:
Continuous Integration/Continuous Delivery: Automate the build, testing, and deployment of the packages that make up the system by using tools like Google Cloud Build, Jenkins and Spinnaker. Not only should you automate the deployment, you should strive to automate processes like canary testing and rollback.
Scale up and scale down: Unless your system load almost never changes, you should automate the scale up of the system in response to increases in load, and scale down in response to sustained drops in load. By scaling up, you ensure your service remains available, and by scaling down you reduce costs. This makes clear sense for high-scale applications, like public websites, but also for smaller applications with irregular load, for instance internal applications that are very busy at certain periods, but barely used at others. For applications that sometimes receive almost no traffic, and for which you can tolerate some initial latency, you should even consider scaling to zero (removing all running instances, and restarting the application when it’s next needed).
Monitoring and automated recovery: You should bake monitoring and logging into your cloud-native systems from inception. Logging and monitoring data streams can naturally be used for monitoring the health of the system, but can have many uses beyond this. For instance, they can give valuable insights into system usage and user behaviour (how many people are using the system, what parts they’re using, what their average latency is, etc). Secondly, they can be used in aggregate to give a measure of overall system health (e.g., a disk is nearly full again, but is it filling faster than usual? What is the relationship between disk usage and service uptake? etc). Lastly, they are an ideal point for attaching automation. Now when that disk fills up, instead of just logging an error, you can also automatically resize the disk to allow the system to keep functioning.
Principle 2: Be smart with state Storing of ‘state’, be that user data (e.g., the items in the users shopping cart, or their employee number) or system state (e.g., how many instances of a job are running, what version of code is running in production), is the hardest aspect of architecting a distributed, cloud-native architecture. You should therefore architect your system to be intentional about when, and how, you store state, and design components to be stateless wherever you can.
Stateless components are easy to:
Scale: To scale up, just add more copies. To scale down, instruct instances to terminate once they have completed their current task.
Repair: To ‘repair’ a failed instance of a component, simply terminate it as gracefully as possible and spin up a replacement.
Roll-back: If you have a bad deployment, stateless components are much easier to roll back, since you can terminate them and launch instances of the old version instead.
Load-Balance across: When components are stateless, load balancing is much simpler since any instance can handle any request. Load balancing across stateful components is much harder, since the state of the user’s session typically resides on the instance, forcing that instance to handle all requests from a given user.
Principle 3: Favor managed services Cloud is more than just infrastructure. Most cloud providers offer a rich set of managed services, providing all sorts of functionality that relieve you of the headache of managing the backend software or infrastructure. However, many organizations are cautious about taking advantage of these services because they are concerned about being ‘locked in’ to a given provider. This is a valid concern, but managed services can often save the organization hugely in time and operational overhead.
Broadly speaking, the decision of whether or not to adopt managed services comes down to portability vs. operational overhead, in terms of both money, but also skills. Crudely, the managed services that you might consider today fall into three broad categories:
Managed open source or open source-compatible services: Services that are managed open source (for instance Cloud SQL) or offer an open-source compatible interface (for instance Cloud Bigtable). This should be an easy choice since there are a lot of benefits in using the managed service, and little risk.
Managed services with high operational savings: Some services are not immediately compatible with open source, or have no immediate open source alternative, but are so much easier to consume than the alternatives, they are worth the risk. For instance, BigQuery is often adopted by organizations because it is so easy to operate.
Everything else: Then there are the hard cases, where there is no easy migration path off of the service, and it presents a less obvious operational benefit. You’ll need to examine these on a case-by-case basis, considering things like the strategic significance of the service, the operational overhead of running it yourself, and the effort required to migrate away.
However, practical experience has shown that most cloud-native architectures favor managed services; the potential risk of having to migrate off of them rarely outweighs the huge savings in time, effort, and operational risk of having the cloud provider manage the service, at scale, on your behalf.
Principle 4: Practice defense in depth Traditional architectures place a lot of faith in perimeter security, crudely a hardened network perimeter with ‘trusted things’ inside and ‘untrusted things’ outside. Unfortunately, this approach has always been vulnerable to insider attacks, as well as external threats such as spear phishing. Moreover, the increasing pressure to provide flexible and mobile working has further undermined the network perimeter.
Cloud-native architectures have their origins in internet-facing services, and so have always needed to deal with external attacks. Therefore they adopt an approach of defense-in-depth by applying authentication between each component, and by minimizing the trust between those components (even if they are ‘internal’). As a result, there is no ‘inside’ and ‘outside’.
Cloud-native architectures should extend this idea beyond authentication to include things like rate limiting and script injection. Each component in a design should seek to protect itself from the other components. This not only makes the architecture very resilient, it also makes the resulting services easier to deploy in a cloud environment, where there may not be a trusted network between the service and its users.
Principle 5: Always be architecting One of the core characteristics of a cloud-native system is that it’s always evolving, and that’s equally true of the architecture. As a cloud-native architect, you should always seek to refine, simplify and improve the architecture of the system, as the needs of the organization change, the landscape of your IT systems change, and the capabilities of your cloud provider itself change. While this undoubtedly requires constant investment, the lessons of the past are clear: to evolve, grow, and respond, IT systems need to live and breath and change. Dead, ossifying IT systems rapidly bring the organization to a standstill, unable to respond to new threats and opportunities.
The only constant is change In the animal kingdom, survival favors those individuals who adapt to their environment. This is not a linear journey from ‘bad’ to ‘best’ or from ‘primitive’ to ‘evolved’, rather everything is in constant flux. As the environment changes, pressure is applied to species to evolve and adapt. Similarly, cloud-native architectures do not replace traditional architectures, but they are better adapted to the very different environment of cloud. Cloud is increasingly the environment in which most of us find ourselves working, and failure to evolve and adapt, as many species can attest, is not a long term option.
The principles described above are not a magic formula for creating a cloud-native architecture, but hopefully provide strong guidelines on how to get the most out of the cloud. As an added benefit, moving and adapting architectures for cloud gives you the opportunity to improve and adapt them in other ways, and make them better able to adapt to the next environmental shift. Change can be hard, but as evolution has shown for billions of years, you don’t have to be the best to survive—you just need to be able to adapt.
If you would like to learn more about the topics in this post, check out the following resources:
Change is hard in any organisation, Changing the change rules at Google on the Google re:work site has some interesting insights on how Google approaches change
Last year, we announced App Engine second generation runtimes, which let you use any language library, have direct network access, and connect to Google Cloud VPC Networks, giving you a more idiomatic developer experience, support for native modules and faster execution. Today, we are excited to announce that all App Engine second generation runtime instances will receive double the memory. In addition, Go 1.12 and PHP 7.3 runtimes for App Engine standard are now generally available.
Doubling the RAM for all App Engine second generation applications lets them more easily load language libraries or scale vertically to increase concurrency via multiprocessing or multithreading. This increased memory limit is already available for all apps running second generation runtimes; you don’t need to take any action to receive the additional memory. This upgrade comes at no additional cost.
Please refer to the table below for more information on memory associated with each instance class.
1: F4_1G and B4_1G have been renamed to F4_HIGHMEM & B4_HIGHMEM for accuracy. The original _1G instance names will continue to work and will receive 2,048 MB of memory.
In the past year, we’ve announced a number of second generation runtimes, including Node.js 10, PHP 7.2, Python 3.7, and Ruby 2.5 (alpha). Today we are announcing the general availability of two new runtimes: Go 1.12 and PHP 7.3.
A1 Comms is one of the UK’s leading mobile phone and telecommunication providers, and is using PHP to as it transforms its business from brick-and-mortar to online retail:
“Moving to second generation runtimes has saved us a lot of debugging time and helped us increase performance by at least 50%. The PHP 7.3 runtime is giving our developers the best of the bleeding edge for Laravel compatibility and speed, while still providing automatic security patching to meet our compliance requirements. It exceeds our expectations for reliability. The inclusion of native support for the OpenCensus module for Stackdriver Trace is also something we are excited about. .” – Sam Melrose, System Engineer, A1 Comms
Additionally, second generation runtimes are interoperable with our recently released Cloud Run, a serverless compute offering that allows you to run any stateless container in a fully managed fashion or on top of an existing GKE cluster. You can take apps that were built on App Engine second generation runtimes and move them to Cloud Run, or vice-versa.
Forget infrastructure and focus on your users While there are a lot of new things to love about second generation runtimes, the core value for App Engine remains: the service allows developers and companies to focus on creating great software while taking advantage of Google Cloud’s world class operations and infrastructure to scale, monitor, and manage their applications.
“App Engine has literally been a game changer for us. Since migrating to it, I’ve yet to have a board meeting where we need to discuss availability or capacity or any other reason for downtime. In fact, during Black Friday while some competitors’ websites slowed or went down, our response time actually improved.” – Jonathan Liversided, IT Director, A1 Comms
At Google Cloud Next, we announced the general availability of Cloud Scheduler, a fully managed cron job service that allows any application to invoke batch, big data and cloud infrastructure operations. Since then, we have added an important new feature that allows you to trigger any service, running anywhere: on-prem, on Google Cloud or any third party datacenter.
Invoke any service with Cloud Scheduler Now, you can securely invoke HTTP targets on a schedule to reach services running on Google Kubernetes Engine (GKE), Compute Engine, Cloud Run, Cloud Functions, or on on-prem systems or elsewhere with a public IP using industry-standard OAuth/OpenID Connect authentication.
With Cloud Scheduler, you get the following benefits:
Reliable delivery: Cloud Scheduler offers at-least-once delivery of a job to the target, guaranteeing that mission-critical jobs are invoked for execution.
Secure Invocation: Use industry standard OAuth/OpenID Connect tokens to invoke your HTTP/S schedules in a secure fashion. (NEW)
Fault-tolerant execution: Cloud Scheduler lets you automate your retries and execute a job in a fault-tolerant manner by deploying to different regions, so you eliminate the risk of single point of failure as seen in traditional cron services.
Unified management experience: Cloud Scheduler lets you invoke your schedules through the UI, CLI or API and still have a single pane of glass management experience. It also supports the familiar Unix cron format to define your job schedules.
Better yet, Cloud Scheduler does all this in a fully managed serverless fashion, with no need to provision the underlying infrastructure, or manually intervene since it automatically retries failed jobs. You also pay only for the operations you run—GCP takes care of all resource provisioning, replication and scaling required to operate Cloud Scheduler. As a developer you simply create your schedules and Cloud Scheduler handles the rest.
How Cloud Scheduler works To schedule a job, you can use the Cloud Scheduler UI, CLI or API to invoke your favorite HTTP/S endpoint, Cloud Pub/Sub topic or App Engine application. Cloud Scheduler runs a job by sending an HTTP request or Cloud Pub/Sub message to a specified target destination on a recurring schedule. The target handler executes the job and returns a response. If the job succeeds, a success code (2xx for HTTP/App Engine and 0 for Pub/Sub) is returned to Cloud Scheduler. If a job fails, an error is sent back to Cloud Scheduler, which then retries the job until the maximum number of attempts is reached. Once the job has been scheduled, you can monitor it on the Cloud Scheduler UI and check the status of the job.
Glue together an end-to-end solution Cloud Scheduler can be used to architect interesting solutions like wiring together a reporting system on a schedule using Cloud Functions, Compute Engine, Cloud Pub/Sub and Stackdriver. Here’s an example from Garrett Kutcha from Target at Cloud Next 2019.
You can also use Cloud Scheduler to do things like schedule database updates and push notifications, trigger CI/CD pipelines, schedule tasks such as image uploads, and invoke cloud functions. Tightly integrated with most Google Cloud Platform (GCP) products, the sky’s the limit with Cloud Scheduler!
Get started today With Cloud Scheduler, you now have a modern, serverless solution to your job scheduling needs. To try out Cloud Scheduler today, check out the quickstart guide. Then, create and configure your own schedules using the documentation or start a free trial on GCP!
In October of last year, we announced Project Strobe—a Google-wide effort to review third-party developer access to Google account and Android device data. As a result, we rolled out an updated user data policy further restricting access to Gmail data. Today we’re announcing plans to extend the same policy to Google Drive as part of Project Strobe.
With this updated policy, we’ll limit the types of apps that have broad access to content or data via Drive APIs. Apps should move to a per-file user consent model, allowing users to more precisely determine what files an app is allowed to access. This means that only certain types of apps can request these scopes from consumer Google accounts. As always, G Suite administrators are in control of their users’ apps.
How to prepare If you’re not a developer, you don’t need to do anything to prepare for these changes. While changes will not go into effect until early next year, we recommend developers begin preparations ahead of time by taking the following steps to ensure their apps using Drive APIs stay compliant and keep working for users. You will not need to go through the verification process if your app is created and used by only your organization (and is marked as internal).
Before getting started, review the Drive updates to the user data policy and FAQ.
If you’ve developed a Drive app that uses any of the restricted scopes, we recommend migrating your app to use the drive.file scope in combination with the Google Picker. This combination will enable users to select the specific files from their Google Drive that they want to allow your app to access. Apps that use the drive.file scope will not be required to go through the restricted scope verification and third-party security assessment outlined in the policy.
If drive.file is not a possible option (e.g. for backup clients), you should begin preparing your app for the restricted scope verification, a process that, among other steps, ensures your use of data is compliant with the Limited Use Requirements and includes a security assessment if your app stores or transmits through servers. Restricted scope verification for the Drive API will begin early next year. Refer to the FAQ for more info.
In the next few months, we will start to notify impacted developers of the policy changes and will provide additional guidance on how to meet the updated policy requirements.
Editors note:Today’s post comes from Andre Fatala, chief technology officer at Brazilian retailer Magazine Luiza. Apigee, Firebase, and Google Cloud Platform (GCP) have helped this 60-year-old company become one of the most successful e-commerce operations in Brazil.
Founded in 1957, Magazine Luiza, or Magalu, is a technology and logistics company focused on the retail sector. In 2018 we posted 60% growth year over year in e-commerce sales, reaching 7 billion in Brazilian Real (nearly 2 billion in USD), with e-commerce contributing 35.7% of our total sales.
From supply chain issues to economic fluctuations, the retail industry in Brazil is complex to say the least. But adopting mobile e-commerce presented us with an entirely new challenge—one that we had to respond to quickly to remain competitive. Brazilian e-commerce players, along with global internet giants, threatened to make inroads into a market in which we have held a leading position for decades. To help achieve our goals, we employed Google Cloud products like the Apigee API management platform, GCP, Firebase, and G Suite.
In 2013 we had an e-commerce platform, and even a library of APIs, but those APIs were accessing an overstretched backend application built with 150,000 lines of code. Deployment of new APIs was slow, we were burdened with undesirable dependencies, and we faced scalability challenges as well as distributed responsibilities across siloed teams.
Knowing we were under threat from competitors, our then-chief operating officer (and current CEO) Frederico Trajano put me in charge of a small team of developers. The team was walled off from IT governance processes and roadblocks from the greater organization, and given the keys to the company’s entire e-commerce operations. That’s around the time when we started using the Apigee API management platform. Apigee helped us to decouple our backend systems from the front end so it was easier and faster for my team to iterate on new apps while other teams maintained our legacy systems of record.
Our new approach accelerated mobile application development, and Firebase has played a big role in this. We started using Firebase soon after we learned about it at Google I/O in 2016. Firebase helped to reduce the complexity of building the apps that we need to reach our customers. We can quickly publish and test new features, and Firebase Crashlytics helps us keep our apps stable and users happy.
Last year, after GCP launched its region here in Brazil, we began deploying workloads onto GCP. We were pleased with the latency of GCP—there just wasn’t any. The speed was notable and this is critical in e-commerce applications. We’d already been using Kubernetes for some time as it’s especially helpful with our multi-cloud strategy. Migrating all our data onto GCP was simple because we used Kubernetes along with our own open-source PaaS.
Believing that we’d have better performance and stability to handle Black Friday traffic running on GCP, we made the decision to migrate 113 apps in less than 60 days before Black Friday. It was a move that paid off: 2018 was our biggest Black Friday ever, where we saw levels of API traffic that were dramatically higher than before. Apigee helped us with our execution, meeting customer demand throughout across our platforms, with visibility across all of our API and application activity.
As a result of these successes, we now have plans to migrate our entire e-commerce platform onto GCP, and our big data team is moving away from our Hadoop environment to an architecture that uses GCP managed services.
With this newfound ease and speed of spinning up new services and customer experiences and adjusting existing ones, everyone is able to work in small teams of five or six people that take care of segments of an application, whether its online checkout, physical store checkout, or order management. We work much more like a software company than a retail company now.
Our approach, powered by Google Cloud technologies, supported us to expand our e-commerce strategy to third-party sellers and create a new digital marketplace. Other merchants can easily join this ecosystem via our API platform. Today, we support more than 3,300 sellers and offer 4.3 million SKUs (compared to our legacy sales and distribution system, which in 2016 supported 50,000 SKUs).
Our goal has been to transform from a traditional retail company with a digital presence to a digital platform with a physical presence and a “human touch”—and now we’re much closer to that vision. People in Brazil enjoy the convenience of ordering from their computers or smartphones, but still appreciate coming into our stores to pick up their items. In fact, two-thirds of Brazilians buy this way. To make it even easier for our customers, we built 12 in-store apps that our salespeople use to make the once-slow sales process much faster—this helped us grow physical store sales 25.8% in 2018!
Google Cloud has been a great partner for us. It’s no small feat to succeed in Brazil’s constantly evolving retail environment, but we feel like we’re on the right path.
The great thing about building applications on G Suite is that you have flexible options. That said, we often get questions from developers about which tools they should use, specifically “should I build my application using Apps Script or App Maker?”
Apps Script and App Maker share a common deployment platform, and are perhaps more similar than they are different. Both let you build custom business applications quickly. Both leverage the same infrastructure. Both let you apply your skills in G Suite.
But they are uniquely different, too. Apps Script’s strength lies in the fact that it helps you integrate your app with G Suite apps like Gmail, Google Drive, Docs or Sheets. As a relative newcomer, App Maker builds off of Apps Script’s strengths, but also lets you build beyond the bounds of G Suite. It allows you to design your own user interface independent of G Suite apps, as well as declaratively design your own data model.
The good news is that you have a choice. Here are important factors to consider when choosing Apps Script or App Maker for your next application.
Consideration #1: Does your app need to run outside of your organization? Choices made for you are often the easiest. In choosing Apps Script or App Maker, sometimes your app requirements will determine which technology you can use.
For example, App Maker requires that all your users have access to the product and they are all on the same domain. If your application needs to run cross-domain or will have external users outside your organization, then App Maker will not work. Another consideration is most apps built in App Maker rely on Cloud SQL for data storage, so if you don’t have access to Google Cloud SQL or would prefer not implementing it, then Apps Script is your choice by default.
Key takeaway: For external or cross-domain users, choose Apps Script.
Consideration #2: What do you feel most comfortable building in? And what coding language do you prefer? Another important factor is a matter of comfort and personal preference. If you think you are more productive building an application within a spreadsheet, especially if one already exists, then start by building on Google Sheets with Apps Script. As a developer, there is slightly more of a learning curve with App Maker and a little more work to do at first to get your app up and running, like working with relational data models or laying out your user interface from scratch. Apps Script relies on the host applications for much of its functionality, and is, for the most part, faster and more flexible to produce a prototype in.
As mentioned, Apps Script and App Maker share many similarities including the same underlying programming language and runtime, which happens to be Google Apps Script. This is great because your skills (and even some code) is transferable, making choosing either option based on programming skill or language essentially a moot point. There are other differences to consider that go beyond the scope of this post, though, like distribution, versioning and the developer experience. But nothing that is dramatic or widens the gap between choosing one over the other.
Key takeaway: Toss Up. Both use Apps Script for coding.
Key takeaway: Choose based on your comfort level. With Apps Script you can take advantage of the G Suite applications for much of the heavy lifting, while App Maker puts you in complete control of how you build out your entire application. And when it comes to code, it’s a toss up because both use Apps Script.
Consideration #3: What experience do you want for your users? Obviously you want your application to be easy to use—it encourages adoption. For this reason, picking a tool that provides the best user experience is important. But beyond making your app easy-to-use, it’s also important to consider how your users will use your app when you design it.
For example, if your app is helping with the sales cycle and all sales communication happens over email, then you really want your application to live within Gmail as a companion app. However, if your application is geared at managing a procurement process, and there is no single or canonical G Suite app that your users use for this task, then it’s best to create a standalone experience.
The simple rule of thumb is that Apps Script was designed to build companion apps that integrate tightly with the host applications, such as Gmail, Sheets, Docs or Slides. These companion apps can be deployed as a document-bound app or as a G Suite add-on, so that users can use the best of what host applications have to offer while also staying in context of how they are working (i.e. avoid switching tabs).
App Maker naturally allows you to design your own user experience from scratch, creating a controlled environment that ensures the user focuses on what you’ve specifically designed in the UI (plus, letting you to tap into G Suite APIs still, like for Gmail, Calendar, Drive, etc.).
Key takeaway: Choose Apps Script to create companion apps that work in context with G Suite. Choose App Maker to create standalone solution apps.
Consideration #4: What data requirements do you need for you app? App Maker offers advanced data modeling functionality by way of its integration with Cloud SQL. This helps you build relational data models, create data views, run data-driven events, implement data level security permissions and more. Another obvious benefit of its Cloud SQL backend is that App Maker can scale to large, complex data models, without requiring much effort by you.
Contrast this with Apps Script, where you either need to create your own data management functionality or rely on storing data with Sheets. The latter is fine for applications that have more simplistic data requirements, but even for most simple CRUD-type applications, App Maker is likely a better choice.
Key takeaway: For advanced data needs, choose App Maker.
Consideration #5: What kind of user interface do you want to implement? Another key difference is the User Interface (UI). App Maker gives you a blank canvas for you to design your own UI (for mobile or desktop) from scratch. The UI builder in App Maker offers two-way data binding which eliminates the need to write plumbing code to connect your UI to data. You can refine your UI using material design themes, CSS and with customizing widget properties, which lets your app have its own unique look. You can further extend the UI with client-side scripts and HTML areas allowing you to add completely custom functionality, but with some “assembly required.”
The UI ‘canvas’ in Apps Script on the other hand, is simply the G Suite application you are working with as the backdrop. This can be optimal because users will easily recognize the environment which makes it more intuitive. It also requires less effort on your part to get started. Much of the customization around your app’s look and feel can be done sans code; for example, with Google Sheets, cells can be used for gathering user inputs and stylized with conditional formatting and data validation (without writing a line of Apps Script). If you need greater custom interaction between users and your app, you can leverage Apps Script to extend the UI with custom dialogs, menus and sidebars right with the confines of the G Suite host app.
Key takeaway: For a more custom, do-it-yourself UI, choose App Maker. For G Suite as a backdrop, go Apps Script.
Tying it all together If you’re in the process of choosing which tool to use to build your application, let us leave you with some final advice.
Apps Script and App Maker are built on the same underlying technology base, so skills and code are transferable.
Sheets can be used to present and store data with Apps Script handling the logic for simple application scenarios. App Maker coupled with Cloud SQL can tackle more complex data requirements with custom UI needs.
Apps Script is better geared for building companion apps inside of G Suite, while standalone apps starting from scratch favor App Maker.
Bottom line: think about the user experience first, then pick the technology that allows you to build that.
Developers love serverless. With serverless, you can focus on code, deploy it, and let the platform take care of the rest—all while only paying for exactly what you use. But traditional serverless solutions can limit what programming languages you can use, or require you to organize our code around functions.
At Google Cloud Next 2019, we launched Cloud Run, a serverless compute platform that lets you run any stateless request-driven container on a fully managed environment. In other words, with Cloud Run, you can take an app—any stateless app—containerize it, and Cloud Run will provision it, scale it up and down, all the way to zero!
Check out how easy it is to get started and deploy your first container to the fully managed version of Cloud Run:
Cloud Run is unique among serverless platforms, and brings a number of benefits:
Do less work with a fully managed solution: With Cloud Run, you can forget about provisioning or managing infrastructure—it does that for you. Cloud Run automatically and quickly scales up or down based on your incoming traffic, and even scales down to zero. In addition, each Cloud Run service gets a stable and secure HTTPS endpoint, and you can easily add your own custom domain for which we automatically provision an SSL certificate. And if you need an easy way to serve static content or cache responses, Cloud Run integrates with Firebase Hosting.
Pay exactly for what you use: Cloud Run charges you for the resources you use only when your containers are processing requests or events, billed to the nearest 100 milliseconds.
Serve web traffic, or process Pub/Sub events: Run publicly accessible web services or APIs, or securely push Pub/Sub events to private Cloud Run microservices.
Leverage the power of containers: Containers have become an industry standard for packaging and deploying code, and let you write your code in your favorite language, with whatever framework or binary library that works for you. If you’re not familiar with containers, don’t be scared; Cloud Run includes official base images for all the most popular languages, and there are examples of Dockerfiles in the documentation.
Enjoy the portability that comes with Knative: Cloud Run implements the Knative serving API, an open-source project to run serverless workloads on top of Kubernetes. That means you can deploy Cloud Run services anywhere Kubernetes runs. And if you need more control over your services (like access to GPU or more memory), you can also deploy these serverless containers in your own GKE cluster instead of using the fully managed environment.
On a run since Google Cloud Next’19 Since it launched in April, Cloud Run has gotten a terrific reception from developers. Early adopters tell us that deploying their favorite apps on Cloud Run “just works.”
“Cloud Run allows us to access, process and serve large amounts of imagery data stored in Google Cloud Storage, with the freedom to use our own custom toolchains and without having to worry about scaling the service to the real time load.” – Thomas Bonfort, R&D Earth Observation Software Engineer at Airbus, and Cloud Run alpha tester
Early Cloud Run alpha testers and then beta users have provided valuable feedback to help us shape and improve the product. Today, we are pleased to launch several new features to address your top requests:
Cloud SQL support With one configuration change, you can now securely and privately connect your Cloud Run services to Cloud SQL instances. Read more in the documentation.
Metrics at a glance As Cloud Run developers, you have been able to benefit from out-of-the-box integration with Stackdriver Logging and Monitoring. But you do not always need the full power of Stackdriver tools. Starting today, the Cloud Run user interface features several key performance indicators, such as:
Comparing the average number of requests per second in the Cloud Run service list
Observing request counts, request latencies, CPU and Memory allocation in a dedicated tab of the Cloud Run service view
New regions Initially offered only in the U.S., we’ve also seen great adoption of Cloud Run among developers in other parts of the world. That’s why we are very happy to announce that we will start Cloud Run’s regional expansion within the next few weeks, starting by opening new regions in Europe and Asia.
Learn more Take our quickstart to deploy your first container to Cloud Run in seconds. For a deep dive on Cloud Run’s features and characteristics, check out my session at Google Cloud Next 2019:
We can’t wait to see what you’ll build with Cloud Run!
When it comes to information modeling, how to denote the relation or relationship between two entities is a key question. Describing the patterns that we see in the real world in terms of entities and their relationships is a fundamental idea that goes back at least as far as the ancient Greeks, and is also the foundation of how we think about information in IT systems today.
For example, relational database technology represents relationships using a foreign key, a value stored in one row in a database table that identifies another row, either in a different table or the same table.
Expressing relationships is very important in APIs too. For example, in a retailer’s API, the entities of the information model might be customers, orders, catalog items, carts, and so on. The API expresses which customer an order is for, or which catalog items are in a cart. A banking API expresses which customer an account belongs to or which account each credit or debit applies to.
The most common way that API developers express relationships is to expose database keys, or proxies for them, in the fields of the entities they expose. However, at least for web APIs, that approach has several disadvantages over the alternative: the web link.
Standardized by the Internet Engineering Task Force (IETF), you can think of a web link as a way of representing relationships on the web. The best-known web links are of course those that appear in HTML web pages expressed using the link or anchor elements, or in HTTP headers. But links can appear in API resources too, and using them instead of foreign keys significantly reduces the amount of information that has to be separately documented by the API provider and learned by the user.
A link is an element in one web resource that includes a reference to another resource along with the name of the relationship between the two resources. The reference to the other entity is written using a special format called Uniform Resource Identifier (URI), for which there is an IETF standard. The standard uses the word ‘resource’ to mean any entity that is referenced by a URI. The relationship name in a link can be thought of as being analogous to the column name of a relational database foreign key column, and the URI in the link is analogous to the foreign key value. By far the most useful URIs are the ones that can be used to get information about the referenced resource using a standard web protocol—such URIs are called Uniform Resource Locators (URLs)—and by far the most important kind of URL for APIs is the HTTP URL.
While links aren’t widely used in APIs, some very prominent web APIs use links based on HTTP URLs to represent relationships, for example the Google Drive API and the GitHub API. Why is that? In this post, I’ll show what using API foreign keys looks like in practice, explain its disadvantages compared to the use of links, and show you how to convert that design to one that uses links.
Representing relationships using foreign keys Consider the popular pedagogic “pet store” application. The application stores electronic records to track pets and their owners. Pets have attributes like name, species and breed. Owners have names and addresses. Each pet has a relationship to its owner—the inverse of the relationship defines the pets owned by a particular owner.
In a typical key-based design the pet store application’s API makes two resources available that look like this:
The relationship between Lassie and Joe is expressed in the representation of Lassie using the “owner” name/value pair. The inverse of the relationship is not expressed. The “owner” value, “98765,” is a foreign key. It is likely that it really is a database foreign key—that is, it is the value of the primary key of some row in some database table—but even if the API implementation has transformed the key values a bit, it still has the general characteristics of a foreign key.
The value “98765” is of limited direct use to a client. For the most common uses, the client needs to compose a URL using this value, and the API documentation needs to describe a formula for performing this transformation. This is most commonly done by defining a URI template, like this:
/people/{person_id}
The inverse of the relationship—the pets belonging to an owner—can also be exposed in the API by implementing and documenting one of the following URI templates (the difference between the two is a question of style, not substance):
APIs that are designed in this way usually require many URI templates to be defined and documented. The most popular language for documenting these templates for APIs isn’t the one defined in the IETF specification—it’s OpenAPI (formerly known as Swagger). Unfortunately, OpenAPI and similar offerings do not provide a way to specify which field values can be plugged into which templates, so some amount of natural language documentation from the provider or guesswork by the client is also required.
In summary, although this style is common, it requires the provider to document, and the client to learn and use, a significant number of URI templates whose usage is not perfectly described by current API specification languages. Fortunately there’s a better option.
Representing relationships using links Imagine the resources above were modified to look like this:
The primary difference is that the relationships are expressed using links, rather than foreign key values. In these examples, the links are expressed using simple JSON name/value pairs (see the section below for a discussion of other approaches to writing links in JSON).
Note also that the inverse relationship of the pet to its owner has been made explicit by adding the “pets” field to the representation of Joe.
Changing “id” to “self” isn’t really necessary or significant, but it’s a common convention to use “self” to identify the resource whose attributes and relationships are specified by the other name/value pairs in the same JSON object. “self” is the name registered at IANA for this purpose.
Viewed from an implementation point of view, replacing all the database keys with links is a fairly simple change—the server converted the database foreign keys into URLs so the client didn’t have to—but it significantly simplifies the API and reduces the coupling of the client and the server. Many URI templates that were essential for the first design are no longer required and can be removed from the API specification and documentation.
The server is now free to change the format of new URLs at any time without affecting clients (of course, the server must continue to honor all previously-issued URLs). The URL passed out to the client by the server will have to include the primary key of the entity in a database plus some routing information, but because the client just echoes the URL back to the server and the client is never required to parse the URL, clients do not have to know the format of the URL. This reduces coupling between the client and server. Servers can even obfuscate their URLs with base64 or similar encoding if they want to emphasize to clients that they should not make assumptions about URL formats or infer meaning from them.
In the example above, I used a relative form of the URIs in the links, for example /people/98765. It might have been slightly more convenient for the client (although less convenient for the formatting of this blog post), if I had expressed the URIs in absolute form, e.g., https://pets.org/people/98765. Clients only need to know the standard rules of URIs defined in the IETF specifications to convert between these two URI forms, so which form you choose to use is not as important as you might at first assume. Contrast this with the conversion from foreign key to URL described previously, which requires knowledge that is specific to the pet store API. Relative URLs have some advantages for server implementers, as described below, but absolute URLs are probably more convenient for most clients, which is perhaps why Google Drive and GitHub APIs use absolute URLs.
In short, using links instead of foreign keys to express relationships in APIs reduces the amount of information a client needs to know to use an API, and reduces the ways in which clients and servers are coupled to each other.
Caveats Here are some things you should think about before using links.
Many API implementations have reverse proxies in front of them for security, load-balancing, and other reasons. Some proxies like to rewrite URLs. When an API uses foreign keys to represent relationships, the only URL that has to be rewritten by a proxy is the main URL of the request. In HTTP, that URL is split between the address line (the first header line) and the host header.
In an API that uses links to express relationships, there will be other URLs in the headers and bodies of both the request and the response that would also need to be rewritten. There are a few different ways of dealing with this:
Don’t rewrite URLs in proxies. I try to avoid URL rewriting, but this may not be possible in your environment.
In the proxy, be careful to find and map all URLs wherever they appear in the header or body of the request and response. I have never done this, because it seems to me to be difficult, error-prone, and inefficient, but others may have done it.
Write all links relatively. In addition to allowing proxies some ability to rewrite URLs, relative URLs may make it easier to use the same code in test and production, because the code does not have to be configured with knowledge of its own host name. Writing links using relative URLs with a single leading slash, as I showed in the example above, has few downsides for the server or the client, but it only allows the proxy to change the host name (more precisely, the parts of the URL called the scheme and authority), not the path. Depending on the design of your URLs, you could allow proxies some ability to rewrite paths if you are willing to write links using relative URLs with no leading slashes, but I have never done this because I think it would be complicated for servers to write those URLs reliably. Relative URLs without leading slashes are also more difficult for clients to use—they need to use a standards-compliant library rather than simple string concatenation to handle those URLs, and they need to be careful to understand and preserve the base URL. Using a standards-compliant library to handle URLs is good practice for clients anyway, but many don’t.
Using links may also cause you to re-examine how you do API versioning. Many APIs like to put version numbers in URLs, like this:
This is the kind of versioning where the data for a single resource can be viewed in more than one “format” at the same time—these are not the sort of versions that replace each other in time sequence as edits are made.
This is closely analogous to being able to see the same web document in different natural languages, for which there is a web standard; it is a pity there isn’t a similar one for versions. By giving each version its own URL, you raise each version to the status of a full web resource. There is nothing wrong with “version URLs” like these, but they are not suitable for expressing links. If a client asks for Lassie in the version 2 format, it does not mean that they also want the version 2 format of Lassie’s owner, Joe, so the server can’t pick which version number to put in a link. There may not even be a version 2 format for owners. It also doesn’t make conceptual sense to use the URL of a particular version in links—Lassie is not owned by a specific version of Joe, she is owned by Joe himself. So, even if you expose a URL of the form /v1/people/98765 to identify a specific version of Joe, you should also expose the URL /people/98765 to identify Joe himself and use the latter in links. Another option is to define only the URL /people/98765 and allow clients to request a specific version by including a request header. There is no standard for this header, but calling it Accept-Version would fit well with the naming of the standard headers. I personally prefer the approach of using a header for versioning and avoiding URLs with version numbers, but URLs with version numbers are popular, and I often implement both a header and “version URLs” because it’s easier to do both than argue about it. For more on API versioning, check out this blog post.
You might still need to document some URL templates In most web APIs, the URL of a new resource is allocated by the server when the resource is created using POST. If you use this method for creation and you are using links for relationships, you do not need to publish a URI template for the URIs of these resources. However, some APIs allow the client to control the URL of a new resource. Letting clients control the URL of new resources makes many patterns of API scripting much easier for client programmers, and it also supports scenarios where an API is used to synchronize an information model with an external information source. HTTP has a special method for this purpose: PUT. PUT means “create the resource at this URL if it does not already exist, otherwise update it”1. If your API allows clients to create new entities using PUT, you have to document the rules for composing new URLs, probably by including a URI template in the API specification. You can also allow clients partial control of URLs by including a primary key-like value in the body or headers of a POST. This doesn’t require a URI template for the POST itself, but the client will still need to learn a URI template to take advantage of the resulting predictability of URIs.
The other place where it makes sense to document URL templates is when the API allows clients to encode queries in URLs. Not every API lets you query its resources, but this can be a very useful feature for clients, and it is natural to let clients encode queries in URLs and use GET to retrieve the result. The following example shows why.
In the example above we included the following name/value pair in the representation of Joe:
The client doesn’t have to know anything about the structure of this URL, beyond what’s written in standard specifications, to use it. This means that a client can get the list of Joe’s pets from this link without learning any query language, and without the API having to document its URL formats—but only if the client first does a GET on /people/98765. If, in addition, the pet store API documents a query capability, the client can compose the same or equivalent query URL to retrieve the pets for an owner without first retrieving the owner—it is sufficient to know the owner’s URI. Perhaps more importantly, the client can also form queries like the following ones that would otherwise not be possible:
The URI specification describes a portion of the HTTP URL for this purpose called the query component—the portion of the URL after the first “?” and before the first “#”. The style of query URI that I prefer always puts client-specified queries in the query component of the URI, but it’s also permissible to express client queries in the path portion of a URL. In either case, you need to describe to clients how to compose these URLs—you are effectively designing and documenting a query language specific to your API. Of course, you can also allow clients to put queries in the request body rather than the URL and use the POST method instead of GET. Since there are practical limits on the size of a URL—anything over 4k bytes is tempting fate—it is a good practice to support POST for queries even if you also support GET.
Because query is such a useful feature in APIs, and because designing and implementing query languages is not easy, technologies like GraphQL have emerged. I have never used GraphQL, so I can’t endorse it, but you may want to evaluate it as an alternative to designing and implementing your own API query capability. API query capabilities, including GraphQL, are best used as a complement to a standard HTTP API for reading and writing resources, not an alternative.
And another thing… What’s the best way to write links in JSON? Unlike HTML, JSON has no built-in mechanism for expressing links. Many people have opinions on how links should be expressed in JSON and some have published their opinions in more or less official-looking documents, but there is no standard ratified by a recognized standards organization at the time of writing. In the examples above, I used simple JSON name/value pairs to express links—this is my preferred style, and is also the style used by Google Drive and GitHub. Another style that you will likely encounter looks like this:
I personally don’t see the merits of this style, but several variants of it have achieved some level of popularity.
There is another style for links in JSON that I do like, which looks like this:
The benefit of this style is that it makes it explicit that “/people/98765” is a URL and not just a string. I learned this pattern from RDF/JSON. One reason to adopt this pattern is that you will probably have to use it anyway whenever you have to show information about one resource nested inside another, as in the following example, and using it everywhere gives a nice uniformity:
Finally, what’s the difference between an attribute and a relationship? I think most people would agree with the statement that JSON does not have a built-in mechanism for expressing links, but there is a way of thinking about JSON that says otherwise. Consider this JSON:
A common view is that shoeSize is an attribute, not a relationship, and 10 is a value, not an entity. However, it is also reasonable to say that the string ’10” is in fact a reference, written in a special notation for writing references to numbers, to the eleventh whole number, which itself is an entity. If the eleventh whole number is a perfectly good entity, and the string ’10’ is just a reference to it, then the name/value pair ‘”shoeSize”: 10’ is conceptually a link, even though it doesn’t use URIs.
The same argument can be made for booleans and strings, so all JSON name/value pairs can be viewed as links. If you think this way of looking at JSON makes sense, then it’s natural to use simple JSON name/value pairs for links to entities that are referenced using URLs in addition to those that are referenced using JSON’s built-in reference notations for numbers, strings, booleans and null.
This argument says more generally that there is no fundamental difference between attributes and relationships; attributes are just relationships between an entity and an abstract or conceptual entity like a number or color that has historically been treated specially. Admittedly, this is a rather abstract way of looking at the world—if you show most people a black cat, and ask them how many objects they see, they will say one. Not many would say they see two objects—a cat, and the color black—and a relationship between them.
Links are simply better Web APIs that pass out database keys rather than links are harder to learn and harder to use for clients. They also couple clients and servers more tightly together by requiring more shared knowledge and so they require more documentation to be written and read. Their only advantage is that because they are so common, programmers have become familiar with them and know how to produce them and consume them. If you strive to offer your clients high-quality APIs that don’t require a ton of documentation and maximize independence of clients from servers, think about exposing URLs rather than database keys in your web APIs.
Each year Google I/O brings together developers from around the globe for talks, hands-on learning, and an inside look at Google’s latest products for developers. This year, we at Google Cloud are celebrating all the ways developers can take advantage of cloud technologies to build the next generation of mobile and web applications. Here’s a look at the key announcements from Google Cloud this week at I/O.
Android phone’s built-in security key is now generally available FIDO security keys provide the strongest protection against phishing and account hijacking and are a must-have for all people who are at risk for targeted attacks, such as IT admins, executives, journalists, and activists. At Next ‘19, we announced beta functionality that lets you use your Android phone as a FIDO security key for your Google and Google Cloud accounts at no additional cost. Today, we are making this feature generally available on phones running Android 7.0+. Admins can also require users to protect their G Suite, GCP and Cloud Identity accounts with a security key, now letting them choose between using a physical security key, their Android phone, or both. To learn more, check out this blog post.
Cloud TPU V2 and V3 Pods are now publicly available in beta We introduced our Cloud TPU Pods to accelerate the largest-scale machine learning (ML) applications, to train ML models in minutes or hours instead of days or weeks. Today, for the first time, our Cloud TPU V2 and V3 Pods are publicly available, in beta, to help ML researchers, engineers, and data scientists iterate faster and train more capable machine learning models. You can learn more by reading this blog post.
Updates to Google Maps Platform We announced the launch of Google Maps Platform a year ago to provide the next generation of our Maps, Routes, and Places offerings for developers. Over the last 12 months, we’ve increased the stability, reliability, and performance of our platform to meet a 99.9% uptime SLA and provide data for over 150,000 places, routes that cover 40 million miles of road, and maps that cover 99% of the world.
Today, we announced support for WebGL-powered data visualization with the open source deck.gl library, designed specifically for visualizing large datasets. Additionally, we opened public beta for the latest version of our Maps SDK for Android, which we migrated to the same infrastructure as Google Maps—making it easier to bring you Google Maps features in future releases. To learn more, read this blog post.
Firebase updates: expanded machine learning capabilities, web performance monitoring, and analytics enhancements
Today, Firebase launched a number of updates to help mobile and web developers by making it easier to build their apps, improve app quality, and grow their businesses. Here are three to know.
ML Kit makes machine learning accessible to all app developers, regardless of machine learning experience. Developers can use out-of-the-box APIs to solve common tasks, or host and deploy their own custom models. These models are available on-device or in the cloud, through Google Cloud AI. Today, we’re launching three more capabilities in beta: the Translation API, the Object Detection & Tracking API, and AutoML Vision Edge. These new capabilities complement previous updates including solutions for natural language processing with Language Identification and Smart Reply APIs.
Firebase Performance Monitoring is a popular way for developers of native mobile apps to troubleshoot performance issues. Today, Performance Monitoring is available in beta for web apps as well, giving web developers a deeper understanding of how real users are experiencing their app in the wild.
Google Analytics for Firebase provides free, unlimited, and robust analytics for developers. Today, we’re announcing a completely rebuilt audience builder with a new interface, including new features like sequences, scoping, time windows, membership duration, and more. With it, you can create audiences for personalization through Remote Config or re-engage them through Cloud Messaging and/or the new App campaigns.
For a comprehensive round-up of the Firebase news from I/O, including these announcements and more, read this post on the Firebase blog.
Beyond the cloud There are plenty more updates to come from Google I/O. To stay up to date, follow Google Developers on Twitter,Facebook, or YouTube, or read the Google Developers blog.
