Migrating App Engine push queues to Cloud Tasks

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

Banner image that shows the Cloud Task logo

Introduction

The previous Module 7 episode of Serverless Migration Station gave developers an idea of how App Engine push tasks work and how to implement their use in an existing App Engine ndb Flask app. In this Module 8 episode, we migrate this app from the App Engine Datastore (ndb) and Task Queue (taskqueue) APIs to Cloud NDB and Cloud Tasks. This makes your app more portable and provides a smoother transition from Python 2 to 3. The same principle applies to upgrading other legacy App Engine apps from Java 8 to 11, PHP 5 to 7, and up to Go 1.12 or newer.

Over the years, many of the original App Engine services such as Datastore, Memcache, and Blobstore, have matured to become their own standalone products, for example, Cloud Datastore, Cloud Memorystore, and Cloud Storage, respectively. The same is true for App Engine Task Queues, whose functionality has been split out to Cloud Tasks (push queues) and Cloud Pub/Sub (pull queues), now accessible to developers and applications outside of App Engine.

Migrating App Engine push queues to Cloud Tasks video

Migrating to Cloud NDB and Cloud Tasks

The key updates being made to the application:

  1. Add support for Google Cloud client libraries in the app’s configuration
  2. Switch from App Engine APIs to their standalone Cloud equivalents
  3. Make required library adjustments, e.g., add use of Cloud NDB context manager
  4. Complete additional setup for Cloud Tasks
  5. Make minor updates to the task handler itself

The bulk of the updates are in #3 and #4 above, and those are reflected in the following “diff”s for the main application file:

Screenshot shows primary differences in code when switching to Cloud NDB & Cloud Tasks

Primary differences switching to Cloud NDB & Cloud Tasks

With these changes implemented, the web app works identically to that of the Module 7 sample, but both the database and task queue functionality have been completely swapped to using the standalone/unbundled Cloud NDB and Cloud Tasks libraries… congratulations!

Next steps

To do this exercise yourself, check out our corresponding codelab which leads you step-by-step through the process. You can use this in addition to the video, which can provide guidance. You can also review the push tasks migration guide for more information. Arriving at a fully-functioning Module 8 app featuring Cloud Tasks sets the stage for a larger migration ahead in Module 9. We’ve accomplished the most important step here, that is, getting off of the original App Engine legacy bundled services/APIs. The Module 9 migration from Python 2 to 3 and Cloud NDB to Cloud Firestore, plus the upgrade to the latest version of the Cloud Tasks client library are all fairly optional, but they represent a good opportunity to perform a medium-sized migration.

All migration modules, their videos (when available), codelab tutorials, and source code, can be found in the migration repo. While the content focuses initially on Python users, we will cover other legacy runtimes soon so stay tuned.

How to use App Engine push queues in Flask apps

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

Banner image that shows the Cloud Task logo

Introduction

Since its original launch in 2008, many of the core Google App Engine services such as Datastore, Memcache, and Blobstore, have matured to become their own standalone products: for example, Cloud Datastore, Cloud Memorystore, and Cloud Storage, respectively. The same is true for App Engine Task Queues with Cloud Tasks. Today’s Module 7 episode of Serverless Migration Station reviews how App Engine push tasks work, by adding this feature to an existing App Engine ndb Flask app.

App Engine push queues in Flask apps video

That app is where we left off at the end of Module 1, migrating its web framework from App Engine webapp2 to Flask. The app registers web page visits, creating a Datastore Entity for each. After a new record is created, the ten most recent visits are displayed to the end-user. If the app only shows the latest visits, there is no reason to keep older visits, so the Module 7 exercise adds a push task that deletes all visits older than the oldest one shown. Tasks execute asynchronously outside the normal application flow.

Key updates

The following are the changes being made to the application:

  1. Add use of App Engine Task Queues (taskqueue) API
  2. Determine oldest visit displayed, logging and saving that timestamp
  3. Create task to delete old visits
  4. Update web page template to display timestamp threshold
  5. Log how many and which visits (by Entity ID) are deleted

Except for #4 which occurs in the HTML template file, these updates are reflected in the “diff”s for the main application file:

Screenshot of App Engine push tasks application source code differences

Adding App Engine push tasks application source code differences

With these changes implemented, the web app now shows the end-user which visits will be deleted by the new push task:

Screenshot of VisitMe example showing last ten site visits. A red circle around older visits being deleted

Sample application output

Next steps

To do this exercise yourself, check out our corresponding codelab which leads you step-by-step through the process. You can use this in addition to the video, which can provide guidance. You can also review the push queue documentation for more information. Arriving at a fully-functioning Module 7 app featuring App Engine push tasks sets the stage for migrating it to Cloud Tasks (and Cloud NDB) ahead in Module 8.

All migration modules, their videos (when available), codelab tutorials, and source code, can be found in the migration repo. While the content focuses initially on Python users, we will cover other legacy runtimes soon so stay tuned.

Exploring serverless with a nebulous app: Deploy the same app to App Engine, Cloud Functions, or Cloud Run

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

Banner image that shows the App Engine, Cloud Functions, and Cloud Run logos

Introduction

Google Cloud offers three distinct ways of running your code or application in a serverless way, each serving different use cases. Google App Engine, our first Cloud product, was created to give users the ability to deploy source-based web applications or mobile backends directly to the cloud without the need of thinking about servers or scaling. Cloud Functions came later for scenarios where you may not have an entire app, great for one-off utility functions or event-driven microservices. Cloud Run is our latest fully-managed serverless product that gives developers the flexibility of containers along with the convenience of serverless.

As all are serverless compute platforms, users recognize they share some similarities along with clear differences, and often, they ask:

  1. How different is deploying code to App Engine, Cloud Functions, or Cloud Run?
  2. Is it challenging to move from one to another if I feel the other may better fit my needs?

We’re going to answer these questions today by sharing a unique application with you, one that can be deployed to all three platforms without changing any application code. All of the necessary changes are done in configuration.

More motivation

Another challenge for developers can be trying to learn how to use another Cloud product, such as this request, paraphrased from a user:

  1. I have a Google App Engine app
  2. I want to call the Cloud Translation API from that app

Sounds simple enough. This user went straight to the App Engine and Translation API documentation where they were able to get started with the App Engine Quickstart to get their app up and going, then found the Translation API setup page and started looking into permissions needed to access the API. However, they got stuck at the Identity and Access Management (IAM) page on roles, being overwhelmed at all the options but no clear path forward. In light of this, let’s add a third question to preceding pair outlined earlier:

  1. How do you access Cloud APIs from a Cloud serverless platform?

Without knowing what that user was going to build, let’s just implement a barebones translator, an “MVP” (minimally viable product) version of a simple “My Google Translate” Python Flask app using the Translation API, one of Google Cloud’s AI/ML “building block” APIs. These APIs are backed by pre-trained machine learning models, giving developers with little or no background in AI/ML the ability to leverage the benefits of machine learning with only API calls.

The application

The app consists of a simple web page prompting the user for a phrase to translate from English to Spanish. The translated results along with the original phrase are presented along with an empty form for a follow-up translation if desired. While the majority of this app’s deployments are in Python 3, there are still many users working on upgrading from Python 2, so some of those deployments are available to help with migration planning. Taking this into account, this app can be deployed (at least) eight different ways:

  1. Local (or hosted) Flask server (Python 2)
  2. Local (or hosted) Flask server (Python 3)
  3. Google App Engine (Python 2)
  4. Google App Engine (Python 3)
  5. Google Cloud Functions (Python 3)
  6. Google Cloud Run (Python 2 via Docker)
  7. Google Cloud Run (Python 3 via Docker)
  8. Google Cloud Run (Python 3 via Cloud Buildpacks)

The following is a brief glance at the files and which configurations they’re for: Screenshot of Nebulous serverless sample app files

Nebulous serverless sample app files

Diving straight into the application, let’s look at its primary function, translate():

@app.route('/', methods=['GET', 'POST'])
def translate(gcf_request=None):
local_request = gcf_request if gcf_request else request
text = translated = None
if local_request.method == 'POST':
text = local_request.form['text'].strip()
if text:
data = {
'contents': [text],
'parent': PARENT,
'target_language_code': TARGET[0],
}
rsp = TRANSLATE.translate_text(request=data)
translated = rsp.translations[0].translated_text
context = {
'orig': {'text': text, 'lc': SOURCE},
'trans': {'text': translated, 'lc': TARGET},
}
return render_template('index.html', **context)

Core component (translate()) of sample application

Some key app components:

  • Upon an initial request (GET), an HTML template is rendered featuring a simple form with an empty text field for the text to translate.
  • The form POSTs back to the app, and in this case, grabs the text to translate, sends the request to the Translation API, receives and displays the results to the user along with an empty form for another translation.
  • There is a special “ifdef” for Cloud Functions near the top to receive a request object because a web framework isn’t used like you’d have with App Engine or Cloud Run, so Cloud Functions provides one for this reason.

The app runs identically whether running locally or deployed to App Engine, Cloud Functions, or Cloud Run. The magic is all in the configuration. The requirements.txt file* is used in all configurations, whether to install third-party packages locally, or to direct the Cloud Build system to automatically install those libraries during deployment. Beyond requirements.txt, things start to differ:

  1. App Engine has an app.yaml file and possibly an appengine_config.py file.
  2. Cloud Run has either a Dockerfile (Docker) or Procfile (Cloud Buildpacks), and possibly a service.yaml file.
  3. Cloud Functions, the “simplest” of the three, has no configuration outside of a package requirements file (requirements.txt, package.json, etc.).

The following is what you should expect to see after completing one translation request: Screenshot of My Google Translate (1990s Edition) in Incognito Window

“My Google Translate” MVP app (Cloud Run edition)

Next steps

The sample app can be run locally or on your own hosting server, but now you also know how to deploy it to each of Cloud’s serverless platforms and what those subtle differences are. You also have a sense of the differences between each platform as well as what it takes to switch from one to another. Lastly, you now know how to access Cloud APIs from these platforms.

The user described earlier was overwhelmed at all the IAM roles and options available because this type of detail is required to provide the most security options for accessing Cloud services, but when prototyping, the fastest on-ramp is to use the default service account that comes with Cloud serverless platforms. These help you get that prototype working while allowing you to learn more about IAM roles and required permissions. Once you’ve progressed far enough to consider deploying to production, you can then follow the best practice of “least privileges” and create your own (user-managed) service accounts with the minimal permissions required so your application functions properly.

To dive in, the code and codelabs (free, self-paced, hands-on tutorials) for each deployment are available in its open source repository. An active Google Cloud billing account is required to deploy this application to each of our serverless platforms even though you can do all of them without incurring charges. More information can be found in the “Cost” section of the repo’s README. We hope this sample app teaches you more about the similarities and differences between our plaforms, shows you how you can “shift” applications comfortably between them, and provides a light introduction to another Cloud API. Also check out my colleague’s post featuring similar content for Node.js.

Cloud NDB to Cloud Datastore migration

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

An optional migration

Serverless Migration Station is a mini-series from Serverless Expeditions focused on helping users on one of Google Cloud’s serverless compute platforms modernize their applications. The video today demonstrates how to migrate a sample app from Cloud NDB (or App Engine ndb) to Cloud Datastore. While Cloud NDB suffices as a current solution for today’s App Engine developers, this optional migration is for those who want to consolidate their app code to using a single client library to talk to Datastore.

Cloud Datastore started as Google App Engine’s original database but matured to becoming its own standalone product in 2013. At that time, native client libraries were created for the new product so non-App Engine apps as well as App Engine second generation apps could access the service. Long-time developers have been using the original App Engine service APIs to access Datastore; for Python, this would be App Engine ndb. While the legacy ndb service is still available, its limitations and lack of availability in Python 3 are why we recommend users switch to standalone libraries like Cloud NDB in the preceding video in this series.

While Cloud NDB lets users break free from proprietary App Engine services and upgrade their applications to Python 3, it also gives non-App Engine apps access to Datastore. However, Cloud NDB’s primary role is a transition tool for Python 2 App Engine developers. Non-App Engine developers and new Python 3 App Engine developers are directed to the Cloud Datastore native client library, not Cloud NDB.

As a result, those with a collection of Python 2 or Python 3 App Engine apps as well as non-App Engine apps may be using completely different libraries (ndb, Cloud NDB, Cloud Datastore) to connect to the same Datastore product. Following the best practices of code reuse, developers should consider consolidating to a single client library to access Datastore. Shared libraries provide stability and robustness with code that’s constantly tested, debugged, and battle-proven. Module 2 showed users how to migrate from App Engine ndb to Cloud NDB, and today’s Module 3 content focuses on migrating from Cloud NDB to Cloud Datastore. Users can also go straight from ndb directly to Cloud Datastore, skipping Cloud NDB entirely.

Migration sample and next steps

Cloud NDB follows an object model identical to App Engine ndb and is deliberately meant to be familiar to long-time Python App Engine developers while use of the Cloud Datastore client library is more like accessing a JSON document store. Their querying styles are also similar. You can compare and contrast them in the “diffs” screenshot below and in the video.

The diffs between the Cloud NDB and Cloud Datastore versions of the sample app

The “diffs” between the Cloud NDB and Cloud Datastore versions of the sample app

All that said, this migration is optional and only useful if you wish to consolidate to using a single client library. If your Python App Engine apps are stable with ndb or Cloud NDB, and you don’t have any code using Cloud Datastore, there’s no real reason to move unless Cloud Datastore has a compelling feature inaccessible from your current client library. If you are considering this migration and want to try it on a sample app before considering for yours, see the corresponding codelab and use the video for guidance.

It begins with the Module 2 code completed in the previous codelab/video; use your solution or ours as the “START”. Both Python 2 (Module 2a folder) and Python 3 (Module 2b folder) versions are available. The goal is to arrive at the “FINISH” with an identical, working app but using a completely different Datastore client library. Our Python 2 FINISH can be found in the Module 3a folder while Python 3’s FINISH is in the Module 3b folder. If something goes wrong during your migration, you can always rollback to START, or compare your solution with our FINISH. We will continue our Datastore discussion ahead in Module 6 as Cloud Firestore represents the next generation of the Datastore service.

All of these learning modules, corresponding videos (when published), codelab tutorials, START and FINISH code, etc., can be found in the migration repo. We hope to also one day cover other legacy runtimes like Java 8 and others, so stay tuned. Up next in Module 4, we’ll take a different turn and showcase a product crossover, showing App Engine developers how to containerize their apps and migrate them to Cloud Run, our scalable container-hosting service in the cloud. If you can’t wait for either Modules 4 or 6, try out their respective codelabs or access the code samples in the table at the repo above. Migrations aren’t always easy, and we hope content like this helps you modernize your apps.

Migrating from App Engine ndb to Cloud NDB

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

Migrating to standalone services

Today we’re introducing the first video showing long-time App Engine developers how to migrate from the App Engine ndb client library that connects to Datastore. While the legacy App Engine ndb service is still available for Datastore access, new features and continuing innovation are going into Cloud Datastore, so we recommend Python 2 users switch to standalone product client libraries like Cloud NDB.

This video and its corresponding codelab show developers how to migrate the sample app introduced in a previous video and gives them hands-on experience performing the migration on a simple app before tackling their own applications. In the immediately preceding “migration module” video, we transitioned that app from App Engine’s original webapp2 framework to Flask, a popular framework in the Python community. Today’s Module 2 content picks up where that Module 1 leaves off, migrating Datastore access from App Engine ndb to Cloud NDB.

Migrating to Cloud NDB opens the doors to other modernizations, such as moving to other standalone services that succeed the original App Engine legacy services, (finally) porting to Python 3, breaking up large apps into microservices for Cloud Functions, or containerizing App Engine apps for Cloud Run.

Moving to Cloud NDB

App Engine’s Datastore matured to becoming its own standalone product in 2013, Cloud Datastore. Cloud NDB is the replacement client library designed for App Engine ndb users to preserve much of their existing code and user experience. Cloud NDB is available in both Python 2 and 3, meaning it can help expedite a Python 3 upgrade to the second generation App Engine platform. Furthermore, Cloud NDB gives non-App Engine apps access to Cloud Datastore.

As you can see from the screenshot below, one key difference between both libraries is that Cloud NDB provides a context manager, meaning you would use the Python with statement in a similar way as opening files but for Datastore access. However, aside from moving code inside with blocks, no other changes are required of the original App Engine ndb app code that accesses Datastore. Of course your “YMMV” (your mileage may vary) depending on the complexity of your code, but the goal of the team is to provide as seamless of a transition as possible as well as to preserve “ndb“-style access.

The difference between the App Engine ndb and Cloud NDB versions of the sample app

The “diffs” between the App Engine ndb and Cloud NDB versions of the sample app

Next steps

To try this migration yourself, hit up the corresponding codelab and use the video for guidance. This Module 2 migration sample “STARTs” with the Module 1 code completed in the previous codelab (and video). Users can use their solution or grab ours in the Module 1 repo folder. The goal is to arrive at the end with an identical, working app that operates just like the Module 1 app but uses a completely different Datastore client library. You can find this “FINISH” code sample in the Module 2a folder. If something goes wrong during your migration, you can always rollback to START, or compare your solution with our FINISH. Bonus content migrating to Python 3 App Engine can also be found in the video and codelab, resulting in a second FINISH, the Module 2b folder.

All of these learning modules, corresponding videos (when published), codelab tutorials, START and FINISH code, etc., can be found in the migration repo. We hope to also one day cover other legacy runtimes like Java 8 and others, so stay tuned! Developers should also check out the official Cloud NDB migration guide which provides more migration details, including key differences between both client libraries.

Ahead in Module 3, we will continue the Cloud NDB discussion and present our first optional migration, helping users move from Cloud NDB to the native Cloud Datastore client library. If you can’t wait, try out its codelab found in the table at the repo above. Migrations aren’t always easy; we hope this content helps you modernize your apps and shows we’re focused on helping existing users as much as new ones.

Migrating from App Engine webapp2 to Flask

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

graphic showing movement with arrows,. settings, lines, and more

Migrating web framework

The Google Cloud team recently introduced a series of codelabs (free, self-paced, hands-on tutorials) and corresponding videos designed to help users on one of our serverless compute platforms modernize their apps, with an initial focus on our earliest users running their apps on Google App Engine. We kick off this content by showing users how to migrate from App Engine’s webapp2 web framework to Flask, a popular framework in the Python community.

While users have always been able to use other frameworks with App Engine, webapp2 comes bundled with App Engine, making it the default choice for many developers. One new requirement in App Engine’s next generation platform (which launched in 2018) is that web frameworks must do their own routing, which unfortunately, means that webapp2 is no longer supported, so here we are. The good news is that as a result, modern App Engine is more flexible, lets users to develop in a more idiomatic fashion, and makes their apps more portable.

For example, while webapp2 apps can run on App Engine, Flask apps can run on App Engine, your servers, your data centers, or even on other clouds! Furthermore, Flask has more users, more published resources, and is better supported. If Flask isn’t right for you, you can select from other WSGI-compliant frameworks such as Django, Pyramid, and others.

Video and codelab content

In this “Module 1” episode of Serverless Migration Station (part of the Serverless Expeditions series) Google engineer Martin Omander and I explore this migration and walk developers through it step-by-step.

In the previous video, we introduced developers to the baseline Python 2 App Engine NDB webapp2 sample app that we’re taking through each of the migrations. In the video above, users see that the majority of the changes are in the main application handler, MainHandler:

The diffs between the webapp2 and Flask versions of the sample app

The “diffs” between the webapp2 and Flask versions of the sample app

Upon (re)deploying the app, users should see no visible changes to the output from the original version:

VisitMe application sample output

VisitMe application sample output

Next steps

Today’s video picks up from where we left off: the Python 2 baseline app in its Module 0 repo folder. We call this the “START”. By the time the migration has completed, the resulting source code, called “FINISH”, can be found in the Module 1 repo folder. If you mess up partway through, you can rewind back to the START, or compare your solution with ours, FINISH. We also hope to one day provide a Python 3 version as well as cover other legacy runtimes like Java 8, PHP 5, and Go 1.11 and earlier, so stay tuned!

All of the migration learning modules, corresponding videos (when published), codelab tutorials, START and FINISH code, etc., can all be found in the migration repo. The next video (Module 2) will cover migrating from App Engine’s ndb library for Datastore to Cloud NDB. We hope you find all these resources helpful in your quest to modernize your serverless apps!

Introducing “Serverless Migration Station” Learning Modules

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

graphic showing movement with arrows,. settings, lines, and more

Helping users modernize their serverless apps

Earlier this year, the Google Cloud team introduced a series of codelabs (free, online, self-paced, hands-on tutorials) designed for technical practitioners modernizing their serverless applications. Today, we’re excited to announce companion videos, forming a set of “learning modules” made up of these videos and their corresponding codelab tutorials. Modernizing your applications allows you to access continuing product innovation and experience a more open Google Cloud. The initial content is designed with App Engine developers in mind, our earliest users, to help you take advantage of the latest features in Google Cloud. Here are some of the key migrations and why they benefit you:

  • Migrate to Cloud NDB: App Engine’s legacy ndb library used to access Datastore is tied to Python 2 (which has been sunset by its community). Cloud NDB gives developers the same NDB-style Datastore access but is Python 2-3 compatible and allows Datastore to be used outside of App Engine.
  • Migrate to Cloud Run: There has been a continuing shift towards containerization, an app modernization process making apps more portable and deployments more easily reproducible. If you appreciate App Engine’s easy deployment and autoscaling capabilities, you can get the same by containerizing your App Engine apps for Cloud Run.
  • Migrate to Cloud Tasks: while the legacy App Engine taskqueue service is still available, new features and continuing innovation are going into Cloud Tasks, its standalone equivalent letting users create and execute App Engine and non-App Engine tasks.

The “Serverless Migration Station” videos are part of the long-running Serverless Expeditions series you may already be familiar with. In each video, Google engineer Martin Omander and I explore a variety of different modernization techniques. Viewers will be given an overview of the task at hand, a deeper-dive screencast takes a closer look at the code or configuration files, and most importantly, illustrates to developers the migration steps necessary to transform the same sample app across each migration.

Sample app

The baseline sample app is a simple Python 2 App Engine NDB and webapp2 application. It registers every web page visit (saving visiting IP address and browser/client type) and displays the most recent queries. The entire application is shown below, featuring Visit as the data Kind, the store_visit() and fetch_visits() functions, and the main application handler, MainHandler.


import os
import webapp2
from google.appengine.ext import ndb
from google.appengine.ext.webapp import template

class Visit(ndb.Model):
'Visit entity registers visitor IP address & timestamp'
visitor = ndb.StringProperty()
timestamp = ndb.DateTimeProperty(auto_now_add=True)

def store_visit(remote_addr, user_agent):
'create new Visit entity in Datastore'
Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put()

def fetch_visits(limit):
'get most recent visits'
return (v.to_dict() for v in Visit.query().order(
-Visit.timestamp).fetch(limit))

class MainHandler(webapp2.RequestHandler):
'main application (GET) handler'
def get(self):
store_visit(self.request.remote_addr, self.request.user_agent)
visits = fetch_visits(10)
tmpl = os.path.join(os.path.dirname(__file__), 'index.html')
self.response.out.write(template.render(tmpl, {'visits': visits}))

app = webapp2.WSGIApplication([
('/', MainHandler),
], debug=True)

Baseline sample application code

Upon deploying this application to App Engine, users will get output similar to the following:

image of a website with text saying VisitMe example

VisitMe application sample output

This application is the subject of today’s launch video, and the main.py file above along with other application and configuration files can be found in the Module 0 repo folder.

Next steps

Each migration learning module covers one modernization technique. A video outlines the migration while the codelab leads developers through it. Developers will always get a starting codebase (“START”) and learn how to do a specific migration, resulting in a completed codebase (“FINISH”). Developers can hit the reset button (back to START) if something goes wrong or compare their solutions to ours (FINISH). The hands-on experience helps users build muscle-memory for when they’re ready to do their own migrations.

All of the migration learning modules, corresponding Serverless Migration Station videos (when published), codelab tutorials, START and FINISH code, etc., can all be found in the migration repo. While there’s an initial focus on Python 2 and App Engine, you’ll also find content for Python 3 users as well as non-App Engine users. We’re looking into similar content for other legacy languages as well so stay tuned. We hope you find all these resources helpful in your quest to modernize your serverless apps!

Modernizing your Google App Engine applications

Posted by Wesley Chun, Developer Advocate, Google Cloud

Modernizing your Google App Engine applications header

Next generation service

Since its initial launch in 2008 as the first product from Google Cloud, Google App Engine, our fully-managed serverless app-hosting platform, has been used by many developers worldwide. Since then, the product team has continued to innovate on the platform: introducing new services, extending quotas, supporting new languages, and adding a Flexible environment to support more runtimes, including the ability to serve containerized applications.

With many original App Engine services maturing to become their own standalone Cloud products along with users’ desire for a more open cloud, the next generation App Engine launched in 2018 without those bundled proprietary services, but coupled with desired language support such as Python 3 and PHP 7 as well as introducing Node.js 8. As a result, users have more options, and their apps are more portable.

With the sunset of Python 2, Java 8, PHP 5, and Go 1.11, by their respective communities, Google Cloud has assured users by expressing continued long-term support of these legacy runtimes, including maintaining the Python 2 runtime. So while there is no requirement for users to migrate, developers themselves are expressing interest in updating their applications to the latest language releases.

Google Cloud has created a set of migration guides for users modernizing from Python 2 to 3, Java 8 to 11, PHP 5 to 7, and Go 1.11 to 1.12+ as well as a summary of what is available in both first and second generation runtimes. However, moving from bundled to unbundled services may not be intuitive to developers, so today we’re introducing additional resources to help users in this endeavor: App Engine “migration modules” with hands-on “codelab” tutorials and code examples, starting with Python.

Migration modules

Each module represents a single modernization technique. Some are strongly recommended, others less so, and, at the other end of the spectrum, some are quite optional. We will guide you as far as which ones are more important. Similarly, there’s no real order of modules to look at since it depends on which bundled services your apps use. Yes, some modules must be completed before others, but again, you’ll be guided as far as “what’s next.”

More specifically, modules focus on the code changes that need to be implemented, not changes in new programming language releases as those are not within the domain of Google products. The purpose of these modules is to help reduce the friction developers may encounter when adapting their apps for the next-generation platform.

Central to the migration modules are the codelabs: free, online, self-paced, hands-on tutorials. The purpose of Google codelabs is to teach developers one new skill while giving them hands-on experience, and there are codelabs just for Google Cloud users. The migration codelabs are no exception, teaching developers one specific migration technique.

Developers following the tutorials will make the appropriate updates on a sample app, giving them the “muscle memory” needed to do the same (or similar) with their applications. Each codelab begins with an initial baseline app (“START”), leads users through the necessary steps, then concludes with an ending code repo (“FINISH”) they can compare against their completed effort. Here are some of the initial modules being announced today:

  • Web framework migration from webapp2 to Flask
  • Updating from App Engine ndb to Google Cloud NDB client libraries for Datastore access
  • Upgrading from the Google Cloud NDB to Cloud Datastore client libraries
  • Moving from App Engine taskqueue to Google Cloud Tasks
  • Containerizing App Engine applications to execute on Cloud Run

Examples

What should you expect from the migration codelabs? Let’s preview a pair, starting with the web framework: below is the main driver for a simple webapp2-based “guestbook” app registering website visits as Datastore entities:

class MainHandler(webapp2.RequestHandler):
'main application (GET) handler'
def get(self):
store_visit(self.request.remote_addr, self.request.user_agent)
visits = fetch_visits(LIMIT)
tmpl = os.path.join(os.path.dirname(__file__), 'index.html')
self.response.out.write(template.render(tmpl, {'visits': visits}))

A “visit” consists of a request’s IP address and user agent. After visit registration, the app queries for the latest LIMIT visits to display to the end-user via the app’s HTML template. The tutorial leads developers a migration to Flask, a web framework with broader support in the Python community. An Flask equivalent app will use decorated functions rather than webapp2‘s object model:

@app.route('/')
def root():
'main application (GET) handler'
store_visit(request.remote_addr, request.user_agent)
visits = fetch_visits(LIMIT)
return render_template('index.html', visits=visits)

The framework codelab walks users through this and other required code changes in its sample app. Since Flask is more broadly used, this makes your apps more portable.

The second example pertains to Datastore access. Whether you’re using App Engine’s ndb or the Cloud NDB client libraries, the code to query the Datastore for the most recent limit visits may look like this:

def fetch_visits(limit):
'get most recent visits'
query = Visit.query()
visits = query.order(-Visit.timestamp).fetch(limit)
return (v.to_dict() for v in visits)

If you decide to switch to the Cloud Datastore client library, that code would be converted to:

def fetch_visits(limit):
'get most recent visits'
query = DS_CLIENT.query(kind='Visit')
query.order = ['-timestamp']
return query.fetch(limit=limit)

The query styles are similar but different. While the sample apps are just that, samples, giving you this kind of hands-on experience is useful when planning your own application upgrades. The goal of the migration modules is to help you separate moving to the next-generation service and making programming language updates so as to avoid doing both sets of changes simultaneously.

As mentioned above, some migrations are more optional than others. For example, moving away from the App Engine bundled ndb library to Cloud NDB is strongly recommended, but because Cloud NDB is available for both Python 2 and 3, it’s not necessary for users to migrate further to Cloud Datastore nor Cloud Firestore unless they have specific reasons to do so. Moving to unbundled services is the primary step to giving users more flexibility, choices, and ultimately, makes their apps more portable.

Next steps

For those who are interested in modernizing their apps, a complete table describing each module and links to corresponding codelabs and expected START and FINISH code samples can be found in the migration module repository. We are also working on video content based on these migration modules as well as producing similar content for Java, so stay tuned.

In addition to the migration modules, our team has also setup a separate repo to support community-sourced migration samples. We hope you find all these resources helpful in your quest to modernize your App Engine apps!

Automate & Extend with Apps Script (Google Cloud for Student Developers)

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

In the previous episode of our new Google Cloud for Student Developers video series, we introduced G Suite REST APIs, showing how to enhance your applications by integrating with Gmail, Drive, Calendar, Docs, Sheets, and Slides. However, not all developers prefer the lower-level style of programming requiring the use of HTTP, OAuth2, and processing the request-response cycle of API usage. Building apps that access Google technologies is open to everyone at any level, not just advanced software engineers.

Enhancing career readiness of non-engineering majors helps make our services more inclusive and helps democratize API functionality to a broader audience. For the budding data scientist, business analyst, DevOps staff, or other technical professionals who don’t code every day as part of their profession, Google Apps Script was made just for you. Rather than thinking about development stacks, HTTP, or authorization, you access Google APIs with objects.

This video blends a standard “Hello World” example with various use cases where Apps Script shines, including cases of automation, add-ons that extend the functionality of G Suite editors like Docs, Sheets, and Slides, accessing other Google or online services, and custom functions for Google Sheets—the ability to add new spreadsheet functions.

One featured example demonstrates the power to reach multiple Google technologies in an expressive way: lots of work, not much code. What may surprise readers is that this entire app, written by a colleague years ago, is comprised of just 4 lines of code:

function sendMap() {
var sheet = SpreadsheetApp.getActiveSheet();
var address = sheet.getRange('A1').getValue();
var map = Maps.newStaticMap().addMarker(address);
GmailApp.sendEmail('[email protected]',
'Map', 'See below.', {attachments:[map]});
}

Apps Script shields its users from the complexities of authorization and “API service endpoints.” Developers only need an object to interface with a service; in this case, SpreadsheetApp to access Google Sheets, and similarly, Maps for Google Maps plus GmailApp for Gmail. Viewers can build this sample line-by-line with its corresponding codelab (a self-paced, hands-on tutorial). This example helps student (and professional) developers…

  1. Build something useful that can be extended into much more
  2. Learn how to accomplish several tasks without a lot of code
  3. Imagine what else is possible with G Suite developer tools

For further exploration, check out this video as well as this one which introduces Apps Script and presents the same code sample with more details. (Note the second video emails the map’s link, but the app has been updated to attach it instead; the code has been updated everywhere else.) You may also access the code at its open source repository. If that’s not enough, learn about other ways you can use Apps Script from its video library. Finally, stay tuned for the next pair of episodes which will cover full sample apps, one with G Suite REST APIs, and another with Apps Script.

We look forward to seeing what you build with Google Cloud.

Google Cloud for Student Developers: Accessing G Suite REST APIs

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

Recently, we introduced the “Google Cloud for Student Developers” video series to encourage students majoring in STEM fields to gain development experience using industry APIs (application programming interfaces) for career readiness. That first episode provided an overview of the G Suite developer landscape while this episode dives deeper, introducing G Suite’s HTTP-based RESTful APIs, starting with Google Drive.

The first code sample has a corresponding codelab (a self-paced, hands-on tutorial) where you can build a simple Python script that displays the first 100 files or folders in your Google Drive. The codelab helps student (and professional) developers…

  1. Realize it is something that they can accomplish
  2. Learn how to create this solution without many lines of code
  3. See what’s possible with Google Cloud APIs

While everyone is familiar with using Google Drive and its web interface, many more doors are opened when you can code Google Drive. Check this blog post and video for a more comprehensive code walkthrough as well as access the code at its open source repository. What may surprise readers is that the entire app can be boiled down to just these 3-4 lines of code (everything else is either boilerplate or security):

    DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))
files = DRIVE.files().list().execute().get('files', [])
for f in files:
print(f['name'], f['mimeType'])

Once an “API service endpoint” to Google Drive is successfully created, calling the list() method in Drive’s files() collection is all that’s needed. By default, files().list() returns the first 100 files/folders—you can set the pageSize parameter for a different amount returned.

The video provides additional ideas of what else is possible by showing you examples of using the Google Docs, Sheets, and Slides APIs, and those APIs will be accessed in a way similar to what you saw for Drive earlier. You’ll also hear about what resources are available for each API, such as documentation, code samples, and links to support pages.

If you wish to further explore coding with G Suite REST APIs, check out some additional videos for the Drive, Sheets, Gmail, Calendar, and Slides APIs. Stay tuned for the next episode which highlights the higher-level Google Apps Script developer platform.

We look forward to seeing what you build with Google Cloud!

Google Cloud for Student Developers: G Suite APIs (intro & overview)

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

Students graduating from STEM majors at universities with development experience using industry APIs (application programming interfaces) have real-world practice that can prove valuable in terms of career readiness.

To that end, the Google Cloud team is creating a “Google Cloud for Student Developers” YouTube video series crafted specifically for the student developer audience.

While viewable by developers with any experience with Google Cloud, this series focuses on developing skills that will help student developers in their future careers. Google Cloud includes a pair of well-known product groups, Google Cloud Platform (GCP) as well as G Suite. While most equate GCP for developers and G Suite for users, many don’t know that behind each G Suite application like Gmail, Google Drive, Calendar, Docs, Sheets, and Slides, are developer APIs.

The Google Cloud higher education team is happy to announce the first of a 5-episode mini-series to kickoff the video collection that shows student developers how they can code G Suite, starting with this first one introducing the G Suite developer landscape. Viewers will hear about the HTTP-based RESTful APIs as well as Google Apps Script, a serverless higher-level development environment that allows for automation, extension of G Suite app functionality, as well as integration of your apps with Gmail, Drive, Calendar, Docs, Sheets, Slides, and many more G Suite, Google, and even external services.

Succeeding episodes dig deeper into the RESTful APIs as well as Apps Script, with the final pair of videos showing students full-fledged apps they can build with G Suite developer tools. To learn more about integrating with G Suite, see its top-level documentation site and overview page as well as the set of all G Suite developer videos. Also stay tuned for new episodes in the series that focus on GCP developer tools. We look forward to seeing what you can build with G Suite, but also with GCP as well… or both at the same time!

Get smart about preparing your app for OAuth verification

Posted by Nafis Zebarjadi, Product Manager and Adam Dawes, Senior Product Manager

Project Strobe was started to help users have control over their data while giving developers more explicit rules of the road to ensure everyone is confident that their data is secure. One result of this effort has been to expand our app verification program to cover more apps and more types of data access. It is important to understand how the process works so that you can optimally build your app and streamline the verification process. Here we walk you through the process of preparing your app for OAuth verification.

Getting prepared for verification

The first thing you should do is confirm whether your app needs verification. App verification is only required if you want to launch your app widely to consumer or enterprise users and the app requests sensitive or restricted scopes. Apps that use non-sensitive scopes, are under development, or are built just for your own G Suite users are not required to go through verification. If the app is just for users within your own organization, choose the ‘Internal’ application type to restrict the app to use within your own organization and skip verification.

Once you initiate app verification, it is not easy to make updates to your app’s Google API configuration. If you make any changes while in the process, you will need to start over again, so it’s critical that you get your app ready before initiating verification to avoid delays.

Determining if your app is using sensitive or restricted scopes

The first thing you need to do is look at your code on each platform to determine which OAuth scopes (Google APIs) your service needs. Be sure to do this on every client; we often see that apps will request different scopes on different platforms, and then initiate app verification on a subset of scopes than your clients actually use. Often, you can find the scopes by searching your code for the string “www.googleapis.com/auth”. Not all legacy scopes contain that string so you may also want to find the code related to the Google API library you’re using (on the specific platform) to see what scopes are being requested, or look at our directory of scopes.

Once you have identified all of the scopes that your apps use, you can check to see whether they are sensitive or restricted by going to the Cloud Console (APIs & Services -> Credentials -> OAuth consent screen -> Scopes for Google APIs) and pressing the ‘Add scope’ button. This will bring up the following window:

Google Cloud Console’s ‘Add Scope’ to your OAuth client tool and how it highlights sensitive scopes.

If the scope has a lock icon, it means that the scope is either sensitive or restricted and that you’ll need to go through app verification before you can widely launch to Google users.

[Note that the tool only lists scopes for APIs that you’ve enabled for your project. If you don’t see a scope listed, you’ll first need to enable the corresponding API for your project from the API Library. The fact that you’re not seeing the scope used in your code may mean that you have clients set up in different projects.]

Setting up the right project structure

Apps are reviewed and approved at the project level so you’ll want to make sure that you’ve configured your clients properly before starting app verification. If you have multiple projects, each one will have to independently go through app verification.

When to add multiple clients to a project: You may have multiple clients for your app to support different platforms like Android, Web and iOS. Ideally, all of these clients should be in the same project because it will smooth out the cross-client consent experience. When clients are in the same project, users only need to provide consent to one of the clients. Other clients can automatically get tokens without forcing the user to go through the consent flow for the same requested scopes again. The user is agreeing to share data with your service regardless of which platform they happen to be using and your service terms should be the same across platforms.

When to separate clients into separate projects: Your company may also have multiple apps that you publish to users. You may or may not want to host the clients related to your different apps in the same project. Generally, if the different apps use the same login system, have the same privacy policy and users recognize the brand of the publisher of all the apps, then it makes sense to have all the clients in the same project. For example, if PersonalFinance Corp has accounting, budgeting and tax apps that all share the same login, privacy policy and users recognize the PersonalFinance Corp brand, then it is best to structure those all in the same project. However, if CoolGames publisher has lots of titles that have different login systems and different privacy policies, or users are more familiar with the individual game titles than the CoolGames brand, then you should use separate projects.

Reorganizing projects: It is not possible to move or reorganize clients once they are created. If you want to make changes, you can either choose to create new clients in a centralized project or get each app verified independently. If you create new clients in a centralized project, you’ll update your apps to use the new client and abandon the old clients. The issue you may encounter with this approach is that your app may have to obtain user consent all over again (if the user hasn’t also consented to your other client). Alternately, you can leave your clients in separate projects; however, each project will have to go through app verification independently and users will have to consent to each of your clients individually.

Setting up test vs production projects: For many developers, it is also helpful to have a parallel test project to your production project. This allows you to easily change scopes or other app properties and test behavior without having to go through app verification.

Configuring your project

If your app does need to be verified, you’ll want to make sure the information about your project is up-to-date to avoid delay.

Project Owners

As we roll out changes across our API ecosystem, it is important to make sure your projects have up-to-date contact information. We often need to send notifications about changes, and have had developers miss important updates because of incorrect contact information which has resulted in their app being unexpectedly disabled. One way to help ensure your team gets notifications is to create a Google Group that aliases to a stable group within your company (and be sure to configure the group to receive emails from non-members). Another option is to create an Organizational Resource in the Cloud Console so that your client assets can be centrally administered and recovered when owners leave the company. It’s also very good practice to ensure the owners of the Android/iOS/Web clients are also owners or editors of the project. Domain verification is also required for every app, so you will also want to add your DNS administrator to the project so that person can easily go through the process.

To update project owners, use Cloud IAM in the Cloud Console (Cloud Console -> IAM and admin -> IAM).

Branding Info and Domain Verification

Branding info includes your app’s name and logo. It is critical that these are accurate because users use these to decide whether they know and trust your app. In the verification process, we will validate that you own the brand and logo and that it matches the information on your web site. If you make changes, your previously approved branding will continue to be shown until the new information can be verified.

OAuth consent screen showing a redirect domain

You will also need to verify the domain associated with your brand. This is true even if you only have Android/iOS versions of your app because you must have a website to publicly host your privacy policy. You start the domain verification process by linking your domain to your project in the Cloud Console (APIs & Services -> Domain verification). You’ll then need to go to the Search Console to prove that you own and control the domain.

Domain verification is a key security feature for your web clients. If you have web clients in your project, each of those must have their Authorized Redirect URIs or Authorized JavaScript Origins match an already verified domain. This enables us to guarantee that OAuth tokens are only returned to your application.

Adding an authorized domain for a Google Cloud Project in the domain verification tab

Scopes

Since you’ve already identified the scopes that your app uses, you should now check to see if you can change scopes to minimize your data access. Our API User Data Policy requires that you only request information that your app needs and that you’re clear to the user about how you will use it. It’s inappropriate to gain access to Google user data for alternate purposes such as advertising and market research.

In particular, you’ll want to try to avoid the use of restricted scopes. The verification process for restricted scopes can take several weeks longer than sensitive scopes. It also requires significant documentation and may involve a third-party security assessment that you must pay for. Currently, only specific Gmail scopes are restricted, but we have announced that most Drive scopes are also becoming restricted in early 2020.

If your app does need to access a restricted scope, consider architecting your app such that the Google user data is only ever stored client-side on the user’s device (like a contact manager app). Storing data in the cloud or on your own servers will require you to obtain a third-party security assessment (at your expense), and could also result in significant work to resolve any security issues found during the assessment.

Once you’ve decided on the scopes your app will need, make sure that they are registered with your project and reflected in your app’s code. We’ve seen many cases where a developer’s code calls a different set of scopes than those that have been registered in the Cloud Console. If your app does this, your users will see an unverified app error. Many developers request troubleshooting help because their users are unexpectedly seeing these errors even though their app was approved. Inevitably, it is because their code does not match what was verified. Similarly, if you need to add new scopes to your application, you’ll need to get those scopes approved before you launch the functionality into your production app (a test client is going to be essential here).

While you’re thinking about scopes, you should also consider how and when you are asking your users for consent. The best practice is to not request scopes at sign-in, but to use incremental authorization to allow a user to access a particular feature when they want it. This is a great way to build trust because the user interacts in a particular feature, can see the benefit of the feature, and understands why granting a particular permission will make the feature more useful.

Privacy Policy

Our goal in verifying apps is to ensure that any data users choose to share with third-parties is well-managed and meets users’ expectations about how it will be used. Your privacy policy is your public contract to your users and a critical proof to us that users’ expectations will be met.

You must include a link to your privacy policy on your website. If the domain where you host that policy isn’t verified, we won’t verify your app. If your app is purely mobile, with no server-side component, you will still need a privacy policy, but it may be very simple and describe that your app only stores data on a user’s device.

Google can not provide guidance on your privacy policy, but if your app requests restricted scopes, we will scrutinize your policy to understand how you plan to use that data and ensure that it conforms to our requirements. Make sure you understand the Limited Use requirements, and consult with your legal counsel to ensure that your privacy policy is consistent with the requirements. To ensure clarity in how your app handles email content, we also recommend adding the following statement to your application’s home page: “App’s use of information received from Gmail APIs will adhere to Google’s Limited Use Requirements.” This is needed when your privacy policy is not specific in how email content is used.

Submitting your app for verification

Once you have your project(s) configured with all the appropriate information, you can submit your app for verification. We have three different types of app verification depending on the scopes you request, each taking a different amount of time to complete. If you start your verification with one set of scopes and later decide you need different scopes, you usually need to finish your existing verification before you can start the process again. This could cause frustration and lengthen your overall verification process.

Brand Verification (2-3 days)

Brand Verification is our simplest process and validates that your brand name and logo belong to you. It is an optional step if your app is requesting non-sensitive scopes like Google Sign-In and typically takes just 2 to 3 business days. If your app doesn’t go through brand verification, users will only see your domain name listed on the consent page.

Sensitive Scope Verification (3-5 days)

Starting in June 2019, we greatly expanded the classification of sensitive scopes and started requiring more extensive verification for new apps that are accessing those scopes. Existing apps that are already accessing sensitive scopes need to go through this verification process in the latter half of 2019.

Sensitive scope verification involves brand and domain validation, checking that the privacy policy is prominently available from your application home page. We also review your app and privacy policy against our API Services: User Data Policy and check for deceptive practices. The privacy policy must disclose the manner in which your application accesses, uses, stores, or shares Google user data. Your use of Google user data must be limited to the practices disclosed in your published privacy policy.

A YouTube or accessible Drive video will also be required to understand how users will experience your request for scopes, showing specifically how they’ll benefit from granting you access. The identity of your app needs to be clear from the video (including the app’s client ID), and you’ll need to highlight the value proposition you communicate to the user before requesting the scopes.

Until verification is completed, users will see an unverified app page when your app requests a scope requiring verification. Up to 100 users may choose to grant access while your app is unverified. After that, users will be blocked from granting access to your app until verification is complete.

Unverified app screen on mobile

Sensitive scope verification usually takes 3 to 5 business days if there aren’t any issues with your app.

Restricted Scope Verification (4-6 weeks)

Restricted scope verification is a much more involved process. In addition to going through all the steps for a sensitive scope verification, your app will also have a much more rigorous privacy policy review to ensure that your use of Google user data conforms to our Limited Use requirements. Only permitted application types will be considered for access to restricted scopes. Finally, if your app stores data on a server, you will need to pass an annual security assessment.

We do error validation before allowing you to click the ‘Submit for Verification’ button. Here are some common reasons why the button is not clickable:

  • No verified domain
  • Privacy policy URL, authorized redirect URIs or origins for your client do not match an authorized domain
  • No new scopes added to the project that require verification

When you submit your app for verification, you will need to provide a written explanation for why your app needs the requested scopes. This explanation should include the nature of the feature and how the user will benefit from using it. It’s also best to include a link to your YouTube video in the original submission to save some back and forth with the review team.

You’ll also be asked again what email should receive questions and notifications about the verification process. Make sure you provide an address that you pay attention to and can receive emails from outside your domain. Questions will go to the person who initiated verification (not necessarily project owners) and the contact email address provided in the verification form. We’ve seen many requests delayed because the developer hasn’t responded to questions from the verification team.

Example form for a project requesting verification

Responding to verification questions

Apps with sensitive and restricted scopes often need to answer questions from the verification team. If you believe it has taken a long time to get a response from the verification team, you should search your inbox for messages from ‘api-oauth-dev-verification-reply’ to ensure that you haven’t missed anything.

By following these guidelines for submitting your app for verification, you can greatly streamline the process of getting your app approved and released to the Google user community. If you have any follow-up questions, be sure to scan the OAuth API Verification FAQ.