Migrating from App Engine Users to Cloud Identity Platform (Module 21)

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

The Serverless Migration Station series is aimed at helping developers modernize their apps running one of Google Cloud’s serverless platforms. The preceding (Migration Module 20) video demonstrates how to add use of App Engine’s Users service to a Python 2 App Engine sample app. Today’s Module 21 video picks up from where that leaves off, migrating that usage to Cloud Identity Platform.
How to migrate the App Engine Users to Cloud Identity Platform
Moving away from proprietary App Engine bundled services like Users makes apps more portable, giving them enough flexibility to:

    Understanding the overall migration

    Overall, Module 21 features major changes to the Module 20 sample app, implementing a move from App Engine bundled services (NDB & Users) to standalone Cloud services (Cloud Datastore & Identity Platform). Identity Platform doesn’t know anything about App Engine admins, so that must be built, requiring the use of the Cloud Resource Manager API. Apps dependent on Python 2 have additional required updates. Let’s discuss in a bit more detail.

    Migration “parts”

    The following changes to the sample app are required:

    • Migrate from App Engine Users (server-side) to Cloud Identity Platform (client-side)
    • Migrate from App Engine NDB, the other bundled service used in Module 20, to Cloud NDB (requires use of the Cloud Datastore API)
    • Use the Cloud Resource Manager* (via its API) to fetch the Cloud project’s IAM allow policy to collate the set of App Engine admin users for the app.
    • Use the Firebase Admin SDK to validate whether the user is an App Engine admin
    • Migrate from Python 2 to 3 (and possibly back to Python 2 [more on this below])
     

    *At the time of this writing, the Resource Manager documentation only features setup instructions for accessing the API from the lower-level Google APIs client library rather than the Resource Manager client library. To learn how to set up the latter, go to the Resource Manager client library documentation directly. The lower-level client library should only be used in circumstances when a Cloud client library doesn’t exist or doesn’t have the features your app needs. One such use case is Python 2, and we’ll be covering that shortly.

     

      Move from App Engine bundled services to standalone Cloud services

      The NDB to Cloud NDB migration is identical to the Module 2 migration content, so it’s not covered in-depth here in Module 21. The primary focus is on switching to Identity Platform to continue supporting user logins as well as implementing use of the Resource Manager and Firebase Admin SDK to build a proxy for recognizing App Engine admin users as provided by the Users service. Below is pseudocode implementing the key changes to the main application where new or updated lines of code are bolded:

      Table showing changes in code 'Before'(Module 20) and 'After'(Module 21)
      Migrating from App Engine Users to Cloud Identity Platform(click to enlarge)

      The key differences to note:

      1. The server-side Users service code vanishes from the main application, moving into the (client-side) web template (not shown here).
      2. Practically all of the new code in the Module 21 app above is for recognizing App Engine admin users. There are no changes to app operations or data models other than Cloud NDB requiring use of Python context managers to wrap all Datastore code (using Python with blocks).

      Complete versions of the app before and after the updates can be found in the Module 20 (Python 2) and Module 21 (Python 3) repo folders, respectively. In addition to the video, be sure to check out the Identity Platform documentation as well as the Module 21 codelab which leads you step-by-step through the migrations discussed.

      Aside from the necessary coding changes as well as moving from server-side to client-side, note that the Users service usage is covered by App Engine’s pricing model while Identity Platform is an independent Cloud service billed by MAUs (monthly active users), so costs should be taken into account if migrating. More information can be found in the Identity Platform pricing documentation.

      Python 2 considerations

      With the sunset of Python 2, Java 8, PHP 5, and Go 1.11, by their respective communities, Google Cloud has assured users by expressing continued long-term support of these legacy App Engine runtimes, including maintaining the Python 2 runtime. So while there is no current requirement for users to migrate, developers themselves are expressing interest in updating their applications to the latest language releases.
      The primary Module 21 migration automatically includes a port from Python 2 to 3 as that’s where most developers are headed. For those with dependencies requiring remaining on Python 2, some additional effort is required:

        The codelab covers this backport in-depth, so check out the specific section for Python 2 users if you’re in this situation. If you don’t want to think about it, just head to the repo for a working Python 2 version of the Module 21 app.

        Wrap-up

        Module 21 features migrations of App Engine bundled services to appropriate standalone Cloud services. While we recommend users modernize their App Engine apps by moving to the latest offerings from Google Cloud, these migrations are not required. In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (that have a 1st generation runtime), meaning you don’t have to migrate to standalone services before porting your app to Python 3. You can continue using App Engine NDB and Users in Python 3 so long as you retrofit your code to access bundled services from next-generation runtimes. Then should you opt to migrate, you can do so on your own timeline.

        If you’re using other App Engine legacy services be sure to check out the other Migration Modules in this series. All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. While our content initially focuses on Python users, the Cloud team is working on covering other language runtimes, so stay tuned. For additional video content, check out our broader Serverless Expeditions series.

        How to use the App Engine Users service (Module 20)

        Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

        Introduction and background

        The Serverless Migration Station video series and corresponding codelabs aim to help App Engine developers modernize their apps, whether it’s upgrading language runtimes like from Python 2 to 3 and Java 8 to 17, or to move laterally to sister serverless platforms like Cloud Functions or Cloud Run. For developers who want more control, like being able to SSH into instances, Compute Engine VMs or GKE, our managed Kubernetes service, are also viable options.

        In order to consider moving App Engine apps to other compute services, developers must move their apps away from its original APIs (now referred to as legacy bundled services), either to Cloud standalone replacement or alternative 3rd-party services. Once no longer dependent on these proprietary services, apps become much more portable. Apps can stay on App Engine while upgrading to its 2nd-generation platform, or move to other compute platforms as listed above.

        Today’s Migration Module 20 content focuses on helping developers refamiliarize themselves with App Engine’s Users service, a user authentication system serving as a lightweight wrapper around Google Sign-In (now called Google Identity Services). The video and its corresponding codelab (self-paced, hands-on tutorial) demonstrate how to add use of the Users service to the sample baseline app from Module 1. After adding the Users service in Module 20, Module 21 follows, showing developers how to migrate that usage to Cloud Identity Platform.

        How to use the App Engine Users service

        Adding use of Users service

        The sample app’s basic functionality consists of registering each page visit in Datastore and displaying the most recent visits. The Users service helps apps support user logins, App Engine administrative (“admin'”) users. It also provides convenient functions for generating login/logout links and retrieving basic user information for logged-in users. Below is a screenshot of the modified app which now supports user logins via the user interface (UI):
        Sample app now supports user logins and App Engine admin users (click to enlarge) 
        Below is the pseudocode reflecting the changes made to support user logins for the sample app, including integrating the Users service and updating what shows up in the UI:
        • If the user is logged in, show their “nickname” (display name or email address) and display a Logout button. If the logged-in user is an App Engine app admin, also display an “admin” badge (between nickname and Logout button).
        • If the user is not logged in, display the username generically as “user”, remove any admin badge, and display a Login button.
        Because the Users service is primarily a user-facing endeavor, the most significant changes take place in the UI, whereas the data model and core functionality of registering visits remain unchanged. The new support for user management primarily results in additional context to be rendered in the web template. New or altered code is bolded to highlight the updates.
        Table showing code 'Before'(Module 1) on left, and 'After' (Module 20) on the right
         Adding App Engine Users service usage to sample app (click to enlarge)

        Wrap-up

        Today’s “migration” consists of adding usage of the App Engine Users service to support user management and recognize App Engine admin users, starting with the Module 1 baseline app and finishing with the Module 20 app. To get hands-on experience doing it yourself, try the codelab and follow along with the video. Then you’ll be ready to upgrade to Identity Platform should you choose to do so.

        In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (that have a 1st generation runtime), meaning you are no longer required to migrate from the Users service to Identity Platform when porting your app to Python 3. You can continue using the Users service in your Python 3 app so long as you retrofit the code to access bundled services from next-generation runtimes.

        If you do want to move to Identity Platform, see the Module 21 content, including its codelab. All Serverless Migration Station content (codelabs, videos, and source code [when available]) are available at its open source repo. While we’re initially focusing on Python users, the Cloud team is covering other runtimes soon, so stay tuned. Also check out other videos in the broader Serverless Expeditions series.

        Migrating from App Engine pull tasks to Cloud Pub/Sub (Module 19)

        Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

        Introduction and background

        The Serverless Migration Station series is aimed at helping developers modernize their apps running one of Google Cloud’s serverless platforms. The preceding (Migration Module 18) video demonstrates how to add use of App Engine’s Task Queue pull tasks service to a Python 2 App Engine sample app. Today’s Module 19 video picks up from where that leaves off, migrating that pull task usage to Cloud Pub/Sub.

        Moving away from proprietary App Engine services like Task Queue makes apps more portable, giving them enough flexibility to:

         

          Understanding the migrations

          Module 19 consists of implementing three different migrations on the Module 18 sample app:

          • Migrate from App Engine NDB to Cloud NDB
          • Migrate from App Engine Task Queue pull tasks to Cloud Pub/Sub
          • Migrate from Python 2 to Python (2 and) 3

          The NDB to Cloud NDB migration is identical to the Module 2 migration content, so it’s not covered in-depth in Module 19. The original app was designed to be Python 2 and 3 compatible, so there’s no work there either. Module 19 boils down to three key updates:

          • Setup: Enable APIs and create Pub/Sub Topic & Subscription
          • How work is created: Publish Pub/Sub messages instead of adding pull tasks
          • How work is processed: Pull messages instead of leasing tasks
          Aside from these physical changes, a key hurdle to overcome is understanding the differences in terminology between pull tasks and Pub/Sub. The following chart attempts to demystify this so developers can more easily grasp how they differ:
          Table of terminology with related GAE Pull Tasks and Cloud Pub/Sub
          Terminology differences between App Engine pull tasks and Cloud Pub/Sub

          Reflecting the chart, these differences can be summarized like this:

          1. With Pull Queues, work is created in pull queues while work is sent to Pub/Sub topics
          2. Task Queue pull tasks are called messages in Pub/Sub
          3. With Task Queues, workers access pull tasks; with Pub/Sub, subscribers receive messages
          4. Leasing a pull task is the same as pulling a message from a Pub/Sub topic via a subscription
          5. Deleting a task from a pull queue when you’re done is analogous to successfully acknowledging a Pub/Sub message
          The video walks developers through the terminology as well as the code changes described above. Below is pseudocode implementing the key changes to the main application (new or updated lines of code bolded):
          Table showing changes in code Before (Module 18) on the left, and After (Module 19) on the right
          Migration from App Engine Task Queue pull tasks to Cloud Pub/Sub

          Observe how most of the code, especially app operations and data models are left relatively unchanged. The only visible changes are switching from App Engine NDB and Task Queue to Cloud NDB and Pub/Sub. Complete versions of the app before and after making the changes can be found in the Module 18 and Module 19 repo folders, respectively. In addition to the video, be sure to check out the Module 19 codelab which leads you step-by-step through the migrations discussed.

          Wrap-up

          Module 19 features a migration of App Engine pull tasks to Cloud Pub/Sub, but developers should note that Pub/Sub itself is not based on pull tasks. It is a fully-featured asynchronous, scalable messaging service that has many more features than the pull functionality provided by Task Queue. For example, Pub/Sub has other features like streaming to BigQuery and push functionality. Pub/Sub push operates differently than Task Queue push tasks, hence why we recommend push tasks be migrated to Cloud Tasks instead (see Module 8). For more information on all of its features, see the Pub/Sub documentation. Because Cloud Tasks doesn’t support pull functionality, we turn to Pub/Sub instead for pull task users.

          While we recommend users move to the latest offerings from Google Cloud, neither of those migrations are required, and should you opt to do so, can do them on your own timeline. In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (that have a 1st generation runtime), meaning you don’t have to migrate to standalone Cloud services before porting your app to Python 3. You can continue using Task Queue in Python 3 so long as you retrofit your code to access bundled services from next-generation runtimes.

          If you’re using other App Engine legacy services be sure to check out the other Migration Modules in this series. All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. While our content initially focuses on Python users, the Cloud team is working on covering other language runtimes, so stay tuned. For additional video content, check out our broader Serverless Expeditions series.

          How to use App Engine pull tasks (Module 18)

          Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

          Introduction and background

          The Serverless Migration Station mini-series helps App Engine developers modernize their apps to the latest language runtimes, such as from Python 2 to 3 or Java 8 to 17, or to sister serverless platforms Cloud Functions and Cloud Run. Another goal of this series is to demonstrate how to move away from App Engine’s original APIs (now referred to as legacy bundled services) to Cloud standalone replacement services. Once no longer dependent on these proprietary services, apps become much more portable, making them flexible enough to:

          App Engine’s Task Queue service provides infrastructure for executing tasks outside of the standard request-response workflow. Tasks may consist of workloads exceeding request timeouts or periodic tangential work. The Task Queue service provides two different queue types, push and pull, for developers to perform auxiliary work.

          Push queues are covered in Migration Modules 7-9, demonstrating how to add use of push tasks to an existing baseline app followed by steps to migrate that functionality to Cloud Tasks, the standalone successor to the Task Queues push service. We turn to pull queues in today’s video where Module 18 demonstrates how to add use of pull tasks to the same baseline sample app. Module 19 follows, showing how to migrate that usage to Cloud Pub/Sub.

          Adding use of pull queues

          In addition to registering page visits, the sample app needs to be modified to track visitors. Visits are comprised of a timestamp and visitor information such as the IP address and user agent. We’ll modify the app to use the IP address and track how many visits come from each address seen. The home page is modified to show the top visitors in addition to the most recent visits:

          Screen grab of the sample app's updated home page tracking visits and visitors
          The sample app’s updated home page tracking visits and visitors

          When visits are registered, pull tasks are created to track the visitors. The pull tasks sit patiently in the queue until they are processed in aggregate periodically. Until that happens, the top visitors table stays static. These tasks can be processed in a number of ways: periodically by a cron or Cloud Scheduler job, a separate App Engine backend service, explicitly by a user (via browser or command-line HTTP request), event-triggered Cloud Function, etc. In the tutorial, we issue a curl request to the app’s endpoint to process the enqueued tasks. When all tasks have completed, the table then reflects any changes to the current top visitors and their visit counts:

          Screen grab of processed pull tasks updated in the top visitors table
          Processed pull tasks update the top visitors table

          Below is some pseudocode representing the core part of the app that was altered to add Task Queue pull task usage, namely a new data model class, VisitorCount, to track visitor counts, enqueuing a (pull) task to update visitor counts when registering individual visits in store_visit(), and most importantly, a new function fetch_counts(), accessible via /log, to process enqueued tasks and update overall visitor counts. The bolded lines represent the new or altered code.

          Adding App Engine Task Queue pull task usage to sample app showing 'Before'[Module 1] on the left and 'After' [Module 18] with altered code on the right
          Adding App Engine Task Queue pull task usage to sample app

          Wrap-up

          This “migration” is comprised of adding Task Queue pull task usage to support tracking visitor counts to the Module 1 baseline app and arrives at the finish line with the Module 18 app. To get hands-on experience doing it yourself, do the codelab by hand and follow along with the video. Then you’ll be ready to upgrade to Cloud Pub/Sub should you choose to do so.

          In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (that have a 1st generation runtime), meaning you are no longer required to migrate pull tasks to Pub/Sub when porting your app to Python 3. You can continue using Task Queue in your Python 3 app so long as you retrofit the code to access bundled services from next-generation runtimes.

          If you do want to move to Pub/Sub, see Module 19, including its codelab. All Serverless Migration Station content (codelabs, videos, and source code) are available at its open source repo. While we’re initially focusing on Python users, the Cloud team is covering other runtimes soon, so stay tuned. Also check out other videos in the broader Serverless Expeditions series.

          Extending support for App Engine bundled services (Module 17)

          Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

          Background

          App Engine initially launched in 2008, providing a suite of bundled services making it convenient for applications to access a database (Datastore), caching service (Memcache), independent task execution (Task Queue), Google Sign-In authentication (Users), or large “blob” storage (Blobstore), or other companion services. However, apps leveraging those services can only run their apps on App Engine.

          To increase app portability and help Google move towards its goal of having the most open cloud on the market, App Engine launched its 2nd-generation service in 2018, initially removing those legacy services. The newer platform allows developers to upgrade apps to the latest language runtimes, such as moving from Python 2 to 3 or Java 8 to 11 (and today, Java 17). One of the major drawbacks to the 1st-generation runtimes is that they’re customized, proprietary, and restrictive in what you can use or can’t.

          Instead, the 2nd-generation platform uses open source runtimes, meaning ability to follow standard development practices, use common/known idioms, and have fewer restrictions of 3rd-party libraries, and obviating the need to copy or “vendor” them with your code. Unfortunately, to use these newer runtimes, migrating away from App Engine services were required because while you could upgrade language releases, there was no access to bundled services, breaking apps or requiring complete rewrites, making it a showstopper for many users.

          Due to their popularity and the desire to ease the upgrade process for customers, the App Engine team restored access to most (but not all) of those services in Fall 2021. Today’s Serverless Migration Station video demonstrates how to continue usage of bundled services available to Python 3 developers.

          Showing App Engine users how to use bundled services on Python 3

          Performing the upgrade

          Modernizing the typical Python 2 App Engine app looks something like this:
          1. Migrate from the webapp2 framework (not available in Python 3)
          2. Port from Python 2 to 3, preserve use of bundled services
          3. Optional migration to Cloud standalone or similar 3rd-party services

          The first step is to move to a standard Python web framework like Flask, Django, Pyramid, etc. Below is some pseudocode from Migration Module 1 demonstrating how to migrate from webapp2 to Flask:

          codeblocks for porting Python 2 sample app from webapp2 to Flask
          Step 1: Port Python 2 sample app from webapp2 to Flask

          The key changes are bolded in the above code snippets. Notice how the App Engine NDB code [the Visit class definition plus store_visit() and fetch_visits() functions] are unaffected by this web framework migration. The full webapp2 code sample can be found in the Module 0 repo folder while the completed migration to Flask sample is located in the Module 1 repo folder.

          After your app has ported frameworks, you’re free to upgrade to Python 3 while preserving access to the bundled services if your app uses any. Below is pseudocode demonstrating how to upgrade the same sample app to Python 3 as well as the code changes needed to continue to use App Engine NDB:

          codeblocks for porting sample app to Python 3, preserving use of NDB bundled service
          Step 2: Port sample app to Python 3, preserving use of NDB bundled service
          The original app was designed to work under both Python 2 and 3 interpreters, so no language changes were required in this case. We added an import of the new App Engine SDK followed by the key update wrapping the WSGI object so the app can access the bundled services. As before, the key updates are bolded. Some updates to configuration are also required, and those are outlined in the documentation and the (Module 17) codelab.

          The NDB code is also left untouched in this migration. Not all of the bundled services feature such a hands-free migration, and we hope to cover some of the more complex ones ahead in Module 22. Java, PHP, and Go users have it even better, requiring fewer or no code changes at all. The Python 2 Flask sample is located in the Module 1 repo folder, and the resulting Python 3 app can be found in the Module 1b repo folder.

          The immediate benefit of step two is the ability to upgrade to a more current version of language runtime. This leaves the third step of migrating off the bundled services as optional, especially if you plan on staying on App Engine for the long-term.

          Additional options

          If you decide to migrate off the bundled services, you can do so on your own timeline. It should be a consideration should you ever want to move to modern serverless platforms such as Cloud Functions or Cloud Run, to lower-level platforms because you want more control, like GKE, our managed Kubernetes service, or Compute Engine VMs.

          Step three is also where the rest of the Serverless Migration Station content may be useful:

          *code samples and codelabs available; videos forthcoming

          As far as moving to modern serverless platforms, if you want to break apart a large App Engine app into multiple microservices, consider Cloud Functions. If your organization has added containerization as part of your software development workflow, consider Cloud Run. It’s suitable for apps if you’re familiar with containers and Docker, but even if you or your team don’t have that experience, Cloud Buildpacks can do the heavy lifting for you. Here are the relevant migration modules to explore:

            Wrap-up

            Early App Engine users appreciate the convenience of the platform’s bundled services, and after listening to user feedback, adding them back to 2nd-generation runtimes is another way we can help developers modernize their apps. Whether upgrading to newer language runtimes to stay on App Engine and continue to use its bundled services, migrating to Cloud standalone products, or shifting to other serverless platforms, the Google Cloud team aims to provide the tools to help streamline your modernization efforts.

            All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. While our content initially focuses on Python users, the Cloud team is working on covering other language runtimes, so stay tuned. Today’s video features a special guest to provide a teaser of what to expect for Java. For additional video content, check out the broader Serverless Expeditions series.

            Migrating from App Engine Blobstore to Cloud Storage (Module 16)

            Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

            Introduction and background

            The most recent Serverless Migration Station video demonstrated how to add use of the App Engine’s Blobstore service to a sample Python 2 App Engine app, kicking off the first of a 2-part series on migrating away from Blobstore. In today’s Module 16 video, we complete this journey, arriving at Cloud Storage. Moving away from proprietary App Engine services like Blobstore makes apps more portable, giving them enough flexibility to:

            Showing App Engine users how to migrate to Cloud Storage

            As described previously, a Blobstore for Python 2 dependency on webapp made the Module 15 content more straightforward to implement if it was still using webapp2. To completely modernize this app here in Module 16, the following migrations should be carried out:

            • Migrate from webapp2 (and webapp) to Flask
            • Migrate from App Engine NDB to Cloud NDB
            • Migrate from App Engine Blobstore to Cloud Storage
            • Migrate from Python 2 to Python (2 and) 3

            Performing the migrations

            Prior to modifying the application code, a variety of configuration updates need to be made. Updates applying only to Python 2 feature a “Py2” designation while those migrating to Python 3 will see “Py3” annotations.

            1. Remove the built-in Jinja2 library from app.yaml—Jinja2 already comes with Flask, so remove use of the older built-in version which may possibly conflict with the contemporary Flask version you’re using. (Py2)
            2. Use of Cloud client libraries (such as those for Cloud NDB and Cloud Storage) require a pair of built-in libraries, grpcio and setuptools, so add those to app.yaml (Py2)
            3. Remove everything in app.yaml except for a valid runtime (Py3)
            4. Add Cloud NDB and Cloud Storage client libraries to requirements.txt (Py2 & Py3)
            5. Create an appengine_config.py supporting both built-in (those in app.yaml) and non built-in (those in requirements.txt) libraries used (Py2)

            The Module 15 app already migrated away from webapp2‘s (Django) templating system to Jinja2. This is useful when migrating to Flask because Jinja2 is Flask’s default template system. Switching from App Engine NDB to Cloud NDB is fairly straightforward as the latter was designed to be mostly compatible with the original. The only change visible in this sample app is to move Datastore calls into Python with blocks.

            The most significant changes occur when moving the upload and download handlers from webapp to Cloud Storage. The video and corresponding codelab go more in-depth into the necessary changes, but in summary, these are the updates required in the main application:

            1. webapp2 is replaced by Flask. Instead of using the older built-in version of Jinja2, use the version that comes with Flask.
            2. App Engine Blobstore and NDB are replaced by Cloud NDB and Cloud Storage, respectively.
            3. The webapp Blobstore handler functionality is replaced by a combination of the io standard library module plus components from Flask and Werkzeug. Furthermore, the handler classes and methods are replaced by Flask functions.
            4. The main handler class and corresponding GET and POST methods are all replaced by a single Flask function.

            The results

            With all the changes implemented, the original Module 15 app still operates identically in Module 16, starting with a form requesting a visit artifact followed by the most recents visits page:
            The sample app’s artifact prompt page


            The sample app’s most recent visits page.

            The only difference is that four migrations have been completed where all of the “infrastructure” is now taken care of by non-App Engine legacy services. Furthermore, the Module 16 app could be either a Python 2 or 3 app. As far as the end-user is concerned, “nothing happened.”

            Migrating sample app from App Engine Blobstore to Cloud Storage

            Wrap-up

            Module 16 featured four different migrations, modernizing the Module 15 app from using App Engine legacy services like NDB and Blobstore to Cloud NDB and Cloud Storage, respectively. While we recommend users move to the latest offerings from Google Cloud, migrating from Blobstore to Cloud Storage isn’t required, and should you opt to do so, can do it on your own timeline. In addition to today’s video, be sure to check out the Module 16 codelab which leads you step-by-step through the migrations discussed.

            In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (that have a 1st generation runtime), meaning you are no longer required to migrate to Cloud Storage when porting your app to Python 3. You can continue using Blobstore in your Python 3 app so long as you retrofit the code to access bundled services from next-generation runtimes.

            If you’re using other App Engine legacy services be sure to check out the other Migration Modules in this series. All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. While our content initially focuses on Python users, the Cloud team is working on covering other language runtimes, so stay tuned. For additional video content, check out our broader Serverless Expeditions series.

            Migrating from App Engine Memcache to Cloud Memorystore (Module 13)

            Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

            Introduction and background

            The previous Module 12 episode of the Serverless Migration Station video series demonstrated how to add App Engine Memcache usage to an existing app that has transitioned from the webapp2 framework to Flask. Today’s Module 13 episode continues its modernization by demonstrating how to migrate that app from Memcache to Cloud Memorystore. Moving from legacy APIs to standalone Cloud services makes apps more portable and provides an easier transition from Python 2 to 3. It also makes it possible to shift to other Cloud compute platforms should that be desired or advantageous. Developers benefit from upgrading to modern language releases and gain added flexibility in application-hosting options.

            While App Engine Memcache provides a basic, low-overhead, serverless caching service, Cloud Memorystore “takes it to the next level” as a standalone product. Rather than a proprietary caching engine, Cloud Memorystore gives users the option to select from a pair of open source engines, Memcached or Redis, each of which provides additional features unavailable from App Engine Memcache. Cloud Memorystore is typically more cost efficient at-scale, offers high availability, provides automatic backups, etc. On top of this, one Memorystore instance can be used across many applications as well as incorporates improvements to memory handling, configuration tuning, etc., gained from experience managing a huge fleet of Redis and Memcached instances.

            While Memcached is more similar to Memcache in usage/features, Redis has a much richer set of data structures that enable powerful application functionality if utilized. Redis has also been recognized as the most loved database by developers in StackOverflow’s annual developers survey, and it’s a great skill to pick up. For these reasons, we chose Redis as the caching engine for our sample app. However, if your apps’ usage of App Engine Memcache is deeper or more complex, a migration to Cloud Memorystore for Memcached may be a better option as a closer analog to Memcache.

            Migrating to Cloud Memorystore for Redis featured video


            Performing the migration

            The sample application registers individual web page “visits,” storing visitor information such as IP address and user agent. In the original app, the most recent visits are cached into Memcache for an hour and used for display if the same user continuously refreshes their browser during this period; caching is a one way to counter this abuse. New visitors or cache expiration results new visits as well as updating the cache with the most recent visits. Such functionality must be preserved when migrating to Cloud Memorystore for Redis.

            Below is pseudocode representing the core part of the app that saves new visits and queries for the most recent visits. Before, you can see how the most recent visits are cached into Memcache. After completing the migration, the underlying caching infrastructure has been swapped out in favor of Memorystore (via language-specific Redis client libraries). In this migration, we chose Redis version 5.0, and we recommend the latest versions, 5.0 and 6.x at the time of this writing, as the newest releases feature additional performance benefits, fixes to improve availability, and so on. In the code snippets below, notice how the calls between both caching systems are nearly identical. The bolded lines represent the migration-affected code managing the cached data.

            Switching from App Engine Memcache to Cloud Memorystore for Redis

            Wrap-up

            The migration covered begins with the Module 12 sample app (“START”). Migrating the caching system to Cloud Memorystore and other requisite updates results in the Module 13 sample app (“FINISH”) along with an optional port to Python 3. To practice this migration on your own to help prepare for your own migrations, follow the codelab to do it by-hand while following along in the video.

            While the code migration demonstrated seems straightforward, the most critical change is that Cloud Memorystore requires dedicated server instances. For this reason, a Serverless VPC connector is also needed to connect your App Engine app to those Memorystore instances, requiring more dedicated servers. Furthermore, neither Cloud Memorystore nor Cloud VPC are free services, and neither has an “Always free” tier quota. Before moving forward this migration, check the pricing documentation for Cloud Memorystore for Redis and Serverless VPC access to determine cost considerations before making a commitment.

            One key development that may affect your decision: In Fall 2021, the App Engine team extended support of many of the legacy bundled services like Memcache to next-generation runtimes, meaning you are no longer required to migrate to Cloud Memorystore when porting your app to Python 3. You can continue using Memcache even when upgrading to 3.x so long as you retrofit your code to access bundled services from next-generation runtimes.

            A move to Cloud Memorystore and today’s migration techniques will be here if and when you decide this is the direction you want to take for your App Engine apps. All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. While our content initially focuses on Python users, we plan to cover other language runtimes, so stay tuned. For additional video content, check out our broader Serverless Expeditions series.

            How to use App Engine Memcache in Flask apps (Module 12)

            Posted by Wesley Chun

            Background

            In our ongoing Serverless Migration Station series aimed at helping developers modernize their serverless applications, one of the key objectives for Google App Engine developers is to upgrade to the latest language runtimes, such as from Python 2 to 3 or Java 8 to 17. Another objective is to help developers learn how to move away from App Engine legacy APIs (now called “bundled services”) to Cloud standalone equivalent services. Once this has been accomplished, apps are much more portable, making them flexible enough to:

            In today’s Module 12 video, we’re going to start our journey by implementing App Engine’s Memcache bundled service, setting us up for our next move to a more complete in-cloud caching service, Cloud Memorystore. Most apps typically rely on some database, and in many situations, they can benefit from a caching layer to reduce the number of queries and improve response latency. In the video, we add use of Memcache to a Python 2 app that has already migrated web frameworks from webapp2 to Flask, providing greater portability and execution options. More importantly, it paves the way for an eventual 3.x upgrade because the Python 3 App Engine runtime does not support webapp2. We’ll cover both the 3.x and Cloud Memorystore ports next in Module 13.

            Got an older app needing an update? We can help with that.

            Adding use of Memcache

            The sample application registers individual web page “visits,” storing visitor information such as the IP address and user agent. In the original app, these values are stored immediately, and then the most recent visits are queried to display in the browser. If the same user continuously refreshes their browser, each refresh constitutes a new visit. To discourage this type of abuse, we cache the same user’s visit for an hour, returning the same cached list of most recent visits unless a new visitor arrives or an hour has elapsed since their initial visit.

            Below is pseudocode representing the core part of the app that saves new visits and queries for the most recent visits. Before, you can see how each visit is registered. After the update, the app attempts to fetch these visits from the cache. If cached results are available and “fresh” (within the hour), they’re used immediately, but if cache is empty, or a new visitor arrives, the current visit is stored as before, and this latest collection of visits is cached for an hour. The bolded lines represent the new code that manages the cached data.

            Adding App Engine Memcache usage to sample app

            Wrap-up

            Today’s “migration” began with the Module 1 sample app. We added a Memcache-based caching layer and arrived at the finish line with the Module 12 sample app. To practice this on your own, follow the codelab doing it by-hand while following the video. The Module 12 app will then be ready to upgrade to Cloud Memorystore should you choose to do so.

            In Fall 2021, the App Engine team extended support of many of the bundled services to next-generation runtimes, meaning you are no longer required to migrate to Cloud Memorystore when porting your app to Python 3. You can continue using Memcache in your Python 3 app so long as you retrofit the code to access bundled services from next-generation runtimes.

            If you do want to move to Cloud Memorystore, stay tuned for the Module 13 video or try its codelab to get a sneak peek. All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. While our content initially focuses on Python users, we hope to one day cover other language runtimes, so stay tuned. For additional video content, check out our broader Serverless Expeditions series.

            How can App Engine users take advantage of Cloud Functions?

            Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

            Introduction

            Recently, we discussed containerizing App Engine apps for Cloud Run, with or without Docker. But what about Cloud Functions… can App Engine users take advantage of that platform somehow? Back in the day, App Engine was always the right decision, because it was the only option. With Cloud Functions and Cloud Run joining in the serverless product suite, that’s no longer the case.

            Back when App Engine was the only choice, it was selected to host small, single-function apps. Yes, when it was the only option. Other developers have created huge monolithic apps for App Engine as well… because it was also the only option. Fast forward to today where code follows more service-oriented or event-driven architectures. Small apps can be moved to Cloud Functions to simplify the code and deployments while large apps could be split into smaller components, each running on Cloud Functions.

            Refactoring App Engine apps for Cloud Functions

            Small, single-function apps can be seen as a microservice, an API endpoint “that does something,” or serve some utility likely called as a result of some event in a larger multi-tiered application, say to update a database row or send a customer email message. App Engine apps require some kind web framework and routing mechanism while Cloud Function equivalents can be freed from much of those requirements. Refactoring these types of App Engine apps for Cloud Functions will like require less overhead, helps ease maintenance, and allow for common components to be shared across applications.

            Large, monolithic applications are often made up of multiple pieces of functionality bundled together in one big package, such as requisitioning a new piece of equipment, opening a customer order, authenticating users, processing payments, performing administrative tasks, and so on. By breaking this monolith up into multiple microservices into individual functions, each component can then be reused in other apps, maintenance is eased because software bugs will identify code closer to their root origins, and developers won’t step on each others’ toes.

            Migration to Cloud Functions

            In this latest episode of Serverless Migration Station, a Serverless Expeditions mini-series focused on modernizing serverless apps, we take a closer look at this product crossover, covering how to migrate App Engine code to Cloud Functions. There are several steps you need to take to prepare your code for Cloud Functions:

            • Divest from legacy App Engine “bundled services,” e.g., Datastore, Taskqueue, Memcache, Blobstore, etc.
            • Cloud Functions supports modern runtimes; upgrade to Python 3, Java 11, or PHP 7
            • If your app is a monolith, break it up into multiple independent functions. (You can also keep a monolith together and containerize it for Cloud Run as an alternative.)
            • Make appropriate application updates to support Cloud Functions

              The first three bullets are outside the scope of this video and its codelab, so we’ll focus on the last one. The changes needed for your app include the following:

              1. Remove unneeded and/or unsupported configuration
              2. Remove use of the web framework and supporting routing code
              3. For each of your functions, assign an appropriate name and install the request object it will receive when it is called.

              Regarding the last point, note that you can have multiple “endpoints” coming into a single function which processes the request path, calling other functions to handle those routes. If you have many functions in your app, separate functions for every endpoint becomes unwieldy; if large enough, your app may be more suited for Cloud Run. The sample app in this video and corresponding code sample only has one function, so having a single endpoint for that function works perfectly fine here.

              This migration series focuses on our earliest users, starting with Python 2. Regarding the first point, the app.yaml file is deleted. Next, almost all Flask resources are removed except for the template renderer (the app still needs to output the same HTML as the original App Engine app). All app routes are removed, and there’s no instantiation of the Flask app object. Finally for the last step, the main function is renamed more appropriately to visitme() along with a request object parameter.

              This “migration module” starts with the (Python 3 version of the) Module 2 sample app, applies the steps above, and arrives at the migrated Module 11 app. Implementing those required changes is illustrated by this code “diff:”

              Migration of sample app to Cloud Functions

              Next steps

              If you’re interested in trying this migration on your own, feel free to try the corresponding codelab which leads you step-by-step through this exercise and use the video for additional guidance.

              All migration modules, their videos (when published), codelab tutorials, START and FINISH code, etc., can be found in the migration repo. We hope to also one day cover other legacy runtimes like Java 8 as well as content for the next-generation Cloud Functions service, so stay tuned. If you’re curious whether it’s possible to write apps that can run on App Engine, Cloud Functions, or Cloud Run with no code changes at all, the answer is yes. Hope this content is useful for your consideration when modernizing your own serverless applications!

            An easier way to move your App Engine apps to Cloud Run

            Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

            Blue header

            An easier yet still optional migration

            In the previous episode of the Serverless Migration Station video series, developers learned how to containerize their App Engine code for Cloud Run using Docker. While Docker has gained popularity over the past decade, not everyone has containers integrated into their daily development workflow, and some prefer “containerless” solutions but know that containers can be beneficial. Well today’s video is just for you, showing how you can still get your apps onto Cloud Run, even If you don’t have much experience with Docker, containers, nor Dockerfiles.

            App Engine isn’t going away as Google has expressed long-term support for legacy runtimes on the platform, so those who prefer source-based deployments can stay where they are so this is an optional migration. Moving to Cloud Run is for those who want to explicitly move to containerization.

            Migrating to Cloud Run with Cloud Buildpacks video

            So how can apps be containerized without Docker? The answer is buildpacks, an open-source technology that makes it fast and easy for you to create secure, production-ready container images from source code, without a Dockerfile. Google Cloud Buildpacks adheres to the buildpacks open specification and allows users to create images that run on all GCP container platforms: Cloud Run (fully-managed), Anthos, and Google Kubernetes Engine (GKE). If you want to containerize your apps while staying focused on building your solutions and not how to create or maintain Dockerfiles, Cloud Buildpacks is for you.

            In the last video, we showed developers how to containerize a Python 2 Cloud NDB app as well as a Python 3 Cloud Datastore app. We targeted those specific implementations because Python 2 users are more likely to be using App Engine’s ndb or Cloud NDB to connect with their app’s Datastore while Python 3 developers are most likely using Cloud Datastore. Cloud Buildpacks do not support Python 2, so today we’re targeting a slightly different audience: Python 2 developers who have migrated from App Engine ndb to Cloud NDB and who have ported their apps to modern Python 3 but now want to containerize them for Cloud Run.

            Developers familiar with App Engine know that a default HTTP server is provided by default and started automatically, however if special launch instructions are needed, users can add an entrypoint directive in their app.yaml files, as illustrated below. When those App Engine apps are containerized for Cloud Run, developers must bundle their own server and provide startup instructions, the purpose of the ENTRYPOINT directive in the Dockerfile, also shown below.

            Starting your web server with App Engine (app.yaml) and Cloud Run with Docker (Dockerfile) or Buildpacks (Procfile)

            Starting your web server with App Engine (app.yaml) and Cloud Run with Docker (Dockerfile) or Buildpacks (Procfile)

            In this migration, there is no Dockerfile. While Cloud Buildpacks does the heavy-lifting, determining how to package your app into a container, it still needs to be told how to start your service. This is exactly what a Procfile is for, represented by the last file in the image above. As specified, your web server will be launched in the same way as in app.yaml and the Dockerfile above; these config files are deliberately juxtaposed to expose their similarities.

            Other than this swapping of configuration files and the expected lack of a .dockerignore file, the Python 3 Cloud NDB app containerized for Cloud Run is nearly identical to the Python 3 Cloud NDB App Engine app we started with. Cloud Run’s build-and-deploy command (gcloud run deploy) will use a Dockerfile if present but otherwise selects Cloud Buildpacks to build and deploy the container image. The user experience is the same, only without the time and challenges required to maintain and debug a Dockerfile.

            Get started now

            If you’re considering containerizing your App Engine apps without having to know much about containers or Docker, we recommend you try this migration on a sample app like ours before considering it for yours. A corresponding codelab leading you step-by-step through this exercise is provided in addition to the video which you can use for guidance.

            All migration modules, their videos (when available), codelab tutorials, and source code, can be found in the migration repo. While our content initially focuses on Python users, we hope to one day also cover other legacy runtimes so stay tuned. Containerization may seem foreboding, but the goal is for Cloud Buildpacks and migration resources like this to aid you in your quest to modernize your serverless apps!

            Containerizing Google App Engine apps for Cloud Run

            Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

            Google App Engine header

            An optional migration

            Serverless Migration Station is a video mini-series from Serverless Expeditions focused on helping developers modernize their applications running on a serverless compute platform from Google Cloud. Previous episodes demonstrated how to migrate away from the older, legacy App Engine (standard environment) services to newer Google Cloud standalone equivalents like Cloud Datastore. Today’s product crossover episode differs slightly from that by migrating away from App Engine altogether, containerizing those apps for Cloud Run.

            There’s little question the industry has been moving towards containerization as an application deployment mechanism over the past decade. However, Docker and use of containers weren’t available to early App Engine developers until its flexible environment became available years later. Fast forward to today where developers have many more options to choose from, from an increasingly open Google Cloud. Google has expressed long-term support for App Engine, and users do not need to containerize their apps, so this is an optional migration. It is primarily for those who have decided to add containerization to their application deployment strategy and want to explicitly migrate to Cloud Run.

            If you’re thinking about app containerization, the video covers some of the key reasons why you would consider it: you’re not subject to traditional serverless restrictions like development language or use of binaries (flexibility); if your code, dependencies, and container build & deploy steps haven’t changed, you can recreate the same image with confidence (reproducibility); your application can be deployed elsewhere or be rolled back to a previous working image if necessary (reusable); and you have plenty more options on where to host your app (portability).

            Migration and containerization

            Legacy App Engine services are available through a set of proprietary, bundled APIs. As you can surmise, those services are not available on Cloud Run. So if you want to containerize your app for Cloud Run, it must be “ready to go,” meaning it has migrated to either Google Cloud standalone equivalents or other third-party alternatives. For example, in a recent episode, we demonstrated how to migrate from App Engine ndb to Cloud NDB for Datastore access.

            While we’ve recently begun to produce videos for such migrations, developers can already access code samples and codelab tutorials leading them through a variety of migrations. In today’s video, we have both Python 2 and 3 sample apps that have divested from legacy services, thus ready to containerize for Cloud Run. Python 2 App Engine apps accessing Datastore are most likely to be using Cloud NDB whereas it would be Cloud Datastore for Python 3 users, so this is the starting point for this migration.

            Because we’re “only” switching execution platforms, there are no changes at all to the application code itself. This entire migration is completely based on changing the apps’ configurations from App Engine to Cloud Run. In particular, App Engine artifacts such as app.yaml, appengine_config.py, and the lib folder are not used in Cloud Run and will be removed. A Dockerfile will be implemented to build your container. Apps with more complex configurations in their app.yaml files will likely need an equivalent service.yaml file for Cloud Run — if so, you’ll find this app.yaml to service.yaml conversion tool handy. Following best practices means there’ll also be a .dockerignore file.

            App Engine and Cloud Functions are sourced-based where Google Cloud automatically provides a default HTTP server like gunicorn. Cloud Run is a bit more “DIY” because users have to provide a container image, meaning bundling our own server. In this case, we’ll pick gunicorn explicitly, adding it to the top of the existing requirements.txt required packages file(s), as you can see in the screenshot below. Also illustrated is the Dockerfile where gunicorn is started to serve your app as the final step. The only differences for the Python 2 equivalent Dockerfile are: a) require the Cloud NDB package (google-cloud-ndb) instead of Cloud Datastore, and b) start with a Python 2 base image.

            Image of The Python 3 requirements.txt and Dockerfile

            The Python 3 requirements.txt and Dockerfile

            Next steps

            To walk developers through migrations, we always “START” with a working app then make the necessary updates that culminate in a working “FINISH” app. For this migration, the Python 2 sample app STARTs with the Module 2a code and FINISHes with the Module 4a code. Similarly, the Python 3 app STARTs with the Module 3b code and FINISHes with the Module 4b code. This way, if something goes wrong during your migration, you can always rollback to START, or compare your solution with our FINISH. If you are considering this migration for your own applications, we recommend you try it on a sample app like ours before considering it for yours. A corresponding codelab leading you step-by-step through this exercise is provided in addition to the video which you can use for guidance.

            All migration modules, their videos (when published), codelab tutorials, START and FINISH code, etc., can be found in the migration repo. We hope to also one day cover other legacy runtimes like Java 8 so stay tuned. We’ll continue with our journey from App Engine to Cloud Run ahead in Module 5 but will do so without explicit knowledge of containers, Docker, or Dockerfiles. Modernizing your development workflow to using containers and best practices like crafting a CI/CD pipeline isn’t always straightforward; we hope content like this helps you progress in that direction!

            Automatic Deployment of Hugo Sites on Firebase Hosting and Drafts on Cloud Run

            Posted by James Ward, Developer Advocate

            Recently I completed the migration of my blog from WordPress to Hugo and I wanted to take advantage of it now being a static site by hosting it on a Content Delivery Network (CDN). With Hugo the source content is plain files instead of rows in a database. In the case of my blog those files are in git on GitHub. But when the source files change, the site needs to be regenerated and redeployed to the CDN. Also, sometimes it is nice to have drafts available for review. I setup a continuous delivery pipeline which deploys changes to my prod site on Firebase Hosting and drafts on Cloud Run, using Cloud Build. Read on for instructions for how to set all this up.

            Step 1a) Setup A New Hugo Project

            If you do not have an existing Hugo project you can create a GitHub copy (i.e. fork) of my Hugo starter repo:

            Step 1b) Setup Existing Hugo Project

            If you have an existing Hugo project you’ll need to add some files to it:

            .firebaserc

            {
              "projects": {
                "production": "hello-hugo"
              }
            }

            cloudbuild-draft.yaml

            steps:
            - name: 'gcr.io/cloud-builders/git'
              entrypoint: '/bin/sh'
              args:
              - '-c'
              - |
                # Get the theme git submodule
                THEME_URL=$(git config -f .gitmodules --get-regexp '^submodule..*.url$' | awk '{ print $2 }')
                THEME_DIR=$(git config -f .gitmodules --get-regexp '^submodule..*.path$' | awk '{ print $2 }')
                rm -rf themes
                git clone $$THEME_URL $$THEME_DIR

            - name: 'gcr.io/cloud-builders/docker'
              entrypoint: '/bin/sh'
              args:
              - '-c'
              - |
                docker build -t gcr.io/$PROJECT_ID/$REPO_NAME-$BRANCH_NAME:$COMMIT_SHA -f - . << EOF
                FROM klakegg/hugo:latest
                WORKDIR /workspace
                COPY . /workspace
                ENTRYPOINT hugo -D -p $$PORT --bind $$HUGO_BIND --renderToDisk --disableLiveReload --watch=false serve
                EOF
                docker push gcr.io/$PROJECT_ID/$REPO_NAME-$BRANCH_NAME:$COMMIT_SHA

            - name: 'gcr.io/cloud-builders/gcloud'
              args:
              - run
              - deploy
              - --image=gcr.io/$PROJECT_ID/$REPO_NAME-$BRANCH_NAME:$COMMIT_SHA
              - --platform=managed
              - --project=$PROJECT_ID
              - --region=us-central1
              - --memory=512Mi
              - --allow-unauthenticated
              - $REPO_NAME-$BRANCH_NAME

            cloudbuild.yaml

            steps:
            - name: 'gcr.io/cloud-builders/git'
              entrypoint: '/bin/sh'
              args:
              - '-c'
              - |
                # Get the theme git submodule
                THEME_URL=$(git config -f .gitmodules --get-regexp '^submodule..*.url$' | awk '{ print $2 }')
                THEME_DIR=$(git config -f .gitmodules --get-regexp '^submodule..*.path$' | awk '{ print $2 }')
                rm -rf themes
                git clone $$THEME_URL $$THEME_DIR

            - name: 'gcr.io/cloud-builders/curl'
              entrypoint: '/bin/sh'
              args:
              - '-c'
              - |
                curl -sL https://github.com/gohugoio/hugo/releases/download/v0.69.2/hugo_0.69.2_Linux-64bit.tar.gz | tar -zxv
                ./hugo

            - name: 'gcr.io/cloud-builders/wget'
              entrypoint: '/bin/sh'
              args:
              - '-c'
              - |
                # Get firebase CLI
                wget -O firebase https://firebase.tools/bin/linux/latest
                chmod +x firebase
                # Deploy site
                ./firebase deploy --project=$PROJECT_ID --only=hosting

            firebase.json

            {
              "hosting": {
                "public": "public"
              }
            }

            Step 2) Setup Cloud Build Triggers

            In the Google Cloud Build console, connect to your newly forked repo: Select the newly forked repo: Create the default push trigger: Edit the new trigger: Set the trigger to only fire on changes to the ^master$ branch: Create a new trigger: Give it a name like drafts-trigger, specify the branch selector as .* (i.e. any branch), and the build configuration file type to “Cloud Build configuration file” with a value of cloudbuild-draft.yaml Setup permissions for the Cloud Build process to manage Cloud Run and Firebase Hosting by visiting the IAM management page, locate the member with the name ending with @cloudbuild.gserviceaccount.com, and select the “pencil” / edit button: Add a role for “Cloud Run Admin” and another for “Firebase Hosting Admin”: Your default “prod” trigger isn’t read to test yet, but you can test the drafts on Cloud Run by going back to the Cloud Build Triggers page, and clicking the “Run Trigger” button on the “drafts-trigger” line. Check the build logs by finding the build in the Cloud Build History. Once the build completes visit the Cloud Run console to find your newly created service which hosts the drafts version of your new blog. Note that the service name includes the branch so that you can see drafts from different branches.

            Step 3) Setup Firebase Hosting

            To setup your production / CDN’d site, login to the firebase console and select your project:

            Now you’ll need your project id, which can be found in the URL on the Firebase Project Overview page. The URL for my project is:

            console.firebase.google.com/project/jw-demo/overview

            Which means my project id is: jw-demo

            Now copy your project id go into your GitHub fork, select the .firebaserc file and click the “pencil” / edit button:

            Replace the hello-hugo string with your project id and commit the changes. This commit will trigger two new builds, one for the production site and one for the drafts site on Cloud Run. You can check the status of those builds on the Cloud Build History page. Once the default trigger (the one for Firebase hosting) finishes, check out your Hugo site running on Firebase Hosting by navigating to (replacing YOUR_PROJECT_ID with the project id you used above): https://YOUR_PROJECT_ID.web.app/

            Your prod and drafts sites are now automatically deploying on new commits!

            Step 4) (Optional) Change Hugo Theme

            There are many themes for Hugo and they are easy to change. Typically themes are pulled into Hugo sites using git submodules. To change the theme, edit your .gitmodules file and set the subdirectories and url. As an example, here is the content when using the mainroad theme:

            [submodule "themes/mainroad"]
             path = themes/mainroad
             url = https://github.com/vimux/mainroad.git

            You will also need to change the theme value in your config.toml file to match the directory name in the themes directory. For example:

            theme = "mainroad"

            Note: At the time of writing this, Cloud Build does not clone git submodules so the cloudbuild.yaml does the cloning instead.

            Step 5) (Optional) Setup Local Editing

            To setup local editing you will first need to clone your fork. You can do this with the GitHub desktop app. Or from the command line:

            git clone --recurse-submodules https://github.com/USER/REPO.git

            Once you have the files locally, install Hugo, and from inside the repo’s directory, run:

            hugo -D serve

            This will serve the drafts in the site. You can check out the site at: localhost:1313

            Committing non-draft changes to master and pushing those changes to GitHub will kick off the build which will deploy them on your prod site. Committing draft to any branch will kick off the build which will deploy them on a Cloud Run site.

            Hopefully that all helps you with hosting your Hugo sites! Let me know if you run into any problems.