Machine Learning Communities: Q4 ‘22 highlights and achievements

Posted by Nari Yoon, Hee Jung, DevRel Community Manager / Soonson Kwon, DevRel Program Manager

Let’s explore highlights and accomplishments of vast Google Machine Learning communities over the last quarter of 2022. We are enthusiastic and grateful about all the activities by the global network of ML communities. Here are the highlights!

ML at DevFest 2022

A group of ML Developers attending DevFest 2022
A large number of members of ML GDE, TFUG, and 3P ML communities participated in DevFests 2022 worldwide covering various ML topics with Google products. Machine Learning with Jax: Zero to Hero (DevFest Conakry) by ML GDE Yannick Serge Obam Akou (Cameroon) and Easy ML on Google Cloud (DevFest Med) by ML GDE Nathaly Alarcon Torrico (Bolivia) hosted great sessions.

ML Community Summit 2022

A group of ML Developers attending ML Community Summit
ML Community Summit 2022 was hosted on Oct 22-23, 2022, in Bangkok, Thailand. Twenty-five most active community members (ML GDE or TFUG organizer) were invited and shared their past activities and thoughts on Google’s ML products. A video sketch from ML Developer Programs team and a blog posting by ML GDE Margaret Maynard-Reid (United States) help us revisit the moments.

TensorFlow

MAXIM in TensorFlow by ML GDE Sayak Paul (India) shows his implementation of the MAXIM family of models in TensorFlow.

Diagram of gMLP block

gMLP: What it is and how to use it in practice with Tensorflow and Keras? by ML GDE Radostin Cholakov (Bulgaria) demonstrates the state-of-the-art results on NLP and computer vision tasks using a lot less trainable parameters than corresponding Transformer models. He also wrote Differentiable discrete sampling in TensorFlow.

Building Computer Vision Model using TensorFlow: Part 2 by TFUG Pune for the developers who want to deep dive into training an object detection model on Google Colab, inspecting the TF Lite model, and deploying the model on an Android application. ML GDE Nitin Tiwari (India) covered detailed aspects for end-to-end training and deployment of object model detection.

Advent of Code 2022 in pure TensorFlow (days 1-5) by ML GDE Paolo Galeone (Italy) solving the Advent of Code (AoC) puzzles using only TensorFlow. The articles contain a description of the solutions of the Advent of Code puzzles 1-5, in pure TensorFlow.

tf.keras.metrics / tf.keras.optimizers by TFUG Taipei helped people learn the TF libraries. They shared basic concepts and how to use them using Colab.
Screen shot of TensorFlow Lite on Android Project Practical Course
A hands-on course on TensorFlow Lite projects on Android by ML GDE Xiaoxing Wang (China) is the book mainly introducing the application of TensorFlow Lite in Android development. The content focuses on applying three typical ML applications in Android development.

Build tensorflow-lite-select-tf-ops.aar and tensorflow-lite.aar files with Colab by ML GDE George Soloupis (Greece) guides how you can shrink the final size of your Android application’s .apk by building tensorflow-lite-select-tf-ops.aar and tensorflow-lite.aar files without the need of Docker or personal PC environment.

TensorFlow Lite and MediaPipe Application by ML GDE XuHua Hu (China) explains how to use TFLite to deploy an ML model into an application on devices. He shared experiences with developing a motion sensing game with MediaPipe, and how to solve problems that we may meet usually.

Train and Deploy TensorFlow models in Go by ML GDE Paolo Galeone (Italy) delivered the basics of the TensorFlow Go bindings, the limitations, and how the tfgo library simplifies their usage.

Keras

Diagram of feature maps concatenated together and flattened
Complete Guide on Deep Learning Architectures, Chapter 1 on ConvNets by ML GDE Merve Noyan (France) brings you into the theory of ConvNets and shows how it works with Keras.
Hazy Image Restoration Using Keras by ML GDE Soumik Rakshit (India) provides an introduction to building an image restoration model using TensorFlow, Keras, and Weights & Biases. He also shared an article Improving Generative Images with Instructions: Prompt-to-Prompt Image Editing with Cross Attention Control.
Mixed precision in Keras based Stable Diffusion
Let’s Generate Images with Keras based Stable Diffusion by ML GDE Chansung Park (Korea) delivered how to generate images with given text and what stable diffusion is. He also talked about Keras-based stable diffusion, basic building blocks, and the advantages of using Keras-based stable diffusion.
A Deep Dive into Transformers with TensorFlow and Keras: Part 1, Part 2, Part3 by ML GDE Aritra Roy Gosthipaty (India) covered the journey from the intuition of attention to formulating the multi-head self-attention. And TensorFlow port of GroupViT in 🤗 transformers library was his contribution to Hugging Face transformers library.

TFX

Digits + TFX banner
How startups can benefit from TFX by ML GDE Hannes Hapke (United States) explains how the San Francisco-based FinTech startup Digits has benefitted from applying TFX early, how TFX helps Digits grow, and how other startups can benefit from TFX too.

Usha Rengaraju (India) shared TensorFlow Extended (TFX) Tutorials (Part 1, Part 2, Part 3) and the following TF projects: TensorFlow Decision Forests Tutorial and FT Transformer TensorFlow Implementation.

Hyperparameter Tuning and ML Pipeline by ML GDE Chansung Park (Korea) explained hyperparam tuning, why it is important; Introduction to KerasTuner, basic usage; how to visualize hyperparam tuning results with TensorBoard; and integration within ML pipeline with TFX.

JAX/Flax

JAX High-performance ML Research by TFUG Taipei and ML GDE Jerry Wu (Taiwan) introduced JAX and how to start using JAX to solve machine learning problems.

[TensorFlow + TPU] GatedTabTransformer[W&B] and its JAX/Flax counterpart GatedTabTransformer-FLAX[W&B] by Usha Rengaraju (India) are tutorial series containing the implementation of GatedTabTransformer paper in both TensorFlow (TPU) and FLAX.
Putting NeRF on a diet: Semantically consistent Few-Shot View Synthesis Implementation
JAX implementation of Diet NeRf by ML GDE Wan Hong Lau (Singapore) implemented the paper “Putting NeRF on a Diet (DietNeRF)” in JAX/Flax. And he also implemented a JAX-and-Flax training pipeline with the ResNet model in his Kaggle notebook, 🐳HappyWhale🔥Flax/JAX⚡TPU&GPU – ResNet Baseline.
Introduction to JAX with Flax (slides) by ML GDE Phillip Lippe (Netherlands) reviewed from the basics of the requirements we have on a DL framework to what JAX has to offer. Further, he focused on the powerful function-oriented view JAX offers and how Flax allows you to use them in training neural networks.
Screen grab of ML GDE David Cardozo and Cristian Garcia during a live coding session of a review of new features, specifically Shared Arrays, in the recent release of JAX
JAX Streams: Exploring JAX 0.4 by ML GDE David Cardozo (Canada) and Cristian Garcia (Colombia) showed a review of new features (specifically Shared Arrays) in the recent release of JAX and demonstrated live coding.
[LiveCoding] Train ResNet/MNIST with JAX/Flax by ML GDE Qinghua Duan (China) demonstrated how to train ResNet using JAX by writing code online.

Kaggle

Low-light Image Enhancement using MirNetv2 by ML GDE Soumik Rakshit (India) demonstrated the task of Low-light Image Enhancement.

Heart disease Prediction and Diabetes Prediction Competition hosted by TFUG Chandigarh were to familiarize participants with ML problems and find solutions using classification techniques.
TensorFlow User Group Bangalore Sentiment Analysis Kaggle Competition 1
TFUG Bangalore Kaggle Competition – Sentiment Analysis hosted by TFUG Bangalore was to find the best sentiment analysis algorithm. Participants were given a set of training data and asked to submit an ML/DL algorithm that could predict the sentiment of a text. The group also hosted Kaggle Challenge Finale + Vertex AI Session to support the participants and guide them in learning how to use Vertex AI in a workflow.

Cloud AI

Better Hardware Provisioning for ML Experiments on GCP by ML GDE Sayak Paul (India) discussed the pain points of provisioning hardware (especially for ML experiments) and how we can get better provision hardware with code using Vertex AI Workbench instances and Terraform.

Jayesh Sharma, Platform Engineer, Zen ML; MLOps workshop with TensorFlow and Vertex AI November 12, 2022|TensorFlow User Group Chennai
MLOps workshop with TensorFlow and Vertex AI by TFUG Chennai targeted beginners and intermediate-level practitioners to give hands-on experience on the E2E MLOps pipeline with GCP. In the workshop, they shared the various stages of an ML pipeline, the top tools to build a solution, and how to design a workflow using an open-source framework like ZenML.
10 Predictions on the Future of Cloud Computing by 2025: Insights from Google Next Conference by ML GDE Victor Dibia (United States) includes a recap of his notes reflecting on the top 10 cloud technology predictions discussed at the Google Cloud Next 2022 keynote.
Workflow of Google Virtual Career Center
O uso do Vertex AI Matching Engine no Virtual Career Center (VCC) do Google Cloud by ML GDE Rubens Zimbres (Brazil) approaches the use of Vertex AI Matching Engine as part of the Google Cloud Virtual Career Center solution.

More practical time-series model with BQML by ML GDE JeongMin Kwon (Korea) introduced BQML and time-series modeling and showed some practical applications with BQML ARIMA+ and Python implementations.

Vertex AI Forecast – Demand Forecasting with AutoML by ML GDE Rio Kurihara (Japan) presented a time series forecast overview, time series fusion transformers, and the benefits and desired features of AutoML.

Research & Ecosystem

AI in Healthcare by ML GDE Sara EL-ATEIF (Morocco) introduced AI applications in healthcare and the challenges facing AI in its adoption into the health system.

Women in AI APAC finished their journey at ML Paper Reading Club. During 10 weeks, participants gained knowledge on outstanding machine learning research, learned the latest techniques, and understood the notion of “ML research” among ML engineers. See their session here.

A Natural Language Understanding Model LaMDA for Dialogue Applications by ML GDE Jerry Wu (Taiwan) introduced the natural language understanding (NLU) concept and shared the operation mode of LaMDA, model fine-tuning, and measurement indicators.

Python library for Arabic NLP preprocessing (Ruqia) by ML GDE Ruqiya Bin (Saudi Arabia) is her first python library to serve Arabic NLP.

Screengrab of ML GDEs Margaret Maynard-Reid and Akash Nain during Chat with ML GDE Akash
Chat with ML GDE Vikram & Chat with ML GDE Aakash by ML GDE Margaret Maynard-Reid (United States) shared the stories of ML GDEs’ including how they became ML GDE and how they proceeded with their ML projects.

Anatomy of Capstone ML Projects 🫀by ML GDE Sayak Paul (India) discussed working on capstone ML projects that will stay with you throughout your career. He covered various topics ranging from problem selection to tightening up the technical gotchas to presentation. And in Improving as an ML Practitioner he shared his learning from experience in the field working on several aspects.

Screen grab of statement of objectives in MLOps Development Environment by ML GDE Vinicius Carida
MLOps Development Environment by ML GDE Vinicius Caridá (Brazil) aims to build a full development environment where you can write your own pipelines connecting MLFLow, Airflow, GCP and Streamlit, and build amazing MLOps pipelines to practice your skills.

Transcending Scaling Laws with 0.1% Extra Compute by ML GDE Grigory Sapunov (UK) reviewed a recent Google article on UL2R. And his posting Discovering faster matrix multiplication algorithms with reinforcement learning explained how AlphaTensor works and why it is important.

Back in Person – Prompting, Instructions and the Future of Large Language Models by TFUG Singapore and ML GDE Sam Witteveen (Singapore) and Martin Andrews (Singapore). This event covered recent advances in the field of large language models (LLMs).

ML for Production: The art of MLOps in TensorFlow Ecosystem with GDG Casablanca by TFUG Agadir discussed the motivation behind using MLOps and how it can help organizations automate a lot of pain points in the ML production process. It also covered the tools used in the TensorFlow ecosystem.

Machine Learning Communities: Q3 ‘22 highlights and achievements

Posted by Nari Yoon, Hee Jung, DevRel Community Manager / Soonson Kwon, DevRel Program Manager

Let’s explore highlights and accomplishments of vast Google Machine Learning communities over the third quarter of the year! We are enthusiastic and grateful about all the activities by the global network of ML communities. Here are the highlights!

TensorFlow/Keras

Load-testing TensorFlow Serving’s REST Interface

Load-testing TensorFlow Serving’s REST Interface by ML GDE Sayak Paul (India) and Chansung Park (Korea) shares the lessons and findings they learned from conducting load tests for an image classification model across numerous deployment configurations.

TFUG Taipei hosted events (Python + Hugging Face-Translation+ tf.keras.losses, Python + Object detection, Python+Hugging Face-Token Classification+tf.keras.initializers) in September and helped community members learn how to use TF and Hugging face to implement machine learning model to solve problems.

Neural Machine Translation with Bahdanau’s Attention Using TensorFlow and Keras and the related video by ML GDE Aritra Roy Gosthipaty (India) explains the mathematical intuition behind neural machine translation.

Serving a TensorFlow image classification model as RESTful and gRPC based services with TFServing, Docker, and Kubernetes

Automated Deployment of TensorFlow Models with TensorFlow Serving and GitHub Actions by ML GDE Chansung Park (Korea) and Sayak Paul (India) explains how to automate TensorFlow model serving on Kubernetes with TensorFlow Serving and GitHub Action.

Deploying 🤗 ViT on Kubernetes with TF Serving by ML GDE Sayak Paul (India) and Chansung Park (Korea) shows how to scale the deployment of a ViT model from 🤗 Transformers using Docker and Kubernetes.

Screenshot of the TensorFlow Forum in the Chinese Language run by the tf.wiki team

Long-term TensorFlow Guidance on tf.wiki Forum by ML GDE Xihan Li (China) provides TensorFlow guidance by answering the questions from Chinese developers on the forum.

photo of a phone with the Hindi letter 'Ohm' drawn on the top half of the screen. Hinidi Character recognition shows the letter Ohm as the Predicted Result below.

Hindi Character Recognition on Android using TensorFlow Lite by ML GDE Nitin Tiwari (India) shares an end-to-end tutorial on training a custom computer vision model to recognize Hindi characters. In TFUG Pune event, he also gave a presentation titled Building Computer Vision Model using TensorFlow: Part 1.

Using TFlite Model Maker to Complete a Custom Audio Classification App by ML GDE Xiaoxing Wang (China) shows how to use TFLite Model Maker to build a custom audio classification model based on YAMNet and how to import and use the YAMNet-based custom models in Android projects.

SoTA semantic segmentation in TF with 🤗 by ML GDE Sayak Paul (India) and Chansung Park (Korea). The SegFormer model was not available on TensorFlow.

Text Augmentation in Keras NLP by ML GDE Xiaoquan Kong (China) explains what text augmentation is and how the text augmentation feature in Keras NLP is designed.

The largest vision model checkpoint (public) in TF (10 Billion params) through 🤗 transformers by ML GDE Sayak Paul (India) and Aritra Roy Gosthipaty (India). The underlying model is RegNet, known for its ability to scale.

A simple TensorFlow implementation of a DCGAN to generate CryptoPunks

CryptoGANs open-source repository by ML GDE Dimitre Oliveira (Brazil) shows simple model implementations following TensorFlow best practices that can be extended to more complex use-cases. It connects the usage of TensorFlow with other relevant frameworks, like HuggingFace, Gradio, and Streamlit, building an end-to-end solution.

TFX

TFX Machine Learning Pipeline from data injection in TFRecord to pushing out Vertex AI

MLOps for Vision Models from 🤗 with TFX by ML GDE Chansung Park (Korea) and Sayak Paul (India) shows how to build a machine learning pipeline for a vision model (TensorFlow) from 🤗 Transformers using the TF ecosystem.

First release of TFX Addons Package by ML GDE Hannes Hapke (United States). The package has been downloaded a few thousand times (source). Google and other developers maintain it through bi-weekly meetings. Google’s Open Source Peer Award has recognized the work.

TFUG São Paulo hosted TFX T1 | E4 & TFX T1 | E5. And ML GDE Vinicius Caridá (Brazil) shared how to train a model in a TFX pipeline. The fifth episode talks about Pusher: publishing your models with TFX.

Semantic Segmentation model within ML pipeline by ML GDE Chansung Park (Korea) and Sayak Paul (India) shows how to build a machine learning pipeline for semantic segmentation task with TFX and various GCP products such as Vertex Pipeline, Training, and Endpoints.

JAX/Flax

Screen shot of Tutorial 2 (JAX): Introduction to JAX+Flax with GitHub Repo and Codelab via university of Amseterdam

JAX Tutorial by ML GDE Phillip Lippe (Netherlands) is meant to briefly introduce JAX, including writing and training neural networks with Flax.

TFUG Malaysia hosted Introduction to JAX for Machine Learning (video) and Leong Lai Fong gave a talk. The attendees learned what JAX is and its fundamental yet unique features, which make it efficient to use when executing deep learning workloads. After that, they started training their first JAX-powered deep learning model.

TFUG Taipei hosted Python+ JAX + Image classification and helped people learn JAX and how to use it in Colab. They shared knowledge about the difference between JAX and Numpy, the advantages of JAX, and how to use it in Colab.

Introduction to JAX by ML GDE João Araújo (Brazil) shared the basics of JAX in Deep Learning Indaba 2022.

A comparison of the performance and overview of issues resulting from changing from NumPy to JAX

Should I change from NumPy to JAX? by ML GDE Gad Benram (Portugal) compares the performance and overview of the issues that may result from changing from NumPy to JAX.

Introduction to JAX: efficient and reproducible ML framework by ML GDE Seunghyun Lee (Korea) introduced JAX/Flax and their key features using practical examples. He explained the pure function and PRNG, which make JAX explicit and reproducible, and XLA and mapping functions which make JAX fast and easily parallelized.

Data2Vec Style pre-training in JAX by ML GDE Vasudev Gupta (India) shares a tutorial for demonstrating how to pre-train Data2Vec using the Jax/Flax version of HuggingFace Transformers.

Distributed Machine Learning with JAX by ML GDE David Cardozo (Canada) delivered what makes JAX different from TensorFlow.

Image classification with JAX & Flax by ML GDE Derrick Mwiti (Kenya) explains how to build convolutional neural networks with JAX/Flax. And he wrote several articles about JAX/Flax: What is JAX?, How to load datasets in JAX with TensorFlow, Optimizers in JAX and Flax, Flax vs. TensorFlow, etc..

Kaggle

DDPMs – Part 1 by ML GDE Aakash Nain (India) and cait-tf by ML GDE Sayak Paul (India) were announced as Kaggle ML Research Spotlight Winners.

Forward process in DDPMs from Timestep 0 to 100

Fresher on Random Variables, All you need to know about Gaussian distribution, and A deep dive into DDPMs by ML GDE Aakash Nain (India) explain the fundamentals of diffusion models.

In Grandmasters Journey on Kaggle + The Kaggle Book, ML GDE Luca Massaron (Italy) explained how Kaggle helps people in the data science industry and which skills you must focus on apart from the core technical skills.

Cloud AI

How Cohere is accelerating language model training with Google Cloud TPUs by ML GDE Joanna Yoo (Canada) explains what Cohere engineers have done to solve scaling challenges in large language models (LLMs).

ML GDE Hannes Hapke (United States) chats with Fillipo Mandella, Customer Engineering Manager at Google

In Using machine learning to transform finance with Google Cloud and Digits, ML GDE Hannes Hapke (United States) chats with Fillipo Mandella, Customer Engineering Manager at Google, about how Digits leverages Google Cloud’s machine learning tools to empower accountants and business owners with near-zero latency.

A tour of Vertex AI by TFUG Chennai for ML, cloud, and DevOps engineers who are working in MLOps. This session was about the introduction of Vertex AI, handling datasets and models in Vertex AI, deployment & prediction, and MLOps.

TFUG Abidjan hosted two events with GDG Cloud Abidjan for students and professional developers who want to prepare for a Google Cloud certification: Introduction session to certifications and Q&A, Certification Study Group.

Flow chart showing shows how to deploy a ViT B/16 model on Vertex AI

Deploying 🤗 ViT on Vertex AI by ML GDE Sayak Paul (India) and Chansung Park (Korea) shows how to deploy a ViT B/16 model on Vertex AI. They cover some critical aspects of a deployment such as auto-scaling, authentication, endpoint consumption, and load-testing.

Photo collage of AI generated images

TFUG Singapore hosted The World of Diffusion – DALL-E 2, IMAGEN & Stable Diffusion. ML GDE Martin Andrews (Singapore) and Sam Witteveen (Singapore) gave talks named “How Diffusion Works” and “Investigating Prompt Engineering on Diffusion Models” to bring people up-to-date with what has been going on in the world of image generation.

ML GDE Martin Andrews (Singapore) have done three projects: GCP VM with Nvidia set-up and Convenience Scripts, Containers within a GCP host server, with Nvidia pass-through, Installing MineRL using Containers – with linked code.

Jupyter Services on Google Cloud by ML GDE Gad Benram (Portugal) explains the differences between Vertex AI Workbench, Colab, and Deep Learning VMs.

Google Cloud's Two Towers Recommender and TensorFlow

Train and Deploy Google Cloud’s Two Towers Recommender by ML GDE Rubens de Almeida Zimbres (Brazil) explains how to implement the model and deploy it in Vertex AI.

Research & Ecosystem

WOMEN DATA SCIENCE, LA PAZ Club de lectura de papers de Machine Learning Read, Learn and Share the knowledge #MLPaperReadingClubs, Nathaly Alarcón, @WIDS_LaPaz #MLPaperReadingClubs

The first session of #MLPaperReadingClubs (video) by ML GDE Nathaly Alarcon Torrico (Bolivia) and Women in Data Science La Paz. Nathaly led the session, and the community members participated in reading the ML paper “Zero-shot learning through cross-modal transfer.”

In #MLPaperReadingClubs (video) by TFUG Lesotho, Arnold Raphael volunteered to lead the first session “Zero-shot learning through cross-modal transfer.”

Screenshot of a screenshare of Zero-shot learning through cross-modal transfer to 7 participants in a virtual call

ML Paper Reading Clubs #1: Zero Shot Learning Paper (video) by TFUG Agadir introduced a model that can recognize objects in images even if no training data is available for the objects. TFUG Agadir prepared this event to make people interested in machine learning research and provide them with a broader vision of differentiating good contributions from great ones.

Opening of the Machine Learning Paper Reading Club (video) by TFUG Dhaka introduced ML Paper Reading Club and the group’s plan.

EDA on SpaceX Falcon 9 launches dataset (Kaggle) (video) by TFUG Mysuru & TFUG Chandigarh organizer Aashi Dutt (presenter) walked through exploratory data analysis on SpaceX Falcon 9 launches dataset from Kaggle.

Screenshot of ML GDE Qinghua Duan (China) showing how to apply the MRC paradigm and BERT to solve the dialogue summarization problem.

Introduction to MRC-style dialogue summaries based on BERT by ML GDE Qinghua Duan (China) shows how to apply the MRC paradigm and BERT to solve the dialogue summarization problem.

Plant disease classification using Deep learning model by ML GDE Yannick Serge Obam Akou (Cameroon) talked on plant disease classification using deep learning model : an end to end Android app (open source project) that diagnoses plant diseases.

TensorFlow/Keras implementation of Nystromformer

Nystromformer Github repository by Rishit Dagli provides TensorFlow/Keras implementation of Nystromformer, a transformer variant that uses the Nyström method to approximate standard self-attention with O(n) complexity which allows for better scalability.

Interview with Doug Duhaime, contributor to Google’s Dev Library

Posted by the Google Dev Library Team

Introducing the Dev Library Contributor Spotlights – a blog series highlighting developers that are supporting the thriving development ecosystem by contributing their resources and tools to Google Dev Library.

We met with Doug Duhaime, Full Stack Developer in Yale University’s Digital Humanities Lab, to discuss his passion for Machine Learning, his processes and what inspired him to release his PixPlot project as an Open Source.

What led you to explore the field of machine learning?

I was an English major in undergrad and in graduate school. I have a PhD in English literature. My dissertation was exploring copyright history and the ways that changes in copyright law affected the book market. How does the institution of fixed duration copyright influence the book market? To answer this question, I had to mine an enormous collection of data – half a million books, published before 1800 – to look at different patterns. That was one of the key projects that got me inspired to further explore the world of Machine Learning.

In fact, one of my projects – the PixPlot library – uses computer vision to analyze image collections, which was also partially used in my research. Part of my research looked at plagiarism detection and how readily people are inclined to copy images once it becomes legal to copy them from other texts. Computer vision helps us to answer these questions and identify key patterns.

I’ve seen machine learning and programming as a way to ask new questions in historical contexts. And there’s a whole field of us – we’re called digital humanists. Yale University, where I’ve been for the last five years, has a fantastic digital humanities program where researchers are asking questions like this and using fun machine learning platforms like TensorFlow to answer those questions.

Screenshot from the PixPlot library showing Image Fields in the Meserve-Kunhardt Collection with the following identified hotspots: Boxers, Buildings, Buttons, Chairs, Gowns

Can you tell us more about the evolution of your PixPlot library project?

We started in Yale’s digital humanities lab with a project called neural neighbors. And the idea here was to find patterns in the Meserve-Kunhardt Collection of images.

Meserve-Kunhardt is a collection of photographs largely from the 19th century that Yale recently acquired. After being acquired by the university, some curators were preparing to identify all this really rich metadata to describe these images. However, they had a backlog, and they needed help to try to make sense of what’s in this collection. And so, Neural Neighbors was our initial attempt to answer this question.

As this project went on, we started running up against limitations and asking bigger questions. For example, instead of just looking at the pictures, what would it be like to look at the entire collection all at once? In order to answer this question, we needed a more performant rendering layer.

So we decided to utilize TensorFlow, which allowed us to extract vector representation of each image. We then compressed the dimensionality of those vectors down to 2D. But for PixPlot, we decided to use a different dimensionality reduction technique called umap. And that brought us to the first release of PixPlot.

The idea here was to take the whole collection, shoot it down into 2D, and then let you move through it and look at the images in the collection wherein we expect images with similar content to be placed close by one another.

And so it’s just evolved from that early genesis and Neural Neighbors through to where it is today.

What inspired you to release PixPlot as an open source project?

In the case of PixPlot, I was working for Yale University, and we had a goal to make as much of our contributions to the software world as possible open and publicly accessible without any commercial terms.

It was a huge privilege to spend time with the lab and build software that others found useful. I would say even more generally, in my personal life, I really like building things that people find useful and, when possible, contributing back to the open source world because, I think, so many of us learn from open source.

Google Dev Library Quote: We look at other people's examples and get excited by tools and projects others are building. And many of those are non-commercial. They're just open and free to the world. And it's great to give back when we can. Doug Duhaime Dev Library Contributor

Find out more content contributed and authored by Doug Duhaime and discover more unique tools and resources on the Google Dev Library website!

Dev Library Letters: 14th Issue

Posted by Garima Mehra, Program Manager

‘Google Dev Library letters’ is curated to bring you some of the best projects developed with Google tech that have been submitted to the Dev Library platform. We hope this brings you the inspiration you need for your next project!

Android

Image-compressor 
by Vinod Baste

Check out Vinod’s Android Image compress library that helps reduce the size of the image by 90% without losing any of its pixels.

SealedX 
by Jaewoong Eum

Learn how to auto-generate extensive sealed classes and interfaces for Android and Kotlin.

Flutter

GitHub Actions to deploy
Flutter Web to gh-pages 
by Sai Rajendra Immadi

Tired of manually deploying the app every time? Or do you want to deploy your flutter web applications to gh-pages? Use this blog as your guide.

Double And Triple Dots in Flutter 
by Lakshydeep Vikram

Learn the reason for using double and triple dots in flutter and where to use them.

Machine Learning

Nystromformer 
by Rishit Dagli

Learn how to use the Nystrom method to approximate standard self-attention. 


Google Cloud

by Ezekias Bokove

Learn how to set up a notification system for Cloud Run services. 

Switch to GCP for cost savings and better performance
by Gaurav Madan

Learn why architects dealing with complex application design and who use well-known Google services should consider the Google Cloud Platform. 


“The Google community includes people with diverse backgrounds. No matter what an individual circumstance is, the platform should support anyone to explore and be creative. We encourage authors to boldly consider diverse backgrounds and to be inclusive when authoring.”

Vinesh Prasanna M

Customer Engineer | Google Cloud 



“Authoring a good code sample is hard. The difficulty comes from the additional pieces you need to add to your respository to keep the code sample fresh and appealing to your developers.”

Brett Morgan

Developer Relations Engineer | Flutter





Want to read more? 
Check out the latest projects and community-authored content by visiting Google Dev Library. 
Submit your projects to showcase your work and inspire developers!

Introducing Discovery Ad Performance Analysis

Posted by Manisha Arora, Nithya Mahadevan, and Aritra Biswas, gPS Data Science team

Overview of Discovery Ads and need for Ad Performance Analysis

Discovery ads, launched in May 2019, allow advertisers to easily extend their reach of social ads users across YouTube, Google Feed and Gmail worldwide. They provide brands a new opportunity to reach 3 billion people as they explore their interests and search for inspiration across their favorite Google feeds (YouTube, Gmail, and Discover) — all with a single campaign. Learn more about Discovery ads here.

Due to these uniquenesses, customers need a data driven method to identify textual & imagery elements in Discovery Ad copies that drive Interaction Rate of their Discovery Ad campaigns, where interaction is defined as the main user action associated with an ad format—clicks and swipes for text and Shopping ads, views for video ads, calls for call extensions, and so on.

Interaction Rate = interaction / impressions

“Customers need a data driven method to identify textual & imagery elements in Discovery Ad copies that drive Interaction Rate of their campaigns.”

– Manisha Arora, Data Scientist

Our analysis approach:

The Data Science team at Google is investing in a machine learning approach to uncover insights from complex unstructured data and provide machine learning based recommendations to our customers. Machine Learning helps us study what works in ads at scale and these insights can greatly benefit the advertisers.

We follow a six-step based approach for Discovery Ad Performance Analysis:

  • Understand Business Goals
  • Build Creative Hypothesis
  • Data Extraction
  • Feature Engineering
  • Machine Learning Modeling
  • Analysis & Insight Generation

To begin with, we work closely with the advertisers to understand their business goals, current ad strategy, and future goals. We closely map this to industry insights to draw a larger picture and provide a customized analysis for each advertiser. As a next step, we build hypotheses that best describe the problem we are trying to solve. An example of a hypothesis can be -”Do superlatives (words like “top”, “best”) in the ad copy drive performance?”

“Machine Learning helps us study what works in ads at scale and these insights can greatly benefit the advertisers.”

– Manisha Arora, Data Scientist

Once we have a hypothesis we are working towards, the next step is to deep-dive into the technical analysis.

Data Extraction & Pre-processing

Our initial dataset includes raw ad text, imagery, performance KPIs & target audience details from historic ad campaigns in the industry. Each Discovery ad contains two text assets (Headline and Description) and one image asset. We then apply ML to extract text and image features from these assets.

Text Feature Extraction

We apply NLP to extract the text features from the ad text. We pass the raw text in the ad headline & description through Google Cloud’s Language API which parses the raw text into our feature set: commonly used keywords, sentiments etc.

Example: 

Image Feature Extraction

We apply Image Processing to extract image features from the ad copy imagery. We pass the raw images through Google Cloud’s Vision API & extract image components including objects, person, background, lighting etc.

Following are the holistic set of features that are extracted from the ad content:

Feature Design


Text Feature Design

There are two types of text features being included in DisCat:

1. Generic text feature

a. These are features returned by Google Cloud’s Language API including sentiment, word / character count, tone (imperative vs indicative), symbols, most frequent words and so on.

2. Industry-specific value propositions

a. These are features that only apply to a specific industry (e.g. finance) that are manually curated by the data science developer in collaboration with specialists and other industry experts.

  • For example, for the finance industry, one value proposition can be “Price Offer”. A list of keywords / phrases that are related to price offers (e.g. “discount”, “low rate”, “X% off”) will be curated based on domain knowledge to identify this value proposition in the ad copies. NLP techniques (e.g. wordnet synset) and manual examination will be used to make sure this list is inclusive and accurate.

Image Feature Design

Like the text features, image features can largely be grouped into two categories:

1. Generic image features

a. These features apply to all images and include the color profile, whether any logos were detected, how many human faces are included, etc.

b. The face-related features also include some advanced aspects: we look for prominent smiling faces looking directly at the camera, we differentiate between individuals vs. small groups vs. crowds, etc.

2. Object-based features

a. These features are based on the list of objects and labels detected in all the images in the dataset, which can often be a massive list including generic objects like “Person” and specific ones like particular dog breeds.

b. The biggest challenge here is dimensionality: we have to cluster together related objects into logical themes like natural vs. urban imagery.

c. We currently have a hybrid approach to this problem: we use unsupervised clustering approaches to create an initial clustering, but we manually revise it as we inspect sample images. The process is:

  • Extract object and label names (e.g. Person, Chair, Beach, Table) from the Vision API output and filter out the most uncommon objects
  • Convert these names to 50-dimensional semantic vectors using a Word2Vec model trained on the Google News corpus
  • Using PCA, extract the top 5 principal components from the semantic vectors. This step takes advantage of the fact that each Word2Vec neuron encodes a set of commonly adjacent words, and different sets represent different axes of similarity and should be weighted differently
  • Use an unsupervised clustering algorithm, namely either k-means or DBSCAN, to find semantically similar clusters of words
  • We are also exploring augmenting this approach with a combined distance metric:

d(w1, w2) = a * (semantic distance) + b * (co-appearance distance)

where the latter is a Jaccard distance metric

Each of these components represents a choice the advertiser made when creating the messaging for an ad. Now that we have a variety of ads broken down into components, we can ask: which components are associated with ads that perform well or not so well?

We use a fixed effects1 model to control for unobserved differences in the context in which different ads were served. This is because the features we are measuring are observed multiple times in different contexts i.e. ad copy, audience groups, time of year & device in which ad is served.

The trained model will seek to estimate the impact of individual keywords, phrases & image components in the discovery ad copies. The model form estimates Interaction Rate (denoted as ‘IR’ in the following formulas) as a function of individual ad copy features + controls:

We use ElasticNet to spread the effect of features in presence of multicollinearity & improve the explanatory power of the model:

“Machine Learning model estimates the impact of individual keywords, phrases, and image components in discovery ad copies.”

– Manisha Arora, Data Scientist

 

Outputs & Insights

Outputs from the machine learning model help us determine the significant features. Coefficient of each feature represents the percentage point effect on CTR.

In other words, if the mean CTR without feature is X% and the feature ‘xx’ has a coeff of Y, then the mean CTR with feature ‘xx’ included will be (X + Y)%. This can help us determine the expected CTR if the most important features are included as part of the ad copies.

Key-takeaways (sample insights):

We analyze keywords & imagery tied to the unique value propositions of the product being advertised. There are 6 key value propositions we study in the model. Following are the sample insights we have received from the analyses:

Shortcomings:

Although insights from DisCat are quite accurate and highly actionable, the moel does have a few limitations:

1. The current model does not consider groups of keywords that might be driving ad performance instead of individual keywords (Example – “Buy Now” phrase instead of “Buy” and “Now” individual keywords).

2. Inference and predictions are based on historical data and aren’t necessarily an indication of future success.

3. Insights are based on industry insights and may need to be tailored for a given advertiser.

DisCat breaks down exactly which features are working well for the ad and which ones have scope for improvement. These insights can help us identify high-impact keywords in the ads which can then be used to improve ad quality, thus improving business outcomes. As next steps, we recommend testing out the new ad copies with experiments to provide a more robust analysis. Google Ads A/B testing feature also allows you to create and run experiments to test these insights in your own campaigns.

Summary

Discovery Ads are a great way for advertisers to extend their social outreach to millions of people across the globe. DisCat helps break down discovery ads by analyzing text and images separately and using advanced ML/AI techniques to identify key aspects of the ad that drives greater performance. These insights help advertisers identify room for growth, identify high-impact keywords, and design better creatives that drive business outcomes.

Acknowledgement

Thank you to Shoresh Shafei and Jade Zhang for their contributions. Special mention to Nikhil Madan for facilitating the publishing of this blog.

Notes

  1. Greene, W.H., 2011. Econometric Analysis, 7th ed., Prentice Hall;

    Cameron, A. Colin; Trivedi, Pravin K. (2005). Microeconometrics: Methods and Applications. ↩

Google Dev Library Letters : 13th Issue

Posted by Garima Mehra, Program Manager

Welcome to the 13th Issue: ‘Google Dev Library letters’ is a technology newsletter curated to bring you some of the best projects developed with Google tech and submitted to the Google Dev Library platform. We are back with another boost of inspiration for your next project!

Hero Content of the month

Check out shortlisted content from the Google technologies of your choice.

Android

Contact Store API by Alex Styl

Contact Store is a modern API that makes access to contacts on Android devices simple to use. It solves for the most frequent use cases and makes developing enjoyable.

Custom Progress Indicator by Samson Achiaga

CustomProgressIndicator library is a simple, customizable progress indicator that gives android applications a nice feel. It saves developers time by creating a unique, customizable loading view.

Flutter

Numbers by Bulent Bariskilic

Discover an app designed to show facts about numbers using the http://numbersapi.com API. The project has been written solely in Dart Language.

Cupertino Icons Gallery by Cephas Brian

Get access to over 1,335 icons in one centralized place – the Cupertino Icons Gallery is an open source, cross-platform space to find all the icons used in Flutter.

Machine Learning


Learn how to build a system by considering two MLOps scenarios – if the model needs to be replaced later and if the model itself has to evolve with the data.

Probing Vision Transformers by Sayak Paul & Aritra Roy

Explore tools in this repository to probe into the representations learned by different families of Vision Transformers.

Google Cloud

Combining Google Apps Script with Google AppSheet by Aryan Irani

Learn how to combine Google AppScript with Google AppSheet to make automation even more powerful.

What a beautiful stream!! by Mandar Chaphalkar

Learn how to create a stream in 6 simple steps now that Google Cloud recently made Datastream CDC generally available.

Curators Corner

Meet our curators who have been working behind the scenes to bring you the best content submissions


Android

“Android development changes fast and it’s great to see developers write blogs to help others learn.

It’s a pleasure to be part of the Android community. I enjoy seeing the android community. I enjoy seeing the Android community flourish by collaborating with each other and sharing their learnings” 

 

Andres Sandoval

Sr. Strategist, Google

Machine Learning



“We are loving the TensorFlow.js submissions we have seen so far, and have no doubt future ones will continue to push the boundaries of what’s possible in this space, and because it is web powered anyone anywhere can try the demos typically with the click of a link!”


Jason Mayes 

Web ML Developer Relations Lead, Google

 

Liked what you read? Checkout the latest projects and community-authored content by visiting our home page or subscribing to our newsletter.

Google Dev Library Letters — 12th Issue

Posted by Garima Mehra, Program Manager

‘Google Dev Library Letters’ is curated to bring you some of the latest projects developed with Google tech submitted to Google Dev Library Platform. We hope this brings you the inspiration you need for your next project!


Android

Shape your Image: Circle, Rounded Square, or Cuts at the corner in Android by Sriyank Siddhartha

Using the MDC library, shape images in just a few lines of code by using ShapeableImageView.

Foso/Ktorfit by Jens Klingenberg

HTTP client / Kotlin Symbol Processor for Kotlin Multiplatform (Js, Jvm, Android, Native, iOS) using KSP and Ktor clients inspired by Retrofit.

Machine Learning Communities: Q2 ‘22 highlights and achievements

Posted by Nari Yoon, Hee Jung, DevRel Community Manager / Soonson Kwon, DevRel Program Manager

Let’s explore highlights and accomplishments of vast Google Machine Learning communities over the second quarter of the year! We are enthusiastic and grateful about all the activities by the global network of ML communities. Here are the highlights!

TensorFlow/Keras

TFUG Agadir hosted #MLReady phase as a part of #30DaysOfML. #MLReady aimed to prepare the attendees with the knowledge required to understand the different types of problems which deep learning can solve, and helped attendees be prepared for the TensorFlow Certificate.

TFUG Taipei hosted the basic Python and TensorFlow courses named From Python to TensorFlow. The aim of these events is to help everyone learn about the basics of Python and TensorFlow, including TensorFlow Hub, TensorFlow API. The event videos are shared every week via Youtube playlist.

TFUG New York hosted Introduction to Neural Radiance Fields for TensorFlow users. The talk included Volume Rendering, 3D view synthesis, and links to a minimal implementation of NeRF using Keras and TensorFlow. In the event, ML GDE Aritra Roy Gosthipaty (India) had a talk focusing on breaking the concepts of the academic paper, NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis into simpler and more ingestible snippets.

TFUG Turkey, GDG Edirne and GDG Mersin organized a TensorFlow Bootcamp 22 and ML GDE M. Yusuf Sarıgöz (Turkey) participated as a speaker, TensorFlow Ecosystem: Get most out of auxiliary packages. Yusuf demonstrated the inner workings of TensorFlow, how variables, tensors and operations interact with each other, and how auxiliary packages are built upon this skeleton.

TFUG Mumbai hosted the June Meetup and 110 folks gathered. ML GDE Sayak Paul (India) and TFUG mentor Darshan Despande shared knowledge through sessions. And ML workshops for beginners went on and participants built up machine learning models without writing a single line of code.

ML GDE Hugo Zanini (Brazil) wrote Realtime SKU detection in the browser using TensorFlow.js. He shared a solution for a well-known problem in the consumer packaged goods (CPG) industry: real-time and offline SKU detection using TensorFlow.js.

ML GDE Gad Benram (Portugal) wrote Can a couple TensorFlow lines reduce overfitting? He explained how just a few lines of code can generate data augmentations and boost a model’s performance on the validation set.

ML GDE Victor Dibia (USA) wrote How to Build An Android App and Integrate Tensorflow ML Models sharing how to run machine learning models locally on Android mobile devices, How to Implement Gradient Explanations for a HuggingFace Text Classification Model (Tensorflow 2.0) explaining in 5 steps about how to verify the model is focusing on the right tokens to classify text. He also wrote how to finetune a HuggingFace model for text classification, using Tensorflow 2.0.

ML GDE Karthic Rao (India) released a new series ML for JS developers with TFJS. This series is a combination of short portrait and long landscape videos. You can learn how to build a toxic word detector using TensorFlow.js.

ML GDE Sayak Paul (India) implemented the DeiT family of ViT models, ported the pre-trained params into the implementation, and provided code for off-the-shelf inference, fine-tuning, visualizing attention rollout plots, distilling ViT models through attention. (code | pretrained model | tutorial)

ML GDE Sayak Paul (India) and ML GDE Aritra Roy Gosthipaty (India) inspected various phenomena of a Vision Transformer, shared insights from various relevant works done in the area, and provided concise implementations that are compatible with Keras models. They provide tools to probe into the representations learned by different families of Vision Transformers. (tutorial | code)

JAX/Flax

ML GDE Aakash Nain (India) had a special talk, Introduction to JAX for ML GDEs, TFUG organizers and ML community network organizers. He covered the fundamentals of JAX/Flax so that more and more people try out JAX in the near future.

ML GDE Seunghyun Lee (Korea) started a project, Training and Lightweighting Cookbook in JAX/FLAX. This project attempts to build a neural network training and lightweighting cookbook including three kinds of lightweighting solutions, i.e., knowledge distillation, filter pruning, and quantization.

ML GDE Yucheng Wang (China) wrote History and features of JAX and explained the difference between JAX and Tensorflow.

ML GDE Martin Andrews (Singapore) shared a video, Practical JAX : Using Hugging Face BERT on TPUs. He reviewed the Hugging Face BERT code, written in JAX/Flax, being fine-tuned on Google’s Colab using Google TPUs. (Notebook for the video)

ML GDE Soumik Rakshit (India) wrote Implementing NeRF in JAX. He attempts to create a minimal implementation of 3D volumetric rendering of scenes represented by Neural Radiance Fields.

Kaggle

ML GDEs’ Kaggle notebooks were announced as the winner of Google OSS Expert Prize on Kaggle: Sayak Paul and Aritra Roy Gosthipaty’s Masked Image Modeling with Autoencoders in March; Sayak Paul’s Distilling Vision Transformers in April; Sayak Paul & Aritra Roy Gosthipaty’s Investigating Vision Transformer Representations; Soumik Rakshit’s Tensorflow Implementation of Zero-Reference Deep Curve Estimation in May and Aakash Nain’s The Definitive Guide to Augmentation in TensorFlow and JAX in June.

ML GDE Luca Massaron (Italy) published The Kaggle Book with Konrad Banachewicz. This book details competition analysis, sample code, end-to-end pipelines, best practices, and tips & tricks. And in the online event, Luca and the co-author talked about how to compete on Kaggle.

ML GDE Ertuğrul Demir (Turkey) wrote Kaggle Handbook: Fundamentals to Survive a Kaggle Shake-up covering bias-variance tradeoff, validation set, and cross validation approach. In the second post of the series, he showed more techniques using analogies and case studies.

TFUG Chennai hosted ML Study Jam with Kaggle and created study groups for the interested participants. More than 60% of members were active during the whole program and many of them shared their completion certificates.

TFUG Mysuru organizer Usha Rengaraju shared a Kaggle notebook which contains the implementation of the research paper: UNETR – Transformers for 3D Biomedical Image Segmentation. The model automatically segments the stomach and intestines on MRI scans.

TFX

ML GDE Sayak Paul (India) and ML GDE Chansung Park (Korea) shared how to deploy a deep learning model with Docker, Kubernetes, and Github actions, with two promising ways – FastAPI (for REST) and TF Serving (for gRPC).

ML GDE Ukjae Jeong (Korea) and ML Engineers at Karrot Market, a mobile commerce unicorn with 23M users, wrote Why Karrot Uses TFX, and How to Improve Productivity on ML Pipeline Development.

ML GDE Jun Jiang (China) had a talk introducing the concept of MLOps, the production-level end-to-end solutions of Google & TensorFlow, and how to use TFX to build the search and recommendation system & scientific research platform for large-scale machine learning training.

ML GDE Piero Esposito (Brazil) wrote Building Deep Learning Pipelines with Tensorflow Extended. He showed how to get started with TFX locally and how to move a TFX pipeline from local environment to Vertex AI; and provided code samples to adapt and get started with TFX.

TFUG São Paulo (Brazil) had a series of online webinars on TensorFlow and TFX. In the TFX session, they focused on how to put the models into production. They talked about the data structures in TFX and implementation of the first pipeline in TFX: ingesting and validating data.

TFUG Stockholm hosted MLOps, TensorFlow in Production, and TFX covering why, what and how you can effectively leverage MLOps best practices to scale ML efforts and had a look at how TFX can be used for designing and deploying ML pipelines.

Cloud AI

ML GDE Chansung Park (Korea) wrote MLOps System with AutoML and Pipeline in Vertex AI on GCP official blog. He showed how Google Cloud Storage and Google Cloud Functions can help manage data and handle events in the MLOps system.

He also shared the Github repository, Continuous Adaptation with VertexAI’s AutoML and Pipeline. This contains two notebooks to demonstrate how to automate to produce a new AutoML model when the new dataset comes in.

TFUG Northwest (Portland) hosted The State and Future of AI + ML/MLOps/VertexAI lab walkthrough. In this event, ML GDE Al Kari (USA) outlined the technology landscape of AI, ML, MLOps and frameworks. Googler Andrew Ferlitsch had a talk about Google Cloud AI’s definition of the 8 stages of MLOps for enterprise scale production and how Vertex AI fits into each stage. And MLOps engineer Chris Thompson covered how easy it is to deploy a model using the Vertex AI tools.

Research

ML GDE Qinghua Duan (China) released a video which introduces Google’s latest 540 billion parameter model. He introduced the paper PaLM, and described the basic training process and innovations.

ML GDE Rumei LI (China) wrote blog postings reviewing papers, DeepMind’s Flamingo and Google’s PaLM.

ML in Action: Campaign to Collect and Share Machine Learning Use Cases

Posted by Hee Jung, Developer Relations Community Manager / Soonson Kwon, Developer Relations Program Manager

ML in Action is a virtual event to collect and share cool and useful machine learning (ML) use cases that leverage multiple Google ML products. This is the first run of an ML use case campaign by the ML Developer Programs team.

Let us announce the winners right now, right here. They have showcased practical uses of ML, and how ML was adapted to real life situations. We hope these projects can spark new applied ML project ideas and provide opportunities for ML community leaders to discuss ML use cases.

4 Winners of “ML in Action” are:

Detecting Food Quality with Raspberry Pi and TensorFlow

By George Soloupis, ML Google Developer Expert (Greece)

 

This project helps people with smell impairment by identifying food degradation. The idea came suddenly when a friend revealed that he has no sense of smell due to a bike crash. Even with experiences attending a lot of IT meetings, this issue was unaddressed and the power of machine learning is something we could rely on. Hence the goal. It is to create a prototype that is affordable, accurate and usable by people with minimum knowledge of computers.

The basic setting of the food quality detection is this. Raspberry Pi collects data from air sensors over time during the food degradation process. This single board computer was very useful! With the GUI, it’s easy to execute Python scripts and see the results on screen. Eight sensors collected data of the chemical elements such as NH3, H2s, O3, CO, and CH4. After operating the prototype for one day, categories were set following the results. The first hours of the food out of the refrigerator as “good” and the rest as “bad”. Then the dataset was evaluated with the help of TensorFlow and the inference was done with TensorFlow Lite.

Since there were no open source prototypes out there with similar goals, it was a complete adventure. Sensors on PCBs and standalone sensors were used to get the best mixture of accuracy, stability and sensitivity. A logic level converter has been used to minimize the use of resistors, and capacitors have been placed for stability. And the result, a compact prototype! The Raspberry Pi could attach directly on with slots for eight sensors. It is developed in such a way that sensors can be replaced at any time. Users can experiment with different sensors. And the inference time values are sent through the bluetooth to a mobile device. As an end result a user with no advanced technical knowledge will be able to see food quality on an app built on Android (Kotlin).

Reference: Github, more to read

* This project is supported by Google Impact Fund.

Election Watch: Applying ML in Analyzing Elections Discourse and Citizen Participation in Nigeria

By Victor Dibia, ML Google Developer Expert (USA)

This project explores the use of GCP tools in ingesting, storing and analyzing data on citizen participation and election discourse in Nigeria. It began on the premise that the proliferation of social media interactions provides an interesting lens to study human behavior, and ask important questions about election discourse in Nigeria as well as interrogate social/demographic questions.

It is based on data collected from twitter between September 2018 to March 2019 (tweets geotagged to Nigeria and tweets containing election related keywords). Overall, the data set contains 25.2 million tweets and retweets, 12.6 million original tweets, 8.6 million geotagged tweets and 3.6 million tweets labeled (using an ML model) as political.

By analyzing election discourse, we can learn a few important things including – issues that drive election discourse, how social media was utilized by candidates, and how participation was distributed across geographic regions in the country. Finally, in a country like Nigeria where updated demographics data is lacking (e.g., on community structures, wealth distribution etc), this project shows how social media can be used as a surrogate to infer relative statistics (e.g., existence of diaspora communities based on election discussion and wealth distribution based on device type usage across the country).

Data for the project was collected using python scripts that wrote tweets from the Twitter streaming api (matching certain criteria) to BigQuery. BigQuery queries were then used to generate aggregate datasets used for visualizations/analysis and training machine learning models (political text classification models to label political text and multi class classification models to label general discourse). The models were built using Tensorflow 2.0 and trained on Colab notebooks powered by GCP GPU compute VMs.

References: Election Watch website, ML models descriptions one, two


Bioacoustic Sound Detector (To identify bird calls in soundscapes)

By Usha Rengaraju, TFUG Organizer (India)

 
(Bird image is taken by Krisztian Toth @unsplash)

“Visionary Perspective Plan (2020-2030) for the conservation of avian diversity, their ecosystems, habitats and landscapes in the country” proposed by the Indian government to help in the conservation of birds and their habitats inspired me to take up this project.

Extinction of bird species is an increasing global concern as it has a huge impact on food chains. Bioacoustic monitoring can provide a passive, low labor, and cost-effective strategy for studying endangered bird populations. Recent advances in machine learning have made it possible to automatically identify bird songs for common species with ample training data. This innovation makes it easier for researchers and conservation practitioners to accurately survey population trends and they’ll be able to regularly and more effectively evaluate threats and adjust their conservation actions.

This project is an implementation of a Bioacoustic monitor using Masked Autoencoders in TensorFlow and Cloud TPUs. The project will be presented as a browser based application using Flask. The deep learning prototype can process continuous audio data and then acoustically recognize the species.

The goal of the project when I started was to build a basic prototype for monitoring of rare bird species in India. In future I would like to expand the project to monitor other endangered species as well.

References: Kaggle Notebook, Colab Notebook, Github, the dataset and more to read


Persona Labs’ Digital Personas

By Martin Andrews and Sam Witteveen, ML Google Developer Experts (Singapore)

Over the last 3 years, Red Dragon AI (a company co-founded by Martin and Sam) has been developing real-time digital “Personas”. The key idea is to enable users to interact with life-like Personas in a format similar to a Zoom call : Speaking to them and seeing them respond in real time, just as a human would. Naturally, each Persona can be tailored to tasks required (by adjusting the appearance, voice, and ‘motivation’ of the dialog system behind the scenes and their corresponding backend APIs).

The components required to make the Personas work effectively include dynamic face models, expression generation models, Text-to-Speech (TTS), dialog backend(s) and Speech Recognition (ASR). Much of this was built on GCP, with GPU VMs running the (many) Deep Learning models and combining the outputs into dynamic WebRTC video that streams to users via a browser front-end.

Much of the previous years’ work focussed on making the Personas’ faces behave in a life-like way, while making sure that the overall latency (i.e. the time between the Persona hearing the user asking a question, to their lips starting the response) is kept low, and the rendering of individual images matches the 25 frames-per-second video rate required. As you might imagine, there were many Deep Learning modeling challenges, coupled with hard engineering issues to overcome.

In terms of backend technologies, Google Cloud GPUs were used to train the Deep Learning models (built using TensorFlow/TFLite, PyTorch/ONNX & more recently JAX/Flax), and the real-time serving is done by Nvidia T4 GPU-enabled VMs, launched as required. Google ASR is currently used as a streaming backend for speech recognition, and Google’s WaveNet TTS is used when multilingual TTS is needed. The system also makes use of Google’s serverless stack with CloudRun and Cloud Functions being used in some of the dialog backends.

Visit the Persona’s website (linked below) and you can see videos that demonstrate several aspects : What the Personas look like; their Multilingual capability; potential applications; etc. However, the videos can’t really demonstrate what the interactivity ‘feels like’. For that, it’s best to get a live demo from Sam and Martin – and see what real-time Deep Learning model generation looks like!

Reference: The Persona Labs website

Machine Learning Communities: Q1 ‘22 highlights and achievements

Posted by Nari Yoon, Hee Jung, DevRel Community Manager / Soonson Kwon, DevRel Program Manager

Let’s explore highlights and accomplishments of vast Google Machine Learning communities over the first quarter of the year! We are enthusiastic and grateful about all the activities that the communities across the globe do. Here are the highlights!

ML Ecosystem Campaign Highlights

ML Olympiad is an associated Kaggle Community Competitions hosted by Machine Learning Google Developers Experts (ML GDEs) or TensorFlow User Groups (TFUGs) sponsored by Google. The first round was hosted from January to March, suggesting solving critical problems of our time. Competition highlights include Autism Prediction Challenge, Arabic_Poems, Hausa Sentiment Analysis, Quality Education, Good Health and Well Being. Thank you TFUG Saudi, New York, Guatemala, São Paulo, Pune, Mysuru, Chennai, Bauchi, Casablanca, Agadir, Ibadan, Abidjan, Malaysia and ML GDE Ruqiya Bin Safi, Vinicius Fernandes Caridá, Yogesh Kulkarni, Mohammed buallay, Sayed Ali Alkamel, Yannick Serge Obam, Elyes Manai, Thierno Ibrahima DIOP, Poo Kuan Hoong for hosting ML Olympiad!

Highlights and Achievements of ML Communities

TFUG organizer Ali Mustufa Shaikh (TFUG Mumbai) and Rishit Dagli won the TensorFlow Community Spotlight award (paper and code). This project was supported by provided Google Cloud credit.

ML GDE Sachin Kumar (Qatar) posted Build a retail virtual agent from scratch with Dialogflow CX – Ultimate Chatbot Tutorials. In this tutorial, you will learn how to build a chatbot and voice bot from scratch using Dialogflow CX, a Conversational AI Platform (CAIP) for building conversational UIs.

ML GDE Ngoc Ba (Vietnam) posted MTet: Multi-domain Translation for English and Vietnamese. This project is about how to collect high quality data and train a state-of-the-art neural machine translation model for Vietnamese. And it utilized Google Cloud TPU, Cloud Storage and related GCP products for faster training.

Kaggle announced the Google Open Source Prize early this year (Winners announcement page). In January, ML GDE Aakash Kumar Nain (India)’s Building models in JAX – Part1 (Stax) was awarded.

In February, ML GDE Victor Dibia (USA)’s notebook Signature Image Cleaning with Tensorflow 2.0 and ML GDE Sayak Paul (India) & Soumik Rakshit’s notebook gaugan-keras were awarded.

TFUG organizer Usha Rengaraju posted Variable Selection Networks (AI for Climate Change) and Probabilistic Bayesian Neural Networks using TensorFlow Probability notebooks on Kaggle. They both got gold medals, and she has become a Triple GrandMaster!

TFUG Chennai hosted the two events, Transformers – A Journey into attention and Intro to Deep Reinforcement Learning. Those events were planned for beginners. Events include introductory sessions explaining the transformers research papers and the basic concept of reinforcement learning.

ML GDE Margaret Maynard-Reid (USA), Nived P A, and Joel Shor posted Our Summer of Code Project on TF-GAN. This article describes enhancements made to the TensorFlow GAN library (TF-GAN) of the last summer.

ML GDE Aakash Nain (India) released a series of tutorials about building models in JAX. In the second tutorial, Aakash uses one of the most famous and most widely used high-level libraries for Jax to build a classifier. In the notebook, you will be taking a deep dive into Flax, too.

ML GDE Bhavesh Bhatt (India) built a model for braille to audio with 95% accuracy. He created a model that translates braille to text and audio, lending a helping hand to people with visual disabilities.

ML GDE Sayak Paul (India) recently wrote Publishing ConvNeXt Models on TensorFlow Hub. This is a contribution from the 30 versions of the model, ready for inference and transfer learning, with documentation and sample code. And he also posted First Steps in GSoC to encourage the fellow ML GDEs’ participation in Google Summer of Code (GSoC).

ML GDE Merve Noyan (Turkey) trained 40 models on keras.io/examples; built demos for them with Streamlit and Gradio. And those are currently being hosted here. She also held workshops entitled NLP workshop with TensorFlow for TFUG Delhi, TFUG Chennai, TFUG Hyderabad and TFUG Casablanca. It covered the basic to advanced topics in NLP right from Transformers till model hosting in Hugging Face, using TFX and TF Serve.

How is Dev Library useful to the open-source community?

Posted by Ankita Tripathi, Community Manager (Dev Library)

Witnessing a plethora of open-source enthusiasts in the developer ecosystem in recent years gave birth to the idea of Google’s Dev Library. The inception of the platform happened in June 2021 with the only objective of giving visibility to developers who have been creating and building projects relentlessly using Google technologies. But why the Dev Library?

Why Dev Library?

Open-source communities are currently at a boom. The past 3 years have seen a surge of folks constantly building in public, talking about open-source contributions, digging into opportunities, and carving out a valuable portfolio for themselves. The idea behind the Dev Library as a whole was also to capture these open-source projects and leverage them for the benefit of other developers.

This platform acted as a gold mine for projects created using Google technologies (Android, Angular, Flutter, Firebase, Machine Learning, Google Assistant, Google Cloud).

With the platform, we also catered to the burning issue – creating a central place for the huge number of projects and articles scattered across various platforms. Therefore, the Dev Library became a one-source platform for all the open source projects and articles for Google technologies.

How can you use the Dev Library?

“It is a library full of quality projects and articles.”

External developers cannot construe Dev Library as the first platform for blog posts or projects, but the vision is bigger than being a mere platform for the display of content. It envisages the growth of developers along with tech content creation. The uniqueness of the platform lies in the curation of its submissions. Unlike other platforms, you don’t get your submitted work on the site by just clicking ‘Submit’. Behind the scenes, Dev Library has internal Google engineers for each product area who:

  • thoroughly assess each submission,
  • check for relevancy, freshness, and quality,
  • approve the ones that pass the check, and reject the others with a note.

It is a painstaking process, and Dev Library requires a 4-6 week turnaround time to complete the entire curation procedure and get your work on the site.

What we aim to do with the platform:

  • Provide visibility: Developers create open-source projects and write articles on platforms to bring visibility to their work and attract more contributions. Dev Library’s intention is to continue to provide this amplification for the efforts and time spent by external contributors.
  • Kickstart a beginner’s open-source contribution journey: The biggest challenge for a beginner to start applying their learnings to build Android or Flutter applications is ‘Where do I start my contributions from’? While we see an open-source placard unfurled everywhere, beginners still struggle to find their right place. With the Dev Library, you get a stack of quality projects hand-picked for you keeping the freshness of the tech and content quality intact. For example, Tomas Trajan, a Dev Library contributor created an Angular material starter project where they have ‘good first issues’ to start your contributions with.
  • Recognition: Your selection of the content on the Dev Library acts as recognition to the tiring hours you’ve put in to build a running open-source project and explain it well. Dev Library also delivers hero content in their monthly newsletter, features top contributors, and is in the process to gamify the developer efforts. As an example, one of our contributors created a Weather application using Android and added a badge ‘Part of Dev Library’.

    With your contributions at one place under the Author page, you can use it as a portfolio for your work while simultaneously increasing your chances to become the next Google Developer Expert (GDE).

Features on the platform

Keeping developers in mind, we’ve updated features on the platform as follows:

  • Added a new product category; Google Assistant – All Google Assistant and Smart home projects now have a designated category on the Dev Library.
  • Integrated a new way to make submissions across product areas via the Advocu form.
  • Introduced a special section to submit Cloud Champion articles on Google Cloud.
  • Included displays on each Author page indicating the expertise of individual contributors
  • Upcoming: An expertise filter to help you segment out content based on Beginner, Intermediate, or Expert levels.

To submit your idea or suggestion, refer to this form, and put down your suggestions.

Contributor Love

Dev Library as a platform is more about the contributors who lie on the cusp of creation and consumption of the available content. Here are some contributors who have utilized the platform their way. Here’s how the Dev Library has helped along their journey:

Roaa Khaddam: Roaa is a Senior Flutter Mobile Developer and Co-Founder at MultiCaret Inc.

How has the Dev Library helped you?

“It gave me the opportunity to share what I created with an incredible community and look at the projects my fellow Flutter mates have created. It acts as a great learning resource.”

Somkiat Khitwongwattana: Somkiat is an Android GDE and a consistent user of Android technology from Thailand.

How has the Dev Library helped you?

“I used to discover new open source libraries and helpful articles for Android development in many places and it took me longer than necessary. But the Dev Library allows me to explore these useful resources in one place.”

Kevin Kreuzer: Kevin is an Angular developer and contributes to the community in various ways.

How has the Dev Library helped you?

“Dev Library is a great tool to find excellent Angular articles or open source projects. Dev Library offers a great filtering function and therefore makes it much easier to find the right open source library for your use case.”

What started as a platform to highlight and showcase some open-source projects has grown into a product where developers can share their learnings, inspire others, and contribute to the ecosystem at large.

Do you have an Open Source learning or project in the form of a blog or GitHub repo you’d like to share? Please submit it to the Dev Library platform. We’d love to add you to our ever growing list of developer contributors!

Stepping up as a Machine Learning Developer —My Experience With the Google Machine Learning Bootcamp

Posted by Hyunkil Kim, Software Quality Engineer at Line Corp.

banner image that includes math chart, brain, and GDS logo

This article is written by Hyunkil Kim who participated in the Machine Learning Bootcamp which is a machine learning training program conducted in Korea to nurture next-generation ML engineers and help them to find jobs.

banner image with text that reads google developers machine learning bootcamp

As a developer, I had developed a certain level of curiosity about machine learning. I had also heard that many former developers were switching their specialization over to machine learning. Thus, I signed up for the <Google Machine Learning Bootcamp>, thinking it would be a good chance to get my feet wet.

I was a bit nervous and excited at the same time after getting the acceptance notification. Wondering if I should go over my Python skills one more time in preparation, I installed the newest version of TensorFlow on my machine. I also skimmed through documents on the basics of machine learning. Those were all unnecessary. To put it bluntly, I had to relearn everything from scratch over the course of the bootcamp. It was quite challenging to be introduced to new concepts I wasn’t familiar with, such as functional API and the concept of functional programming in general, various visualization libraries, and data processing frameworks and services that were new to me. I worked very hard with the mindset of starting fresh.

Journey to Becoming a Machine Learning Engineer

There were three main objectives for the participants: completing the Deep Learning Specialization on Coursera which is based on TensorFlow, acquiring ML certifications(TensorFlow certificate or Google Cloud ML(or Data Science) Engineer certification), and participating in Kaggle competitions. Google Developers team provided the course fee for Coursera and the certification fee and offered many benefits to those who completed the course. You could really make it worth your while as long as you took the initiative and applied your passion.

<Coursera Deep Learning Specialization>

The Coursera class is based on TensorFlow 2.x and requires watching a set amount of instructor Andrew Ng’s lectures on AI every week with screenshots and proof. It was pretty tough at first as the lectures were not in Korean. However, because the class was so famous, I was able to find posts on the internet that broke down the lectures and made them easier to understand. The class also provided reference links, so you could study more on your own once you got used to the class.

While this is not really related to the Coursera class, I also participated in online coding meetups by the bootcamp participants in-between classes as in the picture below, and it was a memorable experience. These are basically sessions held in coffee shops or study rooms where people got together and worked individually on their own coding projects in normal times. Because of the pandemic, we could not meet in person obviously and used Google Meet or Gather town and left our cameras on as we coded. It felt like I was studying with other people, and I liked the solidarity of relating to others.

animated image of cartoon figures in a dining room

<Machine Learning Certifications>

You were required to acquire at least one certification during the bootcamp. I chose to work on the GCP ML Engineer certification. As I used Google Cloud, I had wondered how ML services could be used on cloud. Coursera happened to have a specialization program for the GCP ML certification, so I took it, too. However, in the end, Google’s website offering GCP AI operations and use cases helped me more with the certification than the course on Coursera.

Image of Google Cloud certification awarded to Hyunkil Kim

<Kaggle Competition>

I didn’t get to spend as much time on Kaggle. I didn’t see any current projects that interested me, so I tried the TPS to review what I had learned so far. TPS stands for Tabular Playground Series, which is a beginner-to-intermediate level competition for new-ish Kagglers that are just getting the hang of it. You’re required to predict the value of the target from the provided tabular data. It is slightly more difficult than Titanic Survival Predictions, which is a beginner competition. I chose this competition because I figured it would be a good practice of things I had learned so far, like data analysis, feature engineering, and hyperparameter tuning.

Image of duck shown as Hyunkil Kim's profile picture on the Kaggle dashboard

This was the part where I personally felt like I could have done better. I had many ideas for improving the model or enhancing the performance, but it took way more time to apply and experiment with them than I had expected. If I had known that model learning would take this much time, I would have started working on Coursera, the certification, and the Kaggle competition all at once from the beginning. Maybe I was too nervous about entering a Kaggle competition and put it off until the end. I should have just tried without getting so nervous. I hesitated too long and ended up regretting it a little too late.

<Tech Talk and Career Talk>

The bootcamp also included many other activities, including a weekly Tech Talk on specific themes and recruiting sessions of potential employers. Companies looking for ML talents were invited and had a chance to introduce themselves, explain the available positions, and take questions about joining their workforce. Some companies sent their current Machine Learning engineers to explain how they solved business problems with which models or what kind of data. Some companies focused more on describing the type of people they were looking for in detail. I didn’t know at the time, but I heard that some of the speakers were big names in the industry. Personally, I found these talks very helpful in terms of both finding employment and familiarizing myself with the trends in the industry. The sessions were very inspiring as new ideas kept flowing as I heard about applications of technologies I only knew in theory or thought about what kind of investments in AI would be promising.

Besides the Tech Talks, there were also more relaxed sessions for things like career consultation and resume/CV reviews. There were even sessions by the Googlers, where they personally answered participants’ questions and offered some advice. As I attended various sessions, I noticed that the bootcamp crew and many Tech Talk speakers from hiring companies offered authentic and valuable advice and were very eager to help out the bootcamp participants. Nobody talked about the cold reality of the world out there. Knowing how rare it is to find mentors that offer genuinely constructive feedback and guidance, I personally was very touched and grateful about that.

Concluding the Machine Learning Bootcamp.

The Google Machine Learning Bootcamp captured the essence of what it would be like to work for Google. I felt like they expected you to take your own initiative to do what you wanted. They showed that they were willing and able to help you grow as much as possible as long as you did your best. For example, one of the world’s most famous programmers Jeff Dean was at the kickoff session, and there was even an AMA session with Laurence Moroney, who had developed the training course for TensorFlow. They also allowed maximum freedom about finding teammates for the Kaggle competitions so that you didn’t have to worry about having to carry your team. Things covered in the Tech Talks or recruitment sessions were not included in assignments. They let the participants do their thing freely while promising the best support possible in the industry if needed. I could see how some people would find it too lax that Google lets you study on your own at your own pace.

Image of video conference call with Andrew Ng, Jeff Dean, and Laurence Moroney

I think this was a rare chance to meet people from various backgrounds with the common goal of becoming machine learning engineers or developers. It was a unique experience where I got to talk and study with good people and even do something strange like the online coding meetup. There were also times when I was vainly taking pride in what little knowledge I had, but I ended up putting a lot of work into the bootcamp, wanting to make the most of it and to come ahead of others.

In the end, the take-home message is to “try anything.”

Personally, I was very happy with the experience. I got to be a little more comfortable with machine learning. As a result, I’m able to pay more attention to details related to machine learning at my new job. The challenge of facing something new is a constant of a developer’s life. Still, participating in this bootcamp felt especially meaningful to me, and I enjoyed it thoroughly.

While the bootcamp is over, I heard that some participants are still continuing with their study groups or projects. Wanting to study as a group myself, I also had asked around and volunteered to join a study group, but I ended up studying alone because none of the groups covered the area I was interested in. Even so, many people sharing useful information on Slack helped me as I studied alone, and they are still helping me even after the bootcamp.

At any rate, I keep coming up with various ideas that I want to try in my current job or as a personal project. It feels like I found a new toy that I can have fun with for a while without getting tired of it. I think I’ll start slowly with a small toy project.