Azure Archives - KenkoGeek

19 Jul 2019
By editor
Categories: Azure, Database, Partner, Security, web

How HSBC built its PayMe for Business app on Microsoft Azure

Bank-grade security, super-fast transactions, and analytics

If you live in Asia or have ever traveled there, you’ve probably witnessed the dramatic impact that mobile technology has had on all aspects of day to day life. In Hong Kong in particular, most consumers now use a smart phone daily, presenting new opportunities for organizations to deliver content and services directly to their mobile devices.

As one of the world’s largest international banks, HSBC is building new services on the cloud to enable them to organize their data more efficiently, analyze it to understand their customers better, and make more core customer journeys and features available on mobile first.

HSBC’s retail and business banking teams in Hong Kong have combined the convenience afforded by smart phones with cloud services to allow “cashless” transactions where people can use their smart phone to perform payments digitally. Today, over one and a half million people use HSBC’s PayMe app to exchange money with people in their personal network for free. And businesses are using HSBC’s new PayMe for Business app, built natively on Azure, to collect payments instantly, with 98 percent of all transactions completed in 500 milliseconds or less. Additionally, the businesses can leverage powerful built-in intelligence on the app to improve their sales and operations.

On today’s Microsoft Mechanics episode of “How We Built it,” Alessio Basso, Chief Architect of PayMe from HSBC, explains the approach they took and why.

Bank-grade security, faster time to delivery, dynamic scale and resiliency

The first decision Alessio and team made was to use fully managed services to allow them to go from ideation to a fully operational service in just a few months. Critical to their approach was adopting a microservices-based architecture with Azure Kubernetes Service and Azure Database for MySQL.

They designed each microservice to be independent, with their own instance of Azure managed services, including Azure Database for MySQL, Azure Event Hub, Azure Storage, Azure Key Vault for credentials and secrets management, and more. They architected for this level of isolation to strengthen security and overall application uptime, as shared dependencies are eliminated.

Each microservice can rapidly scale compute and database resources elastically and independently, based on demand. What’s more, Azure Database for MySQL, allows for the creation of read replicas to offload read-only and analytical queries without impacting payment transaction response times.

Also, from a security perspective, because each microservice runs within its own subnet inside of Azure Virtual Network, the team is able to isolate network and communications back and forth between Azure resources with service principals via Virtual Network service endpoints.

Fast and responsive analytics platform

At its core, HSBC’s PayMe is a social app that allows consumers to establish their personal networks, while facilitating the interactions and transactions with the people in their circle and business entities. In order to create more value for both businesses and consumers, Azure Cosmos DB is used for graph data modelled to store customer-merchant-transaction relationships.

Massive amounts of structured and unstructured data from Azure Database for MySQL, Event Hubs, and Storage are streamed and transformed. The team designed an internally developed data ingestion process, feeding an analytical model called S.L.I.M (simple, lightly, integrated model), optimized for analytics queries performance, as well as making data virtually available to the analytics platform, using Azure Databricks Delta’s unmanaged table capability.

Then machine learning within their analytics platform built on Azure Databricks allows for the quick determination of patterns and relationships, as well as for the detection of anomalous activity.

With Azure, organizations can immediately take advantage of new opportunities to deliver content and services directly to mobile devices, including a next-level digital payment platform.

To learn more about how HSBC architected their cashless digital transaction platform, please watch the full episode.
Learn more about achieving microservice independence with your own instance of a Azure managed service like Azure Database for MySQL.

19 Jul 2019
By editor
Categories: Azure, Big Data

MileIQ and Azure Event Hubs: Billions of miles streamed

This post was co-authored by Shubha Vijayasarathy, Program Manager, Azure Messaging (Event Hubs)

With billions of miles logged, MileIQ provides stress-free logging and accurate mileage reports for millions of drivers. Logging and reporting miles driven is a necessity for independent contractors to organizations with employees who need to drive for work. MileIQ automates mileage logging to create accurate records of miles driven, minimizing the effort and time needed with manual calculations. Real-time mileage tracking produces over a million location signal events per hour, requiring fast and resilient event processing that scales.

MileIQ leverages Apache Kafka to ingest massive streams of data:

Event processing: Events that demand time-consuming processing are put into Kafka, and multiple processors consume and process these asynchronously.
Communication among micro-services: Events are published by the event-owning micro-service on Kafka topics. The other micro-services, which are interested in these events, subscribe to these topics to consume the events.
Data Analytics: As all the important events are published on Kafka, the data analytics team subscribes to the topics it is interested in and pulls all the data it requires for data processing.

Growth Challenges

As with any successful venture, growth introduces operational challenges as infrastructure struggles to support the growing demand. In MileIQ’s case, the effort and resources required to maintain Apache Kafka clusters multiplied exponentially with adoption. A seemingly simple task, like modifying a topic’s retention configuration, now becomes an operational burden as the number of Kafka clusters scale to meet the increase in data.

Leveraging a managed service, enabled MileIQ to shift resources from operations and maintenance to focus on new ways to drive business impact. A couple reasons why the MileIQ team selected Azure Event Hubs for Kafka:

Fully managed platform as a service (PaaS): With little configuration or management overhead, Event Hubs for Kafka provides a PaaS Kafka experience without the need to manage, configure, or run Kafka clusters.
Supports multiple Kafka use-cases: Event Hubs for Apache Kafka provides support at the protocol level, enabling integration of existing Kafka applications with no code changes and minimal configuration change. MileIQ’s existing Kafka producers and consumers, as well as other streaming applications like Apache Kafka MirrorMaker and Apache Spark, integrated seamlessly with the Kafka-enabled Event Hub.
Deliver streaming data to Azure Blob storage: The Capture feature of Event Hubs automatically send data from Azure Event Hubs for Kafka to Blob storage. MileIQ uses the data in Blob storage for data analytics and backup.
Enterprise performance: The Dedicated-tier cluster offers single-tenant deployments with a guaranteed 99.99% SLA. MileIQ performance tests showed the Dedicated-tier cluster was able to consistently produce a throughput rate of 6,000 events per second.

* Testing based on one event at a time synchronously to address specific use-cases focused on consistency over throughput. Testing batching and produce asynchronously resulted in a much higher throughput.

Set up for success

As a result of migrating Apache Kafka to a managed service, MileIQ now has the infrastructure needed to support future growth.

“To sum up, our experience switching over to Azure Event Hubs Kafka has been excellent. To start with, the onboarding was straightforward, integration was seamless, and we continue to receive great help and support from the Azure Event Hubs Kafka team. In the near future, we look forward to the release of new features that the Azure Event Hubs Kafka team is working on – Geo Replication, Idempotent Producers, Kafka Streams, etc.”
“Migrating to Azure Event Hubs Kafka was a painless experience. Straightforward onboarding seamless integration, and support from the Event Hubs team every step of the way. We’re excited to see what’s next and look forward to a continued partnership.” – MileIQ

Start streaming data

Data is valuable only when there is an easy way to process and get timely insights from data sources. Azure Event Hubs provides a fully managed distributed stream processing platform with low latency and seamless integration with Apache Kafka applications.

What are you waiting for? Time to get event-ing!

Event Hubs for Apache Kafka overview
Get started: Create Apache Kafka-enabled event hubs
Discover more about Azure Event Hubs
Learn about Dedicated-tier clusters

Enjoyed this blog? Follow us as we update the features list we will start supporting. Leave us your valuable feedback, questions, or comments below.

Happy event-ing!

18 Jul 2019
By editor
Categories: Azure, machine learning

Microsoft makes it easier to build popular language representation model BERT at large scale

This post is co-authored by Rangan Majumder, Group Program Manager, Bing and Maxim Lukiyanov, Principal Program Manager, Azure Machine Learning.

Today we are announcing the open sourcing of our recipe to pre-train BERT (Bidirectional Encoder Representations from Transformers) built by the Bing team, including code that works on Azure Machine Learning, so that customers can unlock the power of training custom versions of BERT-large models using their own data. This will enable developers and data scientists to build their own general-purpose language representation beyond BERT.

The area of natural language processing has seen an incredible amount of innovation over the past few years with one of the most recent being BERT. BERT, a language representation created by Google AI language research, made significant advancements in the ability to capture the intricacies of language and improved the state of the art for many natural language applications, such as text classification, extraction, and question answering. The creation of this new language representation enables developers and data scientists to use BERT as a stepping-stone to solve specialized language tasks and get much better results than when building natural language processing systems from scratch.

The broad applicability of BERT means that most developers and data scientists are able to use a pre-trained variant of BERT rather than building a new version from the ground up with new data. While this is a reasonable solution if the domain’s data is similar to the original model’s data, it will not deliver best-in-class accuracy when crossing over to a new problem space. For example, training a model for the analysis of medical notes requires a deep understanding of the medical domain, providing career recommendations depend on insights from a large corpus of text about jobs and candidates, and legal document processing requires training on legal domain data. In these cases, to maximize the accuracy of the Natural Language Processing (NLP) algorithms one needs to go beyond fine-tuning to pre-training the BERT model.

Additionally, to advance language representation beyond BERT’s accuracy, users will need to change the model architecture, training data, cost function, tasks, and optimization routines. All these changes need to be explored at large parameter and training data sizes. In the case of BERT-large, this can be quite substantial as it has 340 million parameters and trained over 2.5 billion Wikipedia and 800 million BookCorpus words. To support this with Graphical Processing Units (GPUs), the most common hardware used to train deep learning-based NLP models, machine learning engineers will need distributed training support to train these large models. However, due to the complexity and fragility of configuring these distributed environments, even expert tweaking can end up with inferior results from the trained models.

To address these issues, Microsoft is open sourcing a first of a kind, end-to-end recipe for training custom versions of BERT-large models on Azure. Overall this is a stable, predictable recipe that converges to a good optimum for developers and data scientists to try explorations on their own.

“Fine-tuning BERT was really helpful to improve the quality of various tasks important for Bing search relevance,” says Rangan Majumder, Group Program Manager at Bing, who led the open sourcing of this work. “But there were some tasks where the underlying data was different from the original corpus BERT was pre-trained on, and we wanted to experiment with modifying the tasks and model architecture. In order to enable these explorations, our team of scientists and researchers worked hard to solve how to pre-train BERT on GPUs. We could then build improved representations leading to significantly better accuracy on our internal tasks over BERT. We are excited to open source the work we did at Bing to empower the community to replicate our experiences and extend it in new directions that meet their needs.”

“To get the training to converge to the same quality as the original BERT release on GPUs was non-trivial,” says Saurabh Tiwary, Applied Science Manager at Bing. “To pre-train BERT we need massive computation and memory, which means we had to distribute the computation across multiple GPUs. However, doing that in a cost effective and efficient way with predictable behaviors in terms of convergence and quality of the final resulting model was quite challenging. We’re releasing the work that we did to simplify the distributed training process so others can benefit from our efforts.”

Results

To test the code, we trained BERT-large model on a standard dataset and reproduced the results of the original paper on a set of GLUE tasks, as shown in Table 1. To give you estimate of the compute required, in our case we ran training on Azure ML cluster of 8xND40_v2 nodes (64 NVidia V100 GPUs total) for 6 days to reach listed accuracy in the table. The actual numbers you will see will vary based on your dataset and your choice of BERT model checkpoint to use for the upstream tasks.

Table1. GLUE Test results, evaluated by the provided test script on the GLUE development set. The “Average” column is simple average over the table results. F1 scores are reported for QQP and MRPC, Spearman correlations are reported for STS-B, and accuracy scores are reported for the other tasks. The results for tasks with smaller dataset sizes have significant variation and may require multiple fine-tuning runs to reproduce the results.

The code is available in open source on the Azure Machine Learning BERT GitHub repo. Included in the repo is:

A PyTorch implementation of the BERT model from Hugging Face repo.
Raw and pre-processed English Wikipedia dataset.
Data preparation scripts.
Implementation of optimization techniques such as gradient accumulation and mixed precision.
An Azure Machine Learning service Jupyter notebook to launch pre-training of the model.
A set of pre-trained models that can be used in fine-tuning experiments.
Example code with a notebook to perform fine-tuning experiments.

With a simple “Run All” command, developers and data scientists can train their own BERT model using the provided Jupyter notebook in Azure Machine Learning service. The code, data, scripts, and tooling can also run in any other training environment.

Summary

We could not have achieved these results without leveraging the amazing work of the researchers before us, and we hope that the community can take our work and go even further. If you have any questions or feedback, please head over to our GitHub repo and let us know how we can make it better.

Learn how Azure Machine Learning can help you streamline the building, training, and deployment of machine learning models. Start free today.

18 Jul 2019
By editor
Categories: Azure, Database, Developer

Assess the readiness of SQL Server data estates migrating to Azure SQL Database

Migrating hundreds of SQL Server instances and thousands of databases to Azure SQL Database, our Platform as a Service (PaaS) offering, is a considerable task, and to streamline the process as much as possible, you need to feel confident about your relative readiness for migration. Being able to identify low-hanging fruit including the servers and databases that are fully ready or that require minimal effort to prepare for migration eases and accelerates your efforts. We are pleased to share that Azure database target readiness recommendations have been enabled.

The Azure Migrate hub provides a unified view of all your migrations across the servers, applications, and databases. This integration provides customers with a seamless migration experience beginning during the discovery phase. The functionality allows customers to use assessment tools for visibility into the applications currently run on-premises so that they can determine cloud suitability and project the cost of running their applications in the cloud. It also allows customers to compare options between competing public and hybrid cloud options.

Assessing and viewing results

Assessing the overall readiness of your data estate for a migration to Azure SQL Database requires only a few steps:

Provision an instance of Azure Migrate, create a migration project, and then add Data Migration Assistant to the migration solution to perform the assessment.
After you create the migration project, download Data Migration Assistant and run an assessment against one or more SQL Server instances.
Upload the Data Migration Assistant assessment results to the Azure Migrate hub.

In a few minutes, the Azure SQL Database target readiness results will be available in your Azure Migrate project.

You can use single assessment for as many SQL Servers as you want, or you can run multiple parallel assessments and upload them to the Azure Migrate hub. The Azure Migrate hub consolidates all the assessments and a provide summarized view of SQL Server and database readiness.

The Azure Migrate dashboard provides a view of your data estate and its overall readiness for migration. This includes the number of databases that are ready to migrate to Azure SQL Database and to SQL Server hosted on an Azure virtual machine. Readiness is computed based on feature parity and schema compatibility with various Azure SQL Database offerings. The dashboard also provides insight into overall migration blockers and the all-up effort involved with migrating to Azure.

IT pros and database administrators can drill-down further to view a specific set of SQL Server instances and databases for a better understanding their readiness for migration.

The “Assessed database” view provides an overview of individual databases, showing info like migration blockers and readiness for Azure SQL Database and SQL Servers hosted on an Azure virtual machine.

Get started

Migrations can be overwhelming and a bit daunting, but we’re here with the expertise and tools, like Data Migration Assistant, to support you along the way. Discover your readiness results and acceleration your migration.

Get started:

17 Jul 2019
By editor
Categories: Azure, Azure Stream Analytics, Big Data, Internet of Things, Storage, Backup & Recovery

New capabilities in Stream Analytics reduce development time for big data apps

Azure Stream Analytics is a fully managed PaaS offering that enables real-time analytics and complex event processing on fast moving data streams. Thanks to zero-code integration with over 15 Azure services, developers and data engineers can easily build complex pipelines for hot-path analytics within a few minutes. Today, at Inspire, we are announcing various new innovations in Stream Analytics that help further reduce time to value for solutions that are powered by real-time insights. These are as follows:

Bringing the power of real-time insights to Azure Event Hubs customers

Today, we are announcing one-click integration with Event Hubs. Available as a public preview feature, this allows an Event Hubs customer to visualize incoming data and start to write a Stream Analytics query with one click from the Event Hub portal. Once the query is ready, they will be able to operationalize it in few clicks and start deriving real time insights. This will significantly reduce the time and cost to develop real-time analytics solutions.

One-click integration between Event Hubs and Azure Stream Analytics

Augmenting streaming data with SQL reference data support

Reference data is a static or slow changing dataset used to augment real-time data streams to deliver more contextual insights. An example scenario would be currency exchange rates regularly updated to reflect market trends, and then converting a stream of billing events in different currencies to a common currency of choice.

Now generally available (GA), this feature provides out-of-the-box support for Azure SQL Database as reference data input. This includes the ability to automatically refresh your reference dataset periodically. Also, to preserve the performance of your Stream Analytics job, we provide the option to fetch incremental changes from your Azure SQL Database by writing a delta query. Finally, Stream Analytics leverages versioning of reference data to augment streaming data with the reference data that was valid at the time the event was generated. This ensures repeatability of results.

New analytics functions for stream processing

Pattern matching:
With the new MATCH_RECOGNIZE function, you can easily define event patterns using regular expressions and aggregate methods to verify and extract values from the match. This enables you to easily express and run complex event processing (CEP) on your streams of data. For example, this function will enable users to easily author a query to detect “head and shoulder” patterns on the on a stock market feed.
Use of analytics function as aggregate:
You can now use aggregates such as SUM, COUNT, AVG, MIN, and MAX directly with the OVER clause, without having to define a window. Analytics functions as Aggregates enables users to easily express queries such as “Is the latest temperature greater than the maximum temperature reported in the last 24 hours?”

Egress to Azure Data Lake Storage Gen2

Azure Stream Analytics is a central component within the Big Data analytics pipelines of Azure customers. While Stream Analytics focuses on the real-time or hot-path analytics, services like Azure Data Lake help enable batch processing and advanced machine learning. Azure Data Lake Storage Gen2 takes core capabilities from Azure Data Lake Storage Gen1 such as a Hadoop compatible file system, Azure Active Directory, and POSIX based ACLs and integrates them into Azure Blob Storage. This combination enables best in class analytics performance along with storage tiering and data lifecycle management capabilities and the fundamental availability, security, and durability capabilities of Azure Storage.

Azure Stream Analytics now offers native zero-code integration with Azure Data Lake Storage Gen2 output (preview.)

Enhancements to blob output

Native support for Apache parquet format:
Native support for egress in Apache parquet format into Azure Blob Storage is now generally available. Parquet is a columnar format enabling efficient big data processing. By outputting data in parquet format into a blob store or a data lake, you can take advantage of Azure Stream Analytics to power large scale streaming extract, transfer, and load (ETL), to run batch processing, to train machine learning algorithms, or to run interactive queries on your historical data. We are now announcing general availability of this feature for egress to Azure Blob Storage.
Managed identities (formerly MSI) authentication:
Azure Stream Analytics now offers full support for Managed Identity based authentication with Azure Blob Storage on the output side. Customers can continue to use the connection string based authentication model. This feature is available as a public preview.

Many of these features just started rolling out worldwide and will be available in all regions within several weeks.

Feedback

The Azure Stream Analytics team is highly committed to listening to your feedback and letting the user voice influence our future investments. We welcome you to join the conversation and make your voice heard via our UserVoice page.

16 Jul 2019
By editor
Categories: Announcements, Azure

Introducing proximity placement groups

Co-locate your Azure resources for improved application performance

The performance of your applications is central to the success of your IT organization. Application performance can directly impact your ability to increase customer satisfaction and ultimately grow your business.

Many factors can affect the performance of your applications. One of those is network latency which is impacted, among other things, by the physical distance between the virtual machines deployed.

For example, when you place your Microsoft Azure Virtual Machines in a single Azure region, the physical distance between the virtual machines is reduced. Placing them within a single availability zone is another step you can take to deploy your virtual machines closer to each other. However, as the Azure footprint grows, a single availability zone may span multiple physical data centers resulting in network latency that can impact your overall application performance. If a region does not support availability zones or if your application does not use availability zones, the latency between the application tiers may increase as a result.

Today, we are announcing the preview of proximity placement groups. A new capability that we are making available to achieve co-location of your Azure Infrastructure as a Service (IaaS) resources and low network latency among them.

Azure proximity placement groups represent a new logical grouping capability for your Azure Virtual Machines, which in turn is used as a deployment constraint when selecting where to place your virtual machines. In fact, when you assign your virtual machines to a proximity placement group, the virtual machines are placed in the same data center, resulting in lower and deterministic latency for your applications.

When to use proximity placement groups

Proximity placement groups improve the overall application performance by reducing the network latency among virtual machines. You should consider using proximity placement groups for multi-tiered, IaaS-based deployments where application tiers are deployed using multiple virtual machines, availability sets and/or virtual machine scale sets.

As an example, consider the case where each tier in your application is deployed in an availability set or virtual machine scale set for high availability. Using a single proximity placement group for all the tiers of your applications, even if they use different virtual machine SKUs and sizes, will force all the deployments to follow each other and land in the same data center for best latency.

In order to get the best results with proximity placement groups, make sure you’re using accelerated networking and optimize your virtual machines for low latency.

Getting started with proximity placement groups

The easiest way to start with proximity placement groups is to use them with your Azure Resource Manager (ARM) templates.

To create a proximity placement group resource just add the following statement:

{
"apiVersion": "2018-04-01",
"type": "Microsoft.Compute/proximityPlacementGroups",
"name": "[parameters('ppgName')]",
"location": "[resourceGroup().location]"
}

To use this proximity placement group later in the template with a virtual machine (or availability set or virtual machine scale set), just add the following dependency and property:

{
   "name": "[parameters('virtualMachineName')]",
   "type": "Microsoft.Compute/virtualMachines",
   "apiVersion": "2018-06-01",
   "location": "[parameters('location')]",
   "dependsOn": [
     "[concat('Microsoft.Compute/proximityPlacementGroups/', parameters('ppgName'))]"
   ],
   "properties": {
     "proximityPlacementGroup": {
       "id": "[resourceId('Microsoft.Compute/proximityPlacementGroups',parameters('ppgName'))]"
   }
}

To learn more about proximity placement groups, see the following tutorials on using proximity placement groups with PowerShell and CLI.

What to expect when using proximity placement groups

Proximity placement groups offer co-location in the same data center. However, because proximity placement groups represent an additional deployment constraint, allocation failures can occur (for example, you may not be able to place your Azure Virtual Machines in the same proximity placement group.)

When you ask for the first virtual machine in the proximity placement group, the data center is automatically selected. In some cases, a second request for a different virtual machine SKU may fail since it does not exist in the data center already selected. In this case, an OverconstrainedAllocationRequest error will be returned. To troubleshoot, please check to see which virtual machines are available in the chosen region or zone using the Azure portal or APIs. If all of the desired SKUs are available, try changing the order in which you deploy them.

In the case of elastic deployments, which scale out, having a proximity placement group constraint on your deployment may result in a failure to satisfy the request. When using proximity placement groups, we recommend that you ask for all the virtual machines at the same time.

Proximity placement groups are in preview now and are offered free of charge in all public regions.

Please refer to our documentation for additional information about proximity placement groups.

Here’s what we’ve heard from SAP, who participated in the early preview program:

“It is really great to see this feature now publicly available. We are going to make use of it in our standard deployments. My team is automating large scale deployments of SAP landscapes. To ensure best performance of the systems it is essential to ensure low-latency between the different components of the system. Especially critical is the communication between Application server and the database, as well as the latency between HANA VMs when synchronous replication has to be enabled. In the late 2018 we did some measurements in various Azure regions and found out that sometimes the latency was not as expected and not in the optimal range. While discussing this with Microsoft, we were offered to join the early preview and evaluate the Proximity Placement Groups (PPG) feature. During our evaluation we were able to bring down the latency to less than 0.3 ms between all system components, which is more than sufficient to ensure great system performance. Best deterministic results we achieved when PPGs were combined with Network acceleration of VM NICs, which additionally improved the measured latencies.”

Ventsislav Ivanov, Development Architect, SAP

16 Jul 2019
By editor
Categories: Azure, Partner

Digital distribution centers—The future is here

The pace of change has never been as fast as it is now. Globally, the population is becoming more urban and income levels are rising. By 2050, nearly 70 percent of the global population will live in cities or urban areas—that’s six billion people. Consumer behavior has also materially changed over the last decade, and omnichannel retail, personalization, and demand for same day deliveries are growing. To cater to the changing landscape, urban distribution centers that stage products closer to users within large cities are on the rise to enable faster delivery and greater customization.

Within the four walls of the distribution center, picking and packing tasks account for more than 50 percent of the total labor cost of warehousing operations. Access to labor has become increasingly challenging, particularly in urban centers, and staffing levels shoot up five to ten-times normal levels during the holiday season. Space constraints and difficulty in staffing are pushing companies to look at adopting distribution center technologies that cut labor costs, optimizes the flow of products, and improves productivity and utilization of these centers.

Since announcing Microsoft’s $5B commitment to developing an industry leading internet of things (IoT) platform last year, we’ve continued to work with our ecosystem partners to build solutions to address such problems. In “Our IoT Vision and Roadmap” session at Microsoft Build, we announced a partnership with Lenovo and NVIDIA, to bring advanced artificial intelligence (AI) to Azure IoT Edge. The demonstrated solution showed Lenovo hardware, a single SE350 Edge Server, running the Azure IoT Edge runtime with NVIDIA DeepStream to process multiple channels of 1080P/30FPS H265 video streams in real-time, transforming cameras into smart sensors that understand their physical environments and use vision algorithms to find missing products on a shelf or detect damaged goods. Such applications of Azure IoT Edge technology enable customers to quickly and cost effectively deploy retail solutions that optimize their logistics operations.

Today, we are excited to announce the next milestone on this journey, the preview of the Lenovo’s Digital Distribution Center (DDC) solution. Lenovo’s DDC is an IoT solution developed in collaboration with NVIDIA and Microsoft. Through real-time scalable package detection, tracking, and validation, DDC delivers for better optimization and increased utilization of distribution centers for retail, manufacturing, and logistics operations. The solution uses multi-video stream analytics with artificial intelligence and machine learning inferencing to self-learn, optimize, and scale. Additional releases will include geofencing alerts, palletization, depalletization, and last-mile sorting.

DDC is built with Azure IoT Central, Microsoft’s fully managed IoT app platform that makes it easy to connect, monitor, and manage your IoT devices and products. Azure IoT Central simplifies the initial setup of your IoT solution and reduces the management burden, operational costs, and overhead of a typical IoT project. This allows solution builders to apply their energy and unique domain expertise to solving customer needs and creating business value, rather than needing to tackle the operating, managing, securing, and scaling of a global IoT solution. Partners like Lenovo and NVIDIA add unique value through schemas that are relevant to industry solutions like DDC, including common industry hierarchies that organize people, places, and environments.

Join us for a demo of our solution at the Microsoft partner booth during Microsoft Inspire—July 14-18, 2019, in Las Vegas, Nevada. If you are interested in joining our preview program about the solution, please contact [email protected]

13 Jul 2019
By editor
Categories: Artificial Intelligence, Azure

Exploring the Microsoft Healthcare Bot partner program

This post was co-authored by Hadas Bitran, Group Manager, Microsoft Healthcare Israel.

Every day, healthcare organizations are beginning their digital transformation journey with the Microsoft Healthcare Bot Service built on Azure. The Healthcare Bot service empowers healthcare organizations to build and deploy an Artificial Intelligence (AI) powered, compliant, conversational healthcare experience at scale. The service combines built-in medical intelligence with natural language capabilities, extensibility tools, and compliance constructs, allowing healthcare organizations such as providers, payers, pharma, HMOs, and telehealth to give people access to trusted and relevant healthcare services and information.

Healthcare organizations can leverage the Healthcare Bot Service on their digital transformation journey today, as we announced in our blog Microsoft Healthcare Bot brings conversational AI to healthcare. That’s why we are so happy to share more information on the Healthcare Bot Service partner program. Our Healthcare Bot certified partners empower healthcare organizations to successfully deploy virtual assistants on the Microsoft Healthcare Bot service. Working with an official partner, healthcare organizations can achieve the full potential of the Microsoft Healthcare Bot by leveraging the expertise and experience of partners who understand the business needs and challenges in healthcare.

This new program is open to existing Microsoft partners that support organizations in the healthcare domain, and delivers the training and resources required to support customers with end to end solutions using Microsoft’s Healthcare Bot Service. The program is designed to support partner success and enable partners to provide tailored solutions using the Healthcare Bot service as a foundation.

With the power of the cloud and a platform that is uniquely built for healthcare conversational intelligence, partners can quickly demonstrate value and iterate on solutions for customers. Official partners have access to partner-only resources and benefits that will enable them to provide customers with differentiated and value-added offerings such as:

Partner listing in the Healthcare Bot partner directory.
Preferential messaging tiers.
Free demonstration and proof of concept Healthcare Bot Instances.
Direct support channel from the product team.
Partner resources including sales materials, product updates and release notes.

The Microsoft Healthcare Bot service helps partners bring conversational AI to innovative healthcare organizations. Partners can support healthcare organizations to deploy customized conversational experiences at scale, reducing costs and improving outcomes for their patients with virtual assistants built to complement their healthcare services.

The Healthcare Bot provides partners with a comprehensive platform to automate healthcare engagements and provides patients with instant access to the services they need. The service facilitates multi-channel healthcare conversations such as chat bots or handoff to live nurses over Microsoft Teams. Partners can build differentiated offerings and create unique conversational healthcare experiences that support the type of digital interaction required by the patient.

Next steps

Partners interested in certification should submit a request to [email protected]. Healthcare organizations seeking certified Healthcare Bot partners can find more information in the official partner directory.

12 Jul 2019
By editor
Categories: Azure, Azure Marketplace, Partner

Enhancing Microsoft’s commercial marketplace for partners

As we head into the global partner conference Microsoft Inspire on July 14-18, 2019, a big focus is on rethinking how we make it easier for customers to discover, try, and buy cloud-based software and services from our partners. Today, we’re excited to announce new tools, commerce options, and a rewards program through the Microsoft commercial marketplace that help partners leverage this important distribution channel.

Commercial marketplace as a new distribution channel

Many people think of a commercial marketplace as a simple catalog of offer listings which are often difficult to navigate. For customers, they are often linked off to a different experience for trial and purchase. Publishers and partner selling solutions are challenged by how to differentiate their solutions to stand out in the volume of offers.

We are working with our partner community to ensure the commercial marketplace experiences deliver a new distribution channel to drive their business growth. For example, Microsoft AppSource targets business decision makers while Azure Marketplace targets IT and developers. This includes having the commerce capabilities and solution supply to capture the rising customer demand in online enterprise software purchases.

Microsoft’s commercial marketplace has at its core, one product catalog, which includes both Microsoft cloud software and services as well as software and services from our partners built on top of and to connect with one or more cloud services offered by Microsoft (Microsoft 365, Dynamics 365, Microsoft Power Platform, and Azure) publishing as transactable offers. This is not just for independent software vendors (ISVs) creating repeatable intellectual property (IP). The commercial marketplace experiences also support offers from managed service providers (MSPs) and consulting services from systems integrators (SIs) such as one-day assessments, migration offers, and more.

Customers can discover, try, and buy solutions from the marketplace in one of three ways:

Direct from the publishers
Through our field sales teams who retire quota for selling eligible partner solutions, or
Through our global distribution channel, where we now also pay the channel a 10 percent incentive to sell marketplace publisher solutions with a transactable SaaS offer, and who participate in the IP co-sell program.

Customers are looking for quicker buying experiences where they can purchase Microsoft products AND solutions from our partners – together in one place, with one transaction, on a unified invoice, which the commercial marketplace provides.

Using the commercial marketplace as a strategic distribution channel will require partners to think about their business model in new and different ways, which can provide significant new revenue streams. For instance, any publisher can continue to list or trial their solution in Microsoft AppSource or Azure Marketplace, but the impact will likely be similar to what they face today, where the customer discovery experience is crowded due to volume of offers and the publisher struggles to differentiate their solution. However, when a publisher chooses to transact in Microsoft’s commercial marketplace, they get access to a whole new set of benefits and ways to sell:

Gain access to a global reseller channel with over 70,000 cloud solution providers (CSP) in over 140 countries who receive an incentive directly from Microsoft when they resell publisher solutions.
Provides simplified deal-making with custom contract amendments.
Centralized partnership experience via Partner Center for the commercial marketplace onboarding, lead sharing, deal registration, benefits, incentives, sales analytics, and investments.
New go-to-market (GTM) benefits via marketplace rewards that unlocks GTM benefits for publishers as they reach various transaction thresholds.

A single onboarding and management experience

Whether a customer buys direct through Microsoft field sellers or through CSP, each of these channels is accessible and managed by partners through a single ingestion point known as Partner Center. Within Partner Center, publishers can publish marketplace offers and manage their engagements, while resellers can bundle Microsoft software and services with publisher’s software and services. This simplifies customer, publisher, and reseller engagement with one transaction and one invoice.

New commerce options

To accompany this new publisher experience, we’ve released new commerce capabilities that partners of all sizes are already starting to benefit from such as ESRI with site-based SaaS, Barracuda, and Trend Micro who use custom business models for their SaaS-based applications. Approved Contact, Crossware, and MongoDB are also using the per-seat SaaS capabilities and managed services from long-time Microsoft partners like Ingram Micro.

These new commerce enhancements allow publishers to customize their offers to meet customer needs and scale through the global reach of Microsoft’s customer and channel communities.

Marketplace rewards

We’re also sharing marketplace rewards, which is a new benefits program which will enhance the success of publishers with transactable offers in the commercial marketplace. Through the program publishers can unlock sales, marketing, and technical benefits to help accelerate their success. As a publisher’s business grows they’ll continue to unlock more benefits designed to provide support at every stage of their growth. This comes with a new badging program for Microsoft AppSource and Azure Marketplace that will quickly direct customers to partner solutions they can trust, which will work with cloud services from Microsoft. We will be publishing additional details on the program next week during Microsoft Inspire.

With these capabilities, publishers will be able to create new revenue streams, reach new customers in new markets, and grow their business faster than ever before.

Next steps

Learn more about how to onboard and publish your offers at Partner Center, how to list them on Microsoft AppSource and Azure Marketplace, and how to take advantage of the new go-to-market services and onboarding resources.

Visit the Microsoft Inspire site, which will be updated with materials, photos, and keynote replays for more highlights from the event.

12 Jul 2019
By editor
Categories: Announcements, Azure, Data Science

Announcing preview of Azure Data Share

In a world where data volume, variety, and type are exponentially growing, organizations need to collaborate with data of any size and shape. In many cases data is at its most powerful when it can be shared and combined with data that resides outside organizational boundaries with business partners and third parties. For customers, sharing this data in a simple and governed way is challenging. Common data sharing approaches using file transfer protocol (FTP) or web APIs tend to be bespoke development and require infrastructure to manage. These tools do not provide the security or governance required to meet enterprise standards, and they often are not suitable for sharing large datasets. To enable enterprise collaboration, we are excited to unveil Azure Data Share Preview, a new data service for sharing data across organizations.

Simple and safe data sharing

Data professionals in the enterprise can now use Azure Data Share to easily and safely share big data with external organizations in Azure Blob Storage and Azure Data Lake Storage. New services will continue to come online. As a fully managed Azure service, Azure Data Share does not require infrastructure to set up and it scales to meet big data sharing demands. The intuitive interface makes sharing easy and productive, directly from the Azure portal. With just a few clicks data professionals choose which data to share and who to share it with. They can schedule the service to automatically share new or changed data pertaining to specific datasets, as well as stop future updates from flowing through at any time. With Azure Data Share, data professionals have greater control over each data sharing relationship and can govern use by associating term of use with each data share created. To receive the data, recipients must agree to the terms of use specified.

Alongside governance, security is fundamental in Azure Data Share and leverages core Azure security measures to help protect the data.

Enabling data collaboration

Azure Data Share maximizes access to simple and safe data sharing for organizations in many industries. For example, retailers can leverage Azure Data Share to easily share sales inventory and demographic data for demand forecasting and price optimization with their suppliers.

In the finance industry, Microsoft collaborated with Finastra, a multi-billion dollar company and provider of the broadest portfolio of financial services software in the world today that spans retail banking, transaction banking, lending, and treasury and capital markets. Finastra is fully integrating Azure Data Share with their open platform, FusionFabric.cloud, to enable seamless distribution of premium datasets to a wider ecosystem of application developers across the FinTech value chain. These datasets have been curated by Finastra over several years, and by leveraging the data distribution capabilities of Azure Data Share, ingestion by app developers and other partners requires simple wrangling, significantly reducing the go to market timeframe and unlocking net new revenue potential for Finastra.

“Our decision to integrate Azure Data Share with Finastra’s FusionFabric.cloud platform is now a great way to further accelerate innovation via an expanded open ecosystem. Our partnership with Microsoft truly provides us with limitless opportunities to drive transformation in Financial Services.”

– Eli Rosner, Chief Product and Technology Officer, Finastra

Next steps

Industries of all types need a simple and safe way to share data. Azure Data Share opens up new opportunities for innovation and insights to drive greater business impact.

Watch the video about Azure Data Share.
Get started with documentation.
Start using Azure Data Share in the Azure portal.

11 Jul 2019
By editor
Categories: Azure, IT Pro, Management

Two ways to share Azure Advisor recommendations

If your IT organization is like most, you probably work with many different people across many different teams. When it comes to common IT tasks like optimizing your cloud workloads, you might need to interact with several resource owners or even complete a formal review process.

That’s why with Azure Advisor, we’ve made it easy to share recommendations with other people across your teams so you can follow best practices that help you get the most out of Azure. Advisor is a free Azure service that helps you optimize your Azure resources for high availability, security, performance, and cost by providing personalized recommendations based on your usage and configurations.

Here are two ways you can share your Advisor best practice recommendations with your teams.

1. Export a PDF or CSV of your Advisor recommendations

Probably the simplest way to share your Advisor recommendations is by exporting an Advisor recommendation report as a PDF or CSV through the Advisor UI in the Azure portal.

This report shows a summary of your Advisor recommendations by category, subscription, and potential business impact. Then you can easily share it with other teams so the resource owners can take action and optimize their resources for high availability, security, performance, and cost.

If you want to provide a specific view of a subset of your recommendations, you can use the UI filters or drill down into specific categories and recommendations. The recommendation report will only contain what you see on the screen when you generate it, which can help you focus on the most critical optimizations.

2. Use the Advisor API to integrate with your ticketing system or dashboards

The other way to share your Advisor recommendations with other people in your organization is via the Advisor REST API. Using this API, you can connect Advisor with your organization’s ticketing system and assign remediation work, set up an internal working dashboard your teams can review and action, or leverage Advisor’s recommendation data any way you choose.

The visual above shows just one way you can use the Advisor API with your ticketing application to share Advisor recommendations with your teams. Some setup is required, but once this scenario is complete, you can start remediating your recommendations more programmatically which will save you time as you optimize your resources.

This more advanced approach tends to work best for larger organizations, organizations managing a large number of Azure subscriptions and resources that are generating a large number of recommendations, and organizations that have a fairly sophisticated IT practice in place, since it scales well with the size of your deployments.

Visit the Advisor API documentation to learn more.

Get started with Advisor

Visit Advisor in the Azure portal to get started reviewing, sharing, and remediating your recommendations. For more in-depth guidance, visit the documentation. Let us know if you have a suggestion for Advisor by submitting an idea to the Azure Advisor feedback forum.

11 Jul 2019
By editor
Categories: Artificial Intelligence, Azure

Analyze AI enriched content with Azure Search’s knowledge store

Through integration with Cognitive Services APIs, Azure Search has long had the ability to extract text and structure from images and unstructured content. Until recently, this capability was used exclusively in full text search scenarios, exemplified in demos like the JFK files which analyzes diverse content in JPEGs and makes it available for online search. The journey from visual unstructured content, to searchable structured content is enabled by a feature called cognitive search. This capability in Azure Search is now extended with the addition of a knowledge store that saves enrichments for further exploration and analysis beyond search itself.

The knowledge store feature of Azure Search, available in preview, refers to a persistence layer in cognitive search that describes a physical expression of documents created through AI enrichments. Enriched documents are projected into tables or hierarchical JSON, which you can explore using any client app that is able to access Azure Storage. In Azure Search itself, you define the physical expression or shape of the projections in the knowledge store settings within your skillset.

Customers are using a knowledge store (preview) in diverse ways, such as to validate the structure and accuracy of enrichments, generate training data for AI models, and ad-hoc analysis of their data.

For example, the Metropolitan Museum of Art opened access to all images of public domain works in its collection. Enriching the artworks with cognitive search and the knowledge store allowed us to explore the latent relationships within the artworks on different dimensions like time and geography. Questions like how have images of family groups changed over time, or when were domestic animals included in paintings, are now answerable when you are able to identify, extract, and save the information in a knowledge store (preview).

With the knowledge store, anyone with an Azure subscription can apply AI to find patterns, insights, or create dashboards over previously inaccessible content.

What is the knowledge store (preview)?

Cognitive search is the enrichment of documents with AI skills before they are added to your search index. The knowledge store allows you to project the already enriched documents as objects (blobs) in JSON format or tabular data in table storage.

As part of your projection, you can shape the enriched document to meet your needs. This ensures that the projected data aligns with your intended use.

When using tabular projections, a knowledge store (preview) can project your documents to multiple tables while preserving the relationships between the data projected across tables. The knowledge store has several other features like allowing you to save multiple unrelated projections of your data. You can find more information about a knowledge store (preview) in the overview documentation.

Data visualization and analytics

Search enables you to find relevant documents, but when you’re looking to explore your data for corpus wide aggregations or want to visualize changes over time you need your data represented in a form other than a search index.

Leveraging Power BI’s integration with Azure tables, gets your dashboard started with only a few clicks. To identify insights from the enriched documents over dimensions like time or space, simply project your enriched documents into tables, validate that Power BI recognizes the relationships and you should now have your data in a format that is ready to consume within the visuals.

When you create a visual, any filters work, even when your data spans related tables. As an example, the art dashboard was created on the open access data from the MET in the knowledge store and the Art Explorer site uses the search index generated from the same set of enrichments.

The art explorer site allows you to find art works and related works while the Power BI report gives you a visual representation of the corpus and allows you to slice your data along different dimensions. You now can answer questions like “How does body armor evolve over time?”

In this example, a knowledge store (preview) enabled us to analyze the data ad-hoc. In another example, we may for instance enrich invoices or business forms, project the structured data to a knowledge store (preview), and then create a business-critical report.

Improving AI models

A knowledge store (preview) can also help improve the cognitive search experience itself as a data source for training AI models deployed as a custom skill within the enrichment pipeline. Customers deploying an AI model as a custom skill can project a slice of the enriched data shaped to be the source of their machine learning (ML) pipelines. A knowledge store (preview) now serves as a validator of the custom skill as well as a source of new data that can be manually labeled to retrain the model. While the enrichment pipeline operates on each document individually, corpus level skills like clustering require a set of documents to act on. A knowledge store (preview) can operate on the entire corpus to further enrich documents with skills like clustering and save the results back in a knowledge store (preview) or update the documents in the index.

Getting started

To start using a knowledge store (preview) you will need to:

Add a knowledge store (preview) configuration to your skillset.
Optionally, add a shaper skill to the skillset to define the shape of the projected enrichment.
Add a projection for tables, objects, or both to a knowledge store (preview). You may project the output of the shaper skill, or elements from the enriched document directly.

A knowledge store (preview) enables the use of your enriched data in new or improved models, visualizing and exploring the data in tools like Power BI and app based experiences merging the raw and enriched data. We will continue to add more capabilities and updates over the coming months.

For a detailed walkthrough, see the knowledge store (preview) getting started guide.