The official Azure documentation often takes a siloed approach and misses out more advanced Big Data / Machine Learning end-to-end scenarios. Here is a collection of awesome blog posts on Azure Databricks, Azure Data Factory, Azure Data Lake and other related topics. Understand the bigger picture, build configurable end-to-end pipelines, automate deployment, set up accesses and permissions.

I collected links that helped me move forward with data platform implementation. Some of the recommendations and instructions might not be longer valid, as vendors push the boundaries with new functionalities all the time. Hence, don’t copy the provided code or solutions blindly! Use it as inspiration, lighthouse to figure out best solution for you!

Understand the Bigger Picture

• How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh by Zhamak Dehghani
• Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics by Databricks
• Modern Cloud Data Platform for Dummies by Databricks
• Building a Modern Data Warehouse by James Serra
• Architecting Data Lakes (book) by Alice LaPlante & Ben Sharma
• Data Lake Use Cases and Planning Considerations by Melissa Coates
• Azure Data Platform COTS ecosystem by Valdas Maksimavicius
• Modern Data Architectures by Blue Granite
• ABN AMRO’s data and integration mesh by Piethein Strengholt
• The Professional Data Science Manifesto

Intro to Azure Data Platform

• Advancing Analytics YouTube Channel
• Azure Data Platform End2End Labs by Fabio Braga
• Azure Synapse Analytics Overview by James Serra
• An Architects Guide to Delivering Data Insights Using The Microsoft Data Platform by Paul Andrew
• Azure Enterprise Ready Analytics Architecture by Microsoft
• Tips for Bulletproof Data Platform in Azure by Valdas Maksimavicius
• Zones in a Data Lake by Melissa Coates
• Planning for Accounts, Containers, and File Systems for Your Data Lake in Azure Storage by Melissa Coates
• The Hitchhiker’s Guide to the Data Lake

Build Pipelines with Azure Data Factory

• Official Azure Data Factory documentation by Microsoft
• A Complete Introduction to Azure Data Factory by Paul Andrew
• Best Practices For Implementing Azure Data Factory by Paul Andrew
• Series: Beginner’s Guide to Azure Data Factory by Cathrine Wilhelmsen
• Azure Data Factory Framework– EXECUTION RESTARTABILITY
• Cross data factory orchestration
• Set up automated testing for Azure Data Factory

Azure Databricks

• Official Azure Databricks documentation by Databricks
• The Data Engineer’s Guide to Apache Spark by Databricks
• Official Databricks Delta documentation by Databricks
• Azure Databricks Best Practices by Dhruv Kumar and Premal Shah
• Azure Databricks. The Blog of 60 questions by Terry McCann & Simon Whiteley – a set of FAQ. I was positively surprised by answers and wide range of topics covered
• Azure Databricks – Curated by Naveed Hussain – great slides on Azure Databricks
• How do You Size Your Azure Databricks Clusters? by Terry McCann
• Cluster Management Best Practices by Databricks – how to split interactive and production clusters
• 20 rules for Azure Databricks in production by Eimantas Jazukevicius

Azure Purview

• Azure Purview documentation by Microsoft
• Data Lineage from Databricks to Azure Purview
• Exploring Purview’s REST API with Synapse by Raki Rahman
• Azure Purview Overview by Taygan Rifat
• Purview Python management package and pyapacheatlas

DataOps blog posts

• The DataOps Manifesto
• Azure DataOps Sample by Microsoft
• Showcase Dev Ops for ML practices with Azure DataBricks by David Crook – scripts and step by step tutorial
• DevOps In Azure With Databricks And Data Factory by Alexandre Gattiker – sample project walkthrough for Azure Databricks, Azure Data Factory and Azure DevOps (does not include ML)
• CI/CD with Databricks and Azure DevOps by Benjamin Leroux – execute a Databricks notebook, push changes to production upon successful execution and approval by a stage pre-deployment approval process
• Running an Azure Databricks notebook in a CI/CD release stage by Benjamin Leroux – separate Databricks workspaces and move notebooks automatically
• Social Stream Pipeline on Databricks with auto-scaling and CI/CD using Travis by Mor Shemesh
• How to bring your Data Science Project in production by René Bremer – Using Azure Databricks with Spark, Azure Machine Learning Service and Azure DevOps
• Sample notebooks for deploying a Databricks model in an Azure Function by Alexandre Gattiker – demonstrate training a machine learning model and operationalizing it in a Java Azure Function for online prediction
• Azure DevOps CI/CD with Azure Databricks and Data Factory by Beda Tse
• Continuous Integration & Continuous Delivery with Databricks by Databricks
• Understanding MLOps with Azure Databricks by Raki Rahman

Security blog posts

• Data Professional Security Checklist by Valdas Maksimavicius
• Access Azure Data Lake Storage using Azure Active Directory credential passthrough by Microsoft
• Access control in Azure Data Lake Storage Gen2 by Microsoft
• Set Permissions on a Data Object/Tables in Azure Databricks by Databricks
• Data Exfiltration Protection with Azure Databricks
• Data-Level Security In Azure Databricks by Alexandre Gattiker – use Table Access Control and Credential Passthrough to secure your data
• Implementing Enterprise Security in Azure Databricks Part 1 and Part 2 by Tristan Robinson – great blog post on users, accesses & naming
• Azure Data Lake Storage Gen 2: Exploring AAD B2B & ACL Hardening by Karim Vaes
• Different ways to connect to storage in Azure Databricks by Lace Lofranco
• Create and configure a self-hosted Azure Data Factory integration runtime by Microsoft – use self-hosted virtual machine to force Azure Data Factory to use a virtual network

Interested in Microsoft’s Data & AI? Subscribe to the below blogs

• James Serra’s Blog by James Serra (Big Data and Data Warehousing Solution Architect at Microsoft)
• Advancing Analytics and Advancing Analytics YouTube Channel by Terry McCann & Simon Whiteley 
• Cathrine Wilhelmsen’s Blog
• Piethein Strengholt’s books and articles
• Raki Rahman’s Blog
• Mr Paul Andrew by Paul Andrew
• Cloud Architected by Alexandre Gattiker (Cloud Solution Architect for Data & AI at Microsoft)
• Modern Data Engineering by Sean Forgatch (Senior Consultant at Avanade)
• Arvind Shyamsundar’s Technical Blog by Arvind Shyamsundar (Principal Program Manager with the Microsoft Azure / Data Customer Advisory Team)
• Delora Bradish Blog by Delora Bradish (Pragmatic Works Senior Consultant specializing in SSIS, SSAS, SSRS, Power BI, SQL Server database)
• Modern Data & AI Blog by Dave Ruijter (Principal Consultant Data & A.I) and Henko Kors (Azure Solution Architect)
• René Bremer in Towards Data Science by René Bremer (Data Solution Architect at Microsoft)
• SQL Chick by Melissa Coates (Principal Architect at BlueGranite)
• The Data Guy by Benjamin Leroux (Data Platform Solution Architect at Microsoft)
• Karim Vaes’ Blog by Karim Vaes (Cloud Solution Architect)
• MSSQLDUDE Blog by Mark Kromer
• DataHelge by Helge Rege Gårdsvoll
• Data Platform School by Valdas Maksimavicius