Build Your AI Foundation with Fabric Metadata-driven Framework

#fabcon2026 #session #learning #microsoft #fabcon2026-session

Description

Ready to learn how to build your AI foundation? Discover how Microsoft Fabric’s metadata-driven framework, combined with Data Factory and a Fabric database, powers automated data integration for AI-ready environments. Gain insights on governance, scalability, and accelerating AI-driven analytics.

Key Takeaways

What you tested last year… last month… even last week…, is already outdated.
AI doesn’t fail because the model is weak.
AI fails because the data platform is.
Rapid implementation.
Limited customization.
Lower development effort.
Ongoing support and updates.

My Notes

Action Items

Resources & Links

Slides

📥 Download Slides

Build Your AI
Foundation with Fabric
Metadata-driven
Framework
@erwindekreuk.bsky.social
linkedin.com/in/erwindekreuk
erwindekreuk.com
github.com/edkreuk
Erwin de Kreuk
Technology Lead Data, InSpark, Netherlands
https://sessionize.com/erwin-de-kreuk/
Dutchfabricusergroup.com
AI Reality Check: Why Your Results Lag Behind
If you’re still saying “AI wasn’t that good when I tried it…”
This is your wake-up call.
AI Has Accelerated Exponentially
• What you tested last year… last month… even last week…, is already outdated.
The Real Issue Isn’t AI ,It’s the Data Foundation
AI doesn’t fail because the model is weak.
AI fails because the data platform is.
AI Reality Check: Why Your Results Lag Behind
Modern AI Needs a Modern Data Foundation
Fabric-native. Unified. Governed. Real-time. AI-ready.
Agenda
Agenda
Out-of-the-Box Framework
• Ready-to-use.
• Rapid implementation.
• Limited customization.
• Lower development effort.
• Lower upfront costs.
• Ongoing support and updates.
Custom-Made Framework
• Tailored to specific needs.
• Full control over design and features.
• Higher development effort.
• Flexibility and extensibility.
• Higher upfront costs.
Custom-Made Framework
• Based on parameters
• Metadata => Fabric SQL Database /
Json / ……
• Microsoft Fabric but also on Azure
Synapse Analytics and Azure Data
Factory
• Based on a Uniform Data
Architecture
What about the Third option
•Fabric Native Solution
Data
Factory
Analytics
Databases
Real-Time
Intelligence
IQ
Power BI
Fabric Platform
Copilot
OneLake
Governance
Data
Factory
Analytics
Databases
Real-Time
Intelligence
IQ
Power BI
Fabric Platform
Copilot
OneLake
Governance
Fabric Metadata Driven Framework
• Meets most of your needs
• Full control over design and features.
• Less development effort.
• Flexibility and extensibility.
• Lower upfront costs.
• 100 % Fabric Native Solution
Fabric Metadata Driven Framework
Modern data platforms demand agility, scalability, and consistency. FMD
simplifies these challenges by enabling:
• Dynamic, metadata-driven pipelines
• Consistent orchestration across ingestion, processing, and publishing
• Centralized configuration for all data entities
• Alignment with Microsoft Fabric Lakehouse & Medallion Architecture
• Reduced engineering effort through reusable patterns
• Faster delivery with standardized, tested components
Parameters
ACTIVITIES
PIPELINES
NOTEBOOKS
TEMPLATES
Fabric Metadata Driven Framework
• Download Notebook from Github
Fabric Metadata Driven Framework
• Download Notebook from Github
• Create Workspace Name is up to you:
• FMD_XXX_CONFIGURATION
• Assign a capacity
• Import Notebook to this Workspace
Fabric Metadata Driven Framework
• Open Notebook
• NB_SETUP_FMD.ipynb
A strong platform enables speed,
reliability, and innovation at scale.
Data Platform Engineering
• Azure subscription
• Fabric Capacity (Trial)
• Key Vault optional
• Fabric Administrator role
Data Platform Engineering
• Workspace DATA:
• Lakehouses
Data Platform Engineering
• Workspace CODE
• Data Pipelines
• Notebooks
• Variable Libraries
Data Platform Engineering
• Workspace CONFIG
• Fabric database
• Metadata database
• Environments
Item deployment
• Lakehouse
• Medallion Architecture
• Data Landingzone
• Bronze
• Silver
Item deployment
• Variable Library
Item deployment
• Variable Library
• Environment
Item deployment
• Variable Library
• Environment
• Data Pipelines
• Notebooks
Item deployment
• Variable Library
• Environment
• Data Pipelines
• Notebooks
• Fabric Database
Unified data is created at ingestion, not
at the end.
Data Integration
• Fabric database for metadata
Data Integration
• Fabric database for metadata
• Integration
• Tables
• Views
• Stored Procedures
Data Integration
• Execution
• Tables
• Views
• Stored Procedures
• Logging
• Tables
• Views
• Stored Procedures
Data Sources
• SQL Server
• Data Lake Gen 2
• Onelake Tables
• Onelake Files
• FTP / SFTP
• Oracle
• Custom Notebooks(API)
• ADF (Connect to ADF pipeline)
All other connections can easily be added
Every connection needs a new Data Pipeline
How to get data easily into Fabric Metadata Driven data
model
• Metadata driven ingestion Pipeline
• Manual
Metadata driven ingestion Pipeline
• ConnectionGuid
• DatasourceName
• Namespace
• DatasourceType
• Tables(schema+table
Metadata driven ingestion Pipeline
How to add a new connection and entity
Landingzone Entity Table
• Source Schema
• Source Name
• SourceCustomSelect
• File Type
• IsIncremental
• Incremental Column
BronzeLayer Entity Table
• Schema
• Name
• Primarykeys
• CleansingRules
Silver Layer Entity Table
• Schema
• Name
• CleansingRules
Processing is where data turns into
value.
Data Processing ( source to Data Landingzone )
Data Processing (Data LDZ to Bronze/Silver )
Data Processing
• Processing at Scale (Run Multiple)
• Data Quality
• Data Cleansing
• Queing
Data Processing Cleansing rules (BRZ/SLV)
• NB_FMD_DQ_CLEANSING
• Built in Rules
• normalize_text
• split
• fill_nulls
• parse_datetime
Data Cleansing · edkreuk/FMD_FRAMEWORK Wiki
df_norm = normalize_text(
df,
columns=["title"],
args={"case": "lower", "empty_as_null": False}
)
df_norm = normalize_text(
df,
columns=["name"],
args={"case": "title", "collapse_spaces": False}
)
You can’t scale what you can’t see.
Data Observability
• Log Start and End Time of records
• Log Extracted Records
• Log Execution Failure
Logging START
Logging END
Logging Fail
PIPELINE ACTIVITY
COPY ACTIVITY
NOTEBOOK
NOTEBOOK ACTIVITY
Logging
Data Observability
• Add Information about pipelines
• Adding System Variables
Data Observability
• Add Information about pipelines
• Adding System Variables
• Add Information about Notebooks
Data Observability
Conclusion
Out of the Box
Framework
Custom Made
Framework
Fabric Metadatadriven Framework
Ready to use
Rapid Implementation
Lower development effort
Lower upfront cost
Support and Updated
Tailored to specific needs.
Full control over design and features.
Easy to extend.
Your AI Foundation with Fabric is now Ready
• AI-ready data foundation built fully on Fabric, leveraging metadata-driven design for consistency, automation
and reuse.
• Standardized ingestion and processing patterns ensure reliable, repeatable data flows that accelerate ML
and AI development.
• Centralized metadata model enables dynamic configuration, reducing engineering overhead and enabling
scalable AI workloads.
• Medallion Architecture provides clean, structured, high-quality data pipelines ideal for model training, RAG
patterns, and generative AI scenarios.
• Unified Data Platform engineering workspace (notebooks, pipelines, Fabric databases, variables)
streamlines data prep, feature engineering, and experimentation.
• End-to-end observability ensures full logging and monitoring, critical for AI governance, model audits, and
responsible AI frameworks.
• Native Fabric capabilities like OneLake, Real-Time Intelligence, Data Factory, and Copilot create an
integrated platform for building and operationalizing AI.
• Extensible and future-proof foundation positioned to support enterprise-scale AI, including automation,
predictive analytics, and generative AI workloads.
FMD Business Domain Framework
• A modular extension of the Fabric Metadata-Driven Framework (FMD) designed to simplify,
standardize, and automate deployment of Business Domains within Microsoft Fabric.
• The Business Domain Framework enables organizations to scale analytics through a governed,
metadata-driven approach where each business domain (e.g., Sales, HR, Finance, Operations) is
deployed consistently using reusable patterns, automated provisioning, and ready-to-use template
assets.
Feedback /Ideas
THANK YOU