The Agentic Data Estate - Make, Manage, Monitor

#fabcon2026 #session #learning #microsoft #fabcon2026-session

Description

Learn how digital workers, supported by the Agent Framework and digital workforce management best practices, can run your Fabric data estate. Discovery, Observability, Continuous Optimization, Quality & Drift are just some of the key workloads that can be automated using agents.

See how you can identify and implement the digital worker roles which make the most sense for your enterprise.

Key Takeaways

KEY LIMITATIONS
▸ Human-in-the-loop gating at critical
▸ Entra ID managed identities; Key
▸ Data Warehouse with T-SQL +
infer, recommend)
Autonomous agents that monitor data, infer goals, and recommend
authentication, OBO tokens, and user-scoped permissions. Enables

My Notes

Action Items

Resources & Links

Slides

📥 Download Slides

THE AGENTIC
ENTERPRISE DATA ESTATE
Intelligent Automation for Microsoft Fabric, Azure Purview & Beyond
AI Foundry · MS Copilot · Databricks · Power BI
Tom Eastlake & Jason Miles | 2026
Presenters
Tom Eastlake
IBM | Associate Partner | Data & AI | Financial Services
Jason Miles
IBM | Fabric Evangelist | Data & AI | Financial Services
AGENDA
The Enterprise Data Estate
Fabric + Purview + Agentic Framework
Goals of Intelligent Automation
What we're solving for
Current Limitations & Guardrails
What to watch out for
Agentic Framework Components
AI Foundry, MS Copilot
Platform Integrations
Fabric, Purview, Azure, Power BI, Hybrid
Databricks in the Architecture
Where it complements Fabric
Roadmap & Future Capabilities
What's coming that changes everything
Quantum Horizon
IBM & Microsoft quantum vision
GOALS OF INTELLIGENT AUTOMATION
What a properly architected agentic data estate delivers
Reduce Time-to-Visibility
Shift-Left on Governance
Unified metadata, lineage, and DQ scorecards in days —
not months. Continuous discovery replaces manual
cataloging.
Reconcile Purview classifications with Unity Catalog grants
automatically. Detect and fix policy drift before it causes
incidents.
Improve Reliability & Cost
Accelerate MDM Adoption
Proactive drift detection, auto-generated PR fixes, and
optimization guidance across Fabric and Databricks
workloads.
Seeded models, match/merge proposals, golden-record
propagation back to OneLake and semantic models — with
steward approval.
Productized Delivery
Autonomous Operations
MVP in ~90 days with tiered SKUs: Assess, Operate,
Optimize, Master. Fixed-fee MVP plus subscription model.
SLA monitoring, ticket creation with repro steps, runbook
triggers for retry/fail-forward/quarantine — all auditable.
LIMITATIONS & GUARDRAILS
95%
80%
15x
45%
of AI pilots fail
to deliver ROI
encountered risky
agent behaviors
more tokens in
multi-agent systems
cite speed pressure
as governance barrier
KEY LIMITATIONS
▸ Hallucination risk — agents can produce confident but wrong outputs; human validation remains essential for high-stakes actions
▸ State management complexity — race conditions, context leakage, and memory exhaustion in multi-agent workflows
▸ Token cost explosion — multi-agent architectures consume 15x more tokens; budgets and rate limits are non-negotiable
▸ Lineage gaps — bespoke scripts that don't emit lineage create blind spots; require standard transforms or SDK instrumentation
▸ Double-governance risk — conflicting policies between Purview and Unity Catalog require reconciliation agents and clear ownership
REQUIRED GUARDRAILS
Hard stops before publish/execute · Kill switch capability · Deterministic confirmations for irreversible actions · Escalation paths · OpenTelemetry
traces · Rollback triggers
04 | AGENTIC FRAMEWORK COMPONENTS — AI Foundry · Claude Opus · MS Copilot
AGENTIC FRAMEWORK — Component Deep Dive
Azure AI Foundry
Open LLM Options
MS Copilot on Fabric
▸ Cloud-hosted MCP Server for secure tool
orchestration
▸ Foundry Agent Service: persistent agents
with tool catalog
▸ Entra ID (OAuth 2.0 OBO) scoped to user
permissions
▸ Supports 1,400+ business system
connectors
▸ Complex multi-step reasoning for data
estate analysis
▸ Long-context processing for large
metadata graphs
▸ Nuanced natural-language explanations
('why' narratives)
▸ Ideal for Planner/Critic meta-agent
orchestration
▸ In-line SQL completions and KQL query
generation
▸ Notebook code generation and pipeline
troubleshooting
▸ Conversational data exploration across all
workloads
LangGraph / Sem. Kernel
MCP (Model Context
Protocol)
▸ Graph-based agent workflow
orchestration
▸ State management with checkpointing
for replay
▸ Human-in-the-loop gating at critical
decision points
▸ Supports supervisor, pipeline, and fan-out
patterns
▸ Now available on all paid Fabric SKUs
Vector + Graph Stores
▸ Open standard for AI agent → tool
connectivity
▸ Foundry MCP Server: cloud-hosted, public
endpoint
▸ Custom MCP servers via Azure Functions
/ Container Apps
▸ Function calling with tool_choice for
deterministic control
▸ Unified Metadata Graph (UMG) for asset
relationships
▸ Vector embeddings for semantic
similarity search
▸ Knowledge graph traversal for lineage
reasoning
▸ Semantic layer as routing engine for
agent retrieval
PLATFORM INTEGRATIONS
Fabric, Purview, Azure, Power BI — and the broader hybrid estate
Microsoft Fabric
Microsoft Purview
Power BI
Azure & Hybrid
▸ OneLake unified storage
(Delta/Parquet/Iceberg)
▸ Unified Catalog: classifications,
lineage, policies
▸ DirectLake mode for sub-second
queries on OneLake
▸ Entra ID managed identities; Key
Vault for secrets
▸ Data Warehouse with T-SQL +
MERGE support
▸ Data quality scores surfaced in
OneLake catalog
▸ Semantic models enriched with
ML in Fabric
▸ Private endpoints for all data
services
▸ Real-Time Intelligence (KQL,
Eventstream, MQTT)
▸ Sensitivity labels persist across
Fabric workloads
▸ Copilot AI for report generation
and exploration
▸ ADF Self-Hosted IR for onprem/mainframe (CDC, MQ)
▸ Data Factory with 200+
connectors
▸ Governance domains + data
products integration
▸ DQ scorecards, lineage views,
'why' narratives
▸ Mirroring: SQL Server, Oracle,
Snowflake, Cosmos DB
▸ Operations Agents (monitor,
infer, recommend)
▸ Out-of-box governance
dashboard for CDOs
▸ Embedded governance via
Purview classification
▸ OneLake Shortcuts for ADLS, S3,
GCS zero-copy
▸ OneLake Security: row/column
enforcement in preview
▸ Federation with Unity Catalog for
Databricks
▸ Real-time dashboards from
Eventstream data
▸ Event Grid, Logic Apps, Functions
for orchestration
ROADMAP & FUTURE CAPABILITIES
Features on the horizon that will profoundly affect this architecture
OneLake Security GA
Fabric IQ (Intelligence Workload)
Unified row/column security enforced at OneLake level — every Fabric
engine respects the same rules. Currently in preview; expected GA in
2026. Eliminates fragmented security across Spark, SQL, KQL, and
Power BI.
New first-class Fabric citizen bringing AI/ML capabilities natively into
the platform. Expected to become a core workload in 2026, enabling
advanced analytics without leaving the Fabric ecosystem.
Operations Agents
Foundry MCP Server (Cloud)
Autonomous agents that monitor data, infer goals, and recommend
actions — announced at Ignite 2025. Will consume Fabric CU like other
workloads. Represents Fabric's native entry into agentic automation.
Cloud-hosted MCP server at mcp.ai.azure.com with Entra ID
authentication, OBO tokens, and user-scoped permissions. Enables
enterprise-grade multi-agent tool orchestration without local
infrastructure.
Osmos Acquisition
Spark Autoscale Billing
Microsoft acquired Osmos, an agentic AI data engineering platform.
Integration into Fabric will accelerate autonomous data preparation
and transformation workflows — turning OneLake data into analyticsready assets faster.
Pay-as-you-go model for Spark workloads prevents capacity starvation.
Heavy Spark jobs burst beyond capacity allocation, protecting other
workloads. Game-changer for cost management.
THE FUTURE IS
INTELLIGENT & AUTONOMOUS
Digital workers continuously discover, assess, govern, optimize, and remediate your data estate —
with human oversight at every critical juncture.
Start with ONE use case, ONE workflow — validate in 90 days
Build evaluation infrastructure and guardrails FIRST
Use Fabric as the estate manager; Databricks for specialized ML
Begin post-quantum cryptography planning NOW
Treat agents like new team members who learn and improve over time
Sound off.
The mic is all yours.
Influence the product roadmap.
Join the Fabric User Panel
Join the SQL User Panel
Share your feedback directly with our
Fabric product group and researchers.
Influence our SQL roadmap and ensure
it meets your real-life needs
https://aka.ms/JoinFabricUserPanel
https://aka.ms/JoinSQLUserPanel