Fabric Sprawl - Techniques to Optimize Your Fabric Capacity

Description

Learn how to address a sprawling Fabric environment with ungoverned items across multiple workspaces and capacities. Extend FUAM to gain valuable insights that drive decision-making and backlog prioritization. Design the right balance between autonomy and security in your Fabric environment to deliver an optimal experience for both end users and developers.

Key Takeaways

My Notes

Action Items

Slides

📥 Download Slides

Fabric Sprawl
Curtis Stallings
Senior Architect, Baker Tilly US
GitHub - Curts0
Marcus Radue
Senior Architect, Baker Tilly US
https://mrdata.pro
All code shown today
is available on GitHub!
Session Topics
•What is Fabric Sprawl?
•What are the symptoms?
•What are the benefits of
controlling Fabric Sprawl?
•How do you solve for Fabric
Sprawl?
Know Remove Improve
• Know
• FUAM & its extensions
• Remove
• Approach and strategies
• Improve




Artifact consolidation
Security cleanup
CICD standardization
Monitor
What is Fabric Sprawl?
The uncontrolled growth of Fabric artifacts.
What does Fabric Sprawl look like in your Org?
Some look
like this…
Or this…
Or this…
Or this.
Sprawl Symptoms
Duplication of data assets
Accuracy issues
Capacity peaking
It could be worse! Better to have
sprawl inside of Fabric vs. outside…
Security violations
Redundant build efforts
Benefits of Sprawl Removal
Reduce Fabric Spend
Repurpose Fabric
Compute
Resource
Optimization
Streamline Data
Governance
Fabric Sprawl Resolution
Know
Remove
Improve
Know
FUAM - Fabric Unified Admin Monitoring
• An open-source accelerator in
Microsoft's GitHub repository.
• Completely Fabric.
• Data collection of various Fabric
APIs.
• Stores in Fabric Lakehouse.
• Easily installed and extendable.
Different Ways to Extend FUAM
• Add Personal Workspace Data
• Include Semantic Model Metadata
• Perform Lineage Analysis on Fabric
Items
See FUAM documentation on
updating.
Modifications to Existing FUAM
Pipelines & Tenant Setting Changes
Semantic Model
Metadata Example:

  1. Objects identified and added to the
    backlog
    • Backlog prioritized
  2. Users notified (Email, Teams, Office
    Hours)
  3. Access removed
    • Monitor feedback
  4. Archive objects
    • Semantic Link Labs script
  5. Delete objects
    Semantic Link Labs
    combined with data
    collected by FUAM allows
    for easy automated
    archiving.
    Improve
    Fabric Sprawl Prevention
    Artifact
    Consolidation
    Security
    Cleanup
    CI/CD
    Implementation
    Monitor
    Artifact Consolidation
    Collaborate with Data Governance
    Utilize Fabric Tags for archival process
    Promote and Certify content through Fabric Endorsements
    Domain Based Governance
    • Identify data governance roles
    • Executive Sponsor, Domain Owners, Data Owners,
    Data Stewards
    • Select a data governance platform
    • Microsoft Purview
    • Scan, collect, curate, and monitor data assets
    Endorsements
    • Leverage Fabric endorsement labels to make
    data trust visible and actionable
    • Business domain self-service data assets are
    promoted
    • IT created data assets are certified
    • “Golden” record data assets are labeled master
    data
    • Align endorsement with data governance
    processes
    Security Group Cleanup
    Fabric Permission Levels
    Workspace Permissions
    • Evaluate Fabric permissions for coarse-grain
    workspace access




    • Evaluate item permissions for coarse-grain item
    access
    • Evaluate OneLake RBAC permissions for finegrain folder access
    Admin
    Contributor
    Member
    Viewer
    Item Permissions
    (Permission options vary per item)






    • Consider trusted workspace access and private
    endpoints for increased network security
    Semantic model
    Warehouse
    Data Factory
    Lakehouse
    Data science
    Real-Time Intelligence
    OneLake Permissions
    (Data Access Roles)
    • RBAC model
    Fabric as a Product
    Tighter Controls
    Push down content
    Reporting platforms
    Certified
    Push up content
    VS Code + Git Integration
    Department
    Knowledge Management
    Information sharing
    Promoted
    Team
    Business developed and maintained
    Enterprise
    More creators, fewer consumers
    Permanent record
    Subject to compliance
    Change Management
    Fewer creators, more consumers
    IT – developed and maintained
    Fabric Platform
    Individual
    Personal / private views
    Data science exploration
    Looser Controls
    CICD Implementation
    Enterprise IT
    Citizen Developer
    Semantic Model CICD Example
    Audit & Monitor
    • Fabric Unified Admin Monitoring (FUAM)
    • Easy install
    • Extend functionality
    • Analyze and create actionable insights
    • Fabric Workspace Monitoring
    • High-cost entry point (Fabric CUs)
    • Low development time
    • Capture high granularity logging events
    • Log Analytics
    • Low-cost entry point
    • Higher development time
    • Capture high granularity logging events
    Azure Log Analytics
    “How can I keep track of the overall performance of all these visuals when so many users are building out their own content?”
    Azure Log Analytics
    “I want to remove this column, is it being used?”
    Azure Log Analytics
    “Did I refresh a blank partition? How long are my partitions taking to refresh?”
    Azure Log Analytics
    “My processing is slow; how does my encoding look?”
    Azure Log Analytics
    “I got 99 problems, and a dupe key error is definitely one. Can I see the history of my processing errors?”
    Resources
    • FUAM: fabric-toolbox/monitoring/fabric-unified-admin-monitoring at main · microsoft/fabric-toolbox
    • Code resources: GitHub - Curts0
    Sound off.
    The mic is all yours.
    Influence the product roadmap.
    Join the Fabric User Panel
    Join the SQL User Panel
    Share your feedback directly with our
    Fabric product group and researchers.
    Influence our SQL roadmap and ensure
    it meets your real-life needs
    https://aka.ms/JoinFabricUserPanel
    https://aka.ms/JoinSQLUserPanel