From edge to cloud, a well-orchestrated Data Collection Rule enables telemetry to flow intelligently and reliably
Background
Azure Monitor is Microsoft Azure’s comprehensive monitoring solution that collects, analyzes, and acts on telemetry data from both cloud infrastructure and applications. It provides real-time insights through metrics, logs, and alerts, enabling users to ensure optimal performance while quickly detecting and resolving issues.
Within Azure Monitor, I led the design of a product called Data Collection Rules (DCR), a foundational component that defines what data such as syslogs, events, and metrics is collected from various Azure resources, transforms it to meet users' requirements, and sends to destinations like Log Analytics, Azure Monitor Workspace, and Microsoft Fabric, etc. DCR supports over 50,000 monthly active users and plays a critical role in helping customers streamline their observability workflows.
Problem statement
Data Collection Rules (DCR) are often the first step for users to customize data collection and configure data flow, and most of users begin from the resource itself, either a single resource or a group. Study shows 45% of over 50,000 monthly active users start with the "Resources" entry point.
However, prior to this project:
There is no built-in support for scaling across multiple resources.
The configuration experience was confusing and lack of guidance.
Due to technical limits, some popular data types and destinations are not supported.
Project goals
Empower users to configure data collection across multiple resources simultaneously
Streamline the configuration process, ensuring it is intuitive and accessible for users with different level of expertise.
Expand DCR’s capabilities to include widely used data types and destinations
My focus
This project showcases my ability to simplify technical complexity for users with varying levels of expertise. Designing for different types of telemetry data and destinations required careful consideration of data structure, flexibility, and scalability. As I shaped the end-to-end experience from collecting and mapping to transforming and routing data, I focused on making the workflows intuitive and extensible. Because DCR is a core component of Azure Monitor, my work also needed to support system-wide data flow and integration across the broader monitoring ecosystem.
Here are my 4 main areas of focus on the project:
Experience Design: Created clear, low-friction workflows and user interfaces to guide users through configuring complex data collection rules. Built and tested interactive prototypes to validate concepts and refine the experience.
User Research: Collaborated with a researcher to conduct interviews, extract actionable insights from a highly technical audience, and translate pain points into design decisions that shaped product direction.
Product & System Thinking: With 50,000+ users, I aligned solutions with business goals and user value. I developed technical fluency to work with engineers, design balanced solutions within constraints alongside engineers.
Collaboration: Used low-fidelity mockups to lead cross-functional workshops and design reviews, helping teams align goals, surface technical considerations early, and shape the product direction from the start.
Design Examples
Resource-Centric View: Shows whether a resource has any associated DCRs and how many, providing clear visibility into data collection setup.
In contrast to the current flow, where users only see a list of DCRs without visibility into which resources they are associated with, the new resource-centric view allows users to access information from a resource perspective. They can initiate the data collection after selecting resources.
Users can filter the resource-centric view based on various parameters such as DCR status, subscription, resource group, destination types, resource types, etc. This empowers users to efficiently manage the list, even they have lots of resources.
Introduced a new capability that presents three options based on users’ selections, optimized for the most common use cases to reduce effort.
Users are presented with a curated set of options tailored to their needs. They can quickly and easily set up a data pipeline to initiate the monitoring process.
Users also have flexibility to create DCRs from scratch if they prefer not to use the curated options.
Introduced a new capability that allows users to associate with an existing DCR, eliminating the need to reconfigure details and helping reduce costs.
Users are given the option to view DCRs that match all selected resources or just a subset. A warning is displayed when only partial compatibility exists, helping users make confident, informed choices.
Since creation and association are the most frequent DCR-related actions, we tested multiple design variations with users to gather feedback and ensure the flows were both usable and effective.
Outcome
The work I contributed directly resulted in improved customer satisfaction (CSAT) and revenue growth for Azure Monitor. Since its launch:
Over 30,000 of the 50,000 monthly active users (MAU) have switched to the new experience, and the adoption rate continues to grow.
Monthly support cases related to DCR dropped by approximately 30%, decreasing from 200–250 to around 170, reflecting improved usability and reduced user friction.
Here are some highlights from user feedback:
“If I were to start creating data collection rule as I always start from the resources, so first of all, off the back I'm very happy (this design indicates) we are starting from resources and the resource group. So there we we are in agreement, as that is what I'm doing: if you can make it easier for me that's quite good.”
– Systems Admin at Equity Bank
“It's certainly a lot better than what we have at the moment, because at the moment I have to go into each DCR, have a look at the JSON, you know, see where the destinations are, what the transforms are, if there are any and so on.”
-Cyber Security Architect at Becca Consulting
Reflection and next step
Stakeholder management is essential. I learned the importance of documenting leadership feedback carefully to ensure that future iterations address those inputs, avoiding contradictory or repeated feedback.
Building on the successful release of the new DCR experience, I partnered with the product team to define the strategy and vision for monitoring coverage—a capability that had never existed natively at scale in Azure Monitor. Previously, users had to manually build custom dashboards to understand what resources were being monitored and where gaps existed. Our new initiative introduces a scalable, integrated experience that surfaces monitoring coverage out of the box, enabling users to quickly assess their environment and take action. This effort has significantly expanded the scope of my work and elevated the strategic impact of Azure Monitor as a platform. The feature is currently in development and is slated for introduction at Microsoft Ignite 2025 in November.