Runbook Scope and Coverage
Azure PaaS Sources
Azure Storage Accounts, SQL Database, Synapse Analytics, Data Lake Storage, and other native Azure services
On-Premises & Hybrid
SQL Server installations, Oracle databases, network file shares, and hybrid connectivity scenarios
SaaS & External
Databricks workspaces, Azure DevOps, Power BI datasets, and third-party cloud platforms
This runbook applies universally across Development, User Acceptance Testing, and Production environments. Each environment requires careful attention to security controls and governance policies. The procedures outlined herein ensure consistency regardless of the deployment stage, whilst maintaining appropriate separation of duties and access controls. Teams should adapt the specific technical parameters to their environment whilst adhering to the core workflow and validation steps presented throughout this document.
Roles and Responsibilities Matrix
| Role | Key Responsibilities | Required Permissions |
|---|---|---|
| Purview Administrator | Owns the Purview account, manages collections, approves Key Vault connections, and oversees the entire governance framework | Purview Data Source Admin, Collection Admin |
| Data Source Owner | Grants read permissions on the data source, validates business requirements, and ensures data quality standards | Reader, Storage Blob Data Reader |
| Scan Operator / Engineer | Sets up Integration Runtime infrastructure, configures credentials, manages scan schedules and troubleshoots connectivity | Purview Data Reader, Scan Operator |
| Security Administrator | Approves managed private endpoints, configures firewall rules, manages service principals and credential rotation | Key Vault Administrator, Network Contributor |
| Governance Lead | Validates classification accuracy, reviews lineage mappings, determines collection placement and ensures compliance | Purview Data Curator, Policy Author |
Clear delineation of responsibilities prevents configuration drift and ensures accountability throughout the onboarding lifecycle. Each role requires specific technical expertise and organisational authority to execute their duties effectively.
Prerequisites Checklist
Technical Readiness
- Confirm data source type and connection endpoint details
- Identify correct Purview Collection for logical organisation
- Verify data source access permissions for scanning account
- Decide Integration Runtime type based on network topology
- Verify network connectivity paths and requirements
Security & Governance
- Confirm Purview role assignments for team members
- Create or identify Azure Key Vault for credential storage
- Ensure Purview managed identity has Key Vault permissions
- Create Service Principal with least privilege access
- Register credentials securely in Key Vault
Completing this checklist before beginning the technical configuration prevents delays and ensures all stakeholders have provided necessary approvals. Each prerequisite should be documented with the responsible party and completion date. This checklist serves as both a planning tool and an audit trail for governance purposes. Teams should maintain a record of these prerequisites for each data source onboarded, creating a historical reference that aids future troubleshooting and demonstrates compliance with organisational policies.
Integration Runtime Decision Framework
Azure Integration Runtime
Best for: Azure SQL Database, Storage Accounts, Data Lake with public access enabled
Advantages: Simplest setup, no infrastructure management, automatic scaling, Microsoft-managed security
Considerations: Requires public network access to data sources
Managed VNet IR
Best for: Azure resources with private access only, enhanced security requirements
Advantages: Managed private endpoints, no public exposure, Azure-managed infrastructure
Considerations: Requires approval workflow for private endpoints, slightly longer setup time
Self-Hosted IR
Best for: On-premises SQL Server, Oracle databases, file shares, corporate network resources
Advantages: Access to on-premises systems, control over runtime environment
Considerations: Requires Windows VM, ongoing maintenance, manual updates and monitoring
Selecting the appropriate Integration Runtime is critical to successful data source onboarding. The decision impacts security posture, operational overhead, and connectivity reliability. For hybrid scenarios involving both Azure and on-premises resources, organisations may deploy multiple IR types simultaneously. The key consideration is network routing—ensuring the Integration Runtime can reach the data source whilst maintaining security boundaries. Document your IR strategy in your governance framework to ensure consistency across similar data sources.
Step 1: Register Data Source
Navigate to Sources
In Microsoft Purview portal, access Data Map section and select Sources from the navigation menu
Initiate Registration
Click the Register button to begin the data source registration wizard
Select Source Type
Choose the appropriate data source type from the catalogue, such as Azure SQL Database, Storage Account, or on-premises SQL Server
Provide Details
Enter subscription, resource group, resource name, and select the target collection for governance alignment
Complete Registration
Review configuration and click Register to add the source to your Data Map inventory
Data source registration creates the logical representation within Purview but does not yet establish connectivity or initiate scanning. This step is primarily about cataloguing and organising your data estate within the appropriate collection hierarchy. The collection assignment determines access control and governance policies that will apply to discovered assets. Choose collections carefully to align with your organisational structure and data ownership model. Registration can be performed independently of scanning, allowing governance teams to plan their data estate structure before technical implementation begins.
Step 2: Create Key Vault Connection
Configuration Steps
- Navigate to Data Map → Management → Key Vault Connections
- Select "+ New Connection" to begin setup
- Choose your Azure Key Vault from available subscriptions
- Grant Purview managed identity appropriate permissions
- Test connectivity to validate configuration
Required Permissions
The Purview managed identity requires the following Key Vault access policies:
- Get: Retrieve secret values for authentication
- List: Enumerate available secrets for credential configuration
These permissions enable Purview to securely retrieve credentials stored in Key Vault without exposing sensitive information in the Purview interface. Avoid granting broader permissions such as Set, Delete, or Purge unless specifically required by your security policies.
Security Best Practice: Use separate Key Vaults for different environments (Development, UAT, Production) to maintain proper separation of credentials and reduce the risk of credential leakage across environment boundaries.
Key Vault integration is the cornerstone of secure credential management in Purview. By storing credentials in Key Vault rather than directly in Purview, organisations maintain centralised control over sensitive information, enable credential rotation without Purview reconfiguration, and ensure audit trails for all credential access. The connection between Purview and Key Vault uses Azure managed identities, eliminating the need to store additional credentials for this integration itself.
Step 3: Create Scan Credentials
Service Principal
Recommended for most Azure scenarios. Requires Client ID, Tenant ID, and Client Secret stored in Key Vault. Supports certificate-based authentication for enhanced security.
Managed Identity
Ideal for Azure-native resources. Eliminates credential management overhead. Automatically rotates and requires only Reader permissions on target resources.
Username & Password
Necessary for legacy systems and on-premises databases. Store passwords as Key Vault secrets. Implement regular rotation policies to maintain security posture.
Personal Access Token
Required for SaaS platforms like Databricks and Azure DevOps. Generate tokens with minimal scope. Set expiration dates and document renewal procedures.
When creating credentials in Purview, navigate to Data Map → Credentials → "+ New Credential" and select the appropriate authentication method. Reference the Key Vault secret by name rather than entering values directly. Use descriptive naming conventions such as "cred-azsql-prod-spn" to clearly identify the credential type, environment, and authentication method. This naming strategy aids troubleshooting and supports audit requirements. Always apply the principle of least privilege—grant only the minimum permissions necessary for scanning operations, typically Reader and Storage Blob Data Reader for Azure resources.
Step 4: Configure Azure Integration Runtime
Azure Integration Runtime represents the simplest deployment option, requiring no infrastructure management or additional configuration. Microsoft manages the compute resources, scaling, and security updates automatically.
Automatic Provisioning
Azure IR is available by default in every Purview account with no setup required. It operates within Microsoft's managed network infrastructure.
Public Access Requirement
Azure IR can only connect to data sources with public endpoints enabled. Firewall rules must permit access from Azure services.
Optimal Use Cases
Best suited for development environments, proof-of-concept projects, and production sources where public access aligns with security policies.
When configuring scans with Azure IR, ensure your data source firewall settings permit access from Azure services. For Azure Storage Accounts, enable "Allow Azure services on the trusted services list to access this storage account". For Azure SQL Database, add Purview's managed identity to the firewall rules or enable "Allow Azure services and resources to access this server". Azure IR provides the fastest time-to-value for data source onboarding, making it ideal for initial Purview implementations and non-production environments. However, organisations with strict network isolation requirements should consider Managed VNet IR or Self-Hosted IR alternatives.
Step 4: Configure Managed VNet Integration Runtime
Deploy Managed VNet IR
Navigate to Management → Integration Runtimes → "+ New" and select Managed Virtual Network IR option. Provide a descriptive name and select your region.
Create Private Endpoints
Access Data Map → Managed Private Endpoints → "+ Create" and select your target data source. Specify the resource type (SQL, Storage, Synapse).
Approve Endpoints
Navigate to the target resource in Azure Portal → Networking → Private Endpoint Connections. Approve the pending Purview endpoint request.
Verify Status
Return to Purview and confirm the endpoint status shows "Approved". This may take several minutes to update after approval in the Azure Portal.
Managed VNet IR provides enterprise-grade security by establishing private connectivity to Azure resources without exposing them to the public internet. The managed private endpoint approval workflow ensures that data source owners maintain control over inbound connections. This architecture is particularly important for production environments handling sensitive data, regulated industries with compliance requirements, and organisations implementing zero-trust security models. The trade-off for enhanced security is slightly longer initial setup time and the approval workflow dependency. However, once established, Managed VNet IR requires minimal ongoing maintenance as Microsoft manages the underlying infrastructure, security patches, and scaling operations.
Step 4: Configure Self-Hosted Integration Runtime
Installation Requirements
- Windows Server 2016 or later (2019 recommended)
- Minimum 4 CPU cores and 8 GB RAM
- Network connectivity to both Purview and data sources
- Outbound HTTPS (port 443) to Azure
- Administrator rights for installation
Download Installer
In Purview portal, navigate to Management → Integration Runtimes → "+ New" → Self-Hosted. Download the SHIR installer package.
Install on Windows Host
Execute the installer on a Windows machine or VM with network access to your on-premises data sources. Follow the installation wizard.
Register with Purview
Copy the authentication key from Purview portal and paste it into the SHIR configuration manager during registration process.
Verify Connectivity
Ensure outbound HTTPS connectivity to Purview endpoints. Check firewall rules permit required Azure service tags.
Confirm Online Status
Return to Purview portal and verify the SHIR shows "Online" status in the Integration Runtimes list.
Self-Hosted Integration Runtime bridges the gap between Purview's cloud infrastructure and your on-premises data sources. The SHIR acts as a secure gateway, establishing outbound connections to Purview whilst maintaining inbound connectivity to corporate data systems. For high availability, deploy SHIR in a multi-node configuration across separate VMs. This prevents scan failures if a single host becomes unavailable. Monitor SHIR performance metrics and maintain the runtime up-to-date with Microsoft's regular updates. Document your SHIR configuration including hostname, version, and network topology to aid operational support and disaster recovery planning.
Step 5: Create and Configure Scans
Scan configuration represents the final technical step in data source onboarding. Navigate to Data Map → Sources, select your registered data source, and click "+ New Scan" to launch the configuration wizard. The scan setup process involves several critical decisions that impact discovery completeness, performance, and resource consumption.
Basic Configuration
Scan Name: Use descriptive naming such as "prod-azsql-finance-weekly"
Integration Runtime: Select Azure, Managed VNet, or Self-Hosted IR based on earlier decisions
Credential: Choose the appropriate credential created in Step 3
Scope Definition
Database/Container: Specify which databases, schemas, or storage containers to include
Folder Paths: For storage accounts, define specific folder hierarchies to scan
Exclusions: Configure patterns to skip temporary or system tables
Scan Rule Set
Default Rules: Include common file types and metadata extraction patterns
Custom Rules: Define specific file type inclusion/exclusion and column sampling rules
Classification: Enable automatic sensitivity classification and pattern matching
Schedule & Performance
Trigger: Manual, daily, weekly, or custom schedule
Sampling: Percentage of data to sample (100% for complete discovery)
Resource Sets: Enable pattern detection for grouped files
Concurrency: Number of parallel scan operations
After configuration, execute a manual scan to validate connectivity and permissions before enabling scheduled operations. Monitor the initial scan closely to identify any authentication failures, network timeouts, or permission issues. The first scan of a large data source may take considerable time—plan accordingly and avoid scheduling during peak business hours. Document your scan configuration parameters in your governance inventory for future reference and troubleshooting purposes.
Validation and Quality Assurance
Connection Success
All connectivity tests must pass before proceeding to scan configuration
Scan Completion
Target completion rate for initial scans, accounting for expected permission boundaries
Asset Discovery
Verify all expected tables, files, and objects appear in the Data Catalogue
| Validation Step | Expected Result | Owner | Priority |
|---|---|---|---|
| Connection test succeeds | "Connection successful" message displayed | Scan Operator | Critical |
| Scan completes successfully | "Completed" status in scan history without errors | Scan Operator | Critical |
| Assets appear in Catalogue | Tables, files, or objects listed under correct collection | Governance Lead | Critical |
| Classifications applied | Sensitive data columns automatically tagged | Governance Lead | High |
| Lineage visible | End-to-end data flow graph displayed where applicable | Governance Lead | Medium |
| Private endpoint status | All managed endpoints show "Approved" status | Security Admin | Critical |
| Key Vault connection | "Connected" status in Key Vault connections panel | Security Admin | Critical |
Validation must be performed systematically immediately following the initial scan completion. Each validation step confirms a specific aspect of the onboarding process, from basic connectivity through to advanced governance features. Document validation results in your onboarding record, including timestamps, responsible parties, and any issues encountered. Failed validations should trigger the troubleshooting workflow outlined in the next section. Do not proceed to scheduling recurring scans until all critical validation steps pass successfully.
Common Issues and Resolution Strategies
Connection Failed (Network)
Symptoms: "Unable to connect to data source" error during connection test
Root Causes: Firewall rules blocking traffic, private endpoint not approved, incorrect VNet routing
Resolution: Verify outbound/inbound firewall rules, check private endpoint approval status in Azure Portal, validate network security group rules, confirm DNS resolution
Authentication Failed
Symptoms: "401 Unauthorized" or "403 Forbidden" errors in scan logs
Root Causes: Expired credentials, incorrect Key Vault secret reference, missing RBAC permissions
Resolution: Rotate service principal secret in Azure AD and update Key Vault, verify Key Vault connection in Purview, grant appropriate Reader permissions on data source
Scan Hangs or Times Out
Symptoms: Scan status remains "In Progress" indefinitely or shows timeout errors
Root Causes: Self-Hosted IR offline, insufficient compute resources, network latency
Resolution: Restart SHIR service, verify SHIR shows "Online" status, redeploy Managed VNet IR if necessary, reduce scan concurrency settings
No Assets Discovered
Symptoms: Scan completes successfully but catalogue shows no assets
Root Causes: Incorrect scan scope configuration, permissions limited to system objects only
Resolution: Review and expand scan scope to include data objects, verify service principal has access beyond system tables, check exclusion patterns not overly broad
Classifications Not Applied
Symptoms: Assets discovered but no sensitivity labels or classifications appear
Root Causes: Scan rule set misconfigured, classification rules not enabled, sampling percentage too low
Resolution: Apply default system rule set, enable classification rules in scan configuration, increase sampling to 100% for thorough classification
Permission Denied Errors
Symptoms: Scan fails with "Access denied" or "Insufficient permissions" messages
Root Causes: Service principal lacks required roles, managed identity not granted access
Resolution: Grant Reader role on subscription or resource group, add Storage Blob Data Reader for storage accounts, assign db_datareader for SQL databases
Maintain a troubleshooting log documenting each issue encountered during onboarding, the investigation steps taken, and the ultimate resolution. This knowledge base proves invaluable for future onboarding efforts and helps identify systemic issues requiring architectural changes. Many authentication and connectivity issues can be prevented through thorough completion of the prerequisites checklist before beginning technical configuration.
Post-Scan Governance Activities
Immediate Actions
- Review auto-detected classifications for accuracy and completeness
- Approve or override system-suggested sensitivity labels
- Assign data owners and stewards to discovered assets
- Validate asset descriptions and update business glossary terms
- Configure retention policies and lifecycle management rules
Ongoing Operations
- Schedule scan recurrence aligned with data refresh cycles
- Document scan configuration in governance inventory
- Monitor scan success rates and investigate failures
- Review classification rules quarterly for effectiveness
- Update asset metadata as business context evolves
Post-scan governance represents the transition from technical onboarding to operational data management. The Governance Lead should review all discovered assets within 48 hours of initial scan completion to ensure accurate representation in the catalogue. This includes verifying that sensitive data is properly classified, assets are placed in appropriate collections, and business context is captured through descriptions and glossary term associations.
Lineage documentation requires special attention. Whilst Purview automatically detects lineage for many Azure services, complex transformations and external data flows may require manual documentation. Work with data engineering teams to capture these relationships, ensuring the lineage graph provides complete visibility into data movement and transformation across your estate.
Establish a regular cadence for scan execution based on data volatility. Rapidly changing sources may require daily scanning, whilst static reference data can be scanned weekly or monthly. Balance scan frequency against compute costs and operational overhead. Configure alerts to notify appropriate teams of scan failures, ensuring issues are addressed promptly before they impact data discovery or governance processes.
Security and Governance Controls
30 Days
Review managed private endpoints and delete unused connections to reduce attack surface
90 Days
Rotate all service principal secrets and update Key Vault with new credentials
Quarterly
Audit scan credential inventory and review access logs for anomalous activity
Quarterly
Review collection RBAC assignments and remove unnecessary administrative access
Ongoing
Maintain Self-Hosted IR patching schedule and monitor uptime metrics
Compliance Requirement: Many regulatory frameworks mandate regular credential rotation and access reviews. Document your control execution in compliance management systems to demonstrate adherence to organisational security policies and external audit requirements.
Security controls must be embedded into operational procedures rather than treated as one-time setup tasks. Establish automated reminders for credential rotation schedules to prevent lapses in security hygiene. Service principal secrets should never exceed 90-day lifespans, and ideally should be rotated monthly for production systems handling sensitive data. When rotating credentials, update Key Vault secrets first, then verify Purview scans continue to function correctly before removing the old secret entirely.
Collection-level role-based access control requires ongoing attention. The principle of least privilege dictates that users should have access only to the collections and assets necessary for their job functions. Regularly review collection memberships, especially after organisational changes or team restructuring. Remove default administrative assignments and ensure each collection has clearly designated owners who are accountable for governance within their domain. Maintain an access control matrix documenting who can register sources, execute scans, and modify metadata across your Purview instance.
Documentation Deliverables
Data Source Registration Record
Comprehensive inventory including source name, type, collection assignment, Integration Runtime selection, credential identifier, and registration date. Maintain version history to track configuration changes over time.
Scan Configuration Log
Detailed record of scan schedules, rule sets applied, credential references, scope definitions, and performance parameters. Include scan history with success rates and identified issues for trend analysis.
Access Control Matrix
Authoritative source documenting which users and groups can register data sources, execute scans, modify metadata, or view discovered assets. Map permissions to job roles for easier onboarding and offboarding.
Credential Inventory
Secure registry of all credentials including linked Key Vault secret names, authentication methods, assigned data sources, creation dates, expiry dates, and responsible owners. Essential for rotation planning.
Documentation serves multiple critical purposes: operational support, audit compliance, knowledge transfer, and disaster recovery. Each deliverable should be maintained in a centralised location accessible to authorised team members. Version control is essential—track changes to configurations over time to support troubleshooting and understand the evolution of your data governance programme. Consider implementing an automated documentation system that extracts configuration details directly from Purview APIs, ensuring your records remain current as changes are made in the platform.
Onboarding Workflow Summary
Complete Prerequisites
Verify all readiness items, obtain approvals, and prepare infrastructure
Register Data Source
Add source to Data Map with appropriate collection assignment
Configure Security
Establish Key Vault connection, create credentials, and apply least privilege
Deploy Integration Runtime
Set up connectivity infrastructure based on network topology
Create and Execute Scan
Configure scan parameters, validate connectivity, and run initial discovery
Validate Results
Verify asset discovery, classifications, and governance metadata
Document Configuration
Record all settings, credentials, and operational procedures
This workflow represents the complete end-to-end process for onboarding a new data source to Microsoft Purview Data Map. Each step builds upon the previous, and skipping steps or rushing through prerequisites inevitably leads to troubleshooting delays later. For teams onboarding multiple similar data sources, consider creating templates that capture common configuration patterns, reducing setup time whilst maintaining consistency. The entire process typically requires 2-4 hours for straightforward Azure resources, or 1-2 days for complex hybrid environments with extensive security requirements. Plan accordingly when scheduling onboarding activities and communicate timelines clearly to stakeholders expecting immediate data discovery capabilities.
Data Source Decommission Procedure
When a data source reaches end-of-life or is replaced by alternative systems, proper decommissioning ensures clean governance records and prevents orphaned configuration elements. Follow this systematic approach to remove data sources from Purview whilst maintaining audit trails for compliance purposes.
Disable Scheduled Scans
Navigate to the data source in Purview and edit all active scans, changing their schedule from automated to manual. This prevents new scan executions whilst preserving historical scan results for audit purposes. Allow 24 hours to ensure no in-flight scans are interrupted.
Archive Asset Metadata
Export asset metadata, classifications, and lineage information before deletion. This historical record may be required for regulatory compliance or future reference. Store exports in secure archival storage with appropriate retention policies.
Remove Credentials
Delete the associated credential objects from Purview's credential store. Then access Azure Key Vault and delete the corresponding secrets. Verify no other data sources reference these credentials before deletion to prevent unintended disruption.
Delete Private Endpoints
For Managed VNet IR configurations, navigate to Managed Private Endpoints and delete any endpoints created specifically for this data source. In the Azure Portal, also remove the corresponding private endpoint resources from the target subscription.
Decommission Integration Runtime
If using a dedicated Self-Hosted IR, unregister the runtime from Purview, uninstall the software from the Windows host, and decommission the VM if no longer required. For shared SHIR installations, verify no other data sources depend on the runtime before removal.
Unregister Data Source
Navigate to Data Map → Sources, locate the data source, and select "Unregister". This removes the source from your governance inventory. Note that associated discovered assets will be marked as orphaned but may be retained based on your retention policies.
Update Documentation
Mark the data source as decommissioned in your governance inventory, including decommission date, reason, and responsible party. Update any architecture diagrams or data flow documentation to reflect the removal.
Retention Consideration: Some organisations require preservation of discovered assets and scan history even after data source decommission for audit and compliance purposes. Consult your legal and compliance teams to determine appropriate retention periods before executing permanent deletions.
Continuous Improvement and Next Steps
Operational Excellence
- Establish scan health monitoring dashboards and alert mechanisms
- Implement automated testing for credential validity before expiration
- Create self-service onboarding portals for common source types
- Develop runbook templates for frequently onboarded sources
- Conduct quarterly reviews of scan configurations and optimise performance
Advanced Capabilities
- Implement custom classification rules for organisation-specific data patterns
- Configure data quality rules and monitoring for critical assets
- Establish lineage capture for complex ETL processes and applications
- Integrate Purview with data loss prevention and compliance systems
- Develop governance metrics and reporting for stakeholder visibility
Successful Purview adoption extends beyond initial data source onboarding. Organisations realise maximum value through continuous refinement of governance processes, expansion of metadata coverage, and integration with broader data management initiatives. Establish feedback loops with data consumers to understand gaps in metadata, classifications, or lineage visibility. Use these insights to refine scan configurations, enhance classification rules, and improve data quality monitoring.
As your Purview maturity grows, consider implementing advanced scenarios such as automated sensitivity labelling based on discovered classifications, integration with Azure Policy for enforcement of governance standards, and custom APIs that surface Purview metadata in business intelligence tools or data catalogues. Measure success through metrics such as percentage of data estate catalogued, time-to-discovery for new assets, and reduction in data-related incidents or compliance violations.
This runbook represents the foundation of your Purview data governance programme. Regular updates to procedures, incorporating lessons learnt from each onboarding exercise, ensure the document remains relevant and accurate. Schedule annual reviews of the runbook to incorporate new Purview features, updated security best practices, and evolving organisational requirements. Share successes and challenges with your broader data community to foster a culture of continuous improvement in data governance practices.