239 ) Metadata Management

Metadata Management and Modern Data Governance Tools

Metadata management forms the backbone of data governance, data lineage, and data quality architectures. It ensures that organizations know what data they have, where it lives, who owns it, how it transforms, and whether it can be trusted.

What is Metadata Management?

Metadata management is the administration, organization, and integration of structural, descriptive, administrative, and operational information about data assets. It creates a centralized inventory (often called a Data Catalog) mapping data elements to their technical origins and business contexts.

Categories of Metadata

1. Technical Metadata: Database schemas, table names, column data types, constraints, index configurations, ETL pipelines, and API structures.
   
   
2. Business Metadata: Business glossaries, data definitions, ownership, policies, security classifications, compliance tagging (e.g., GDPR, CCPA), and data quality metrics.
   
   
3. Operational Metadata: Execution logs, job run times, refresh frequency, error rates, and data lineage history.
   
   

Core Tools for Metadata Management

Enterprise metadata management platforms combine automated harvesting (crawling) with collaborative business curation:

• Microsoft Purview: A unified governance solution deeply integrated with cloud ecosystems to map data across on-premises, multi-cloud, and SaaS software.
  
  
• Collibra Data Intelligence Cloud: An enterprise data governance platform centered heavily on business process workflows, data stewardship, and comprehensive business glossaries.
  
  
• Alation: Known for its behavioral analytics and collaborative data catalog, utilizing AI to crowdsource data documentation.
  
  
• Informatica Enterprise Data Catalog (EDC): Features AI/ML-driven scanning (CLAIRE engine) for automated asset discovery and metadata parsing across massive enterprise footprints.
  
  

Creating Database Metadata: Step-by-Step Menus and Commands

To map and document metadata for a database, tools generally use two pathways: automated technical scanning or manual asset creation.

1. Microsoft Purview

In Purview, metadata about a database is automatically extracted, categorized, and cataloged through the Register and Scan framework.

Step-by-Step Instructions:

1. Open the Portal:
   
   
   • Navigate to the Azure Portal, search for your Microsoft Purview account, and click Open Microsoft Purview governance portal.
     
     
2. Access the Data Map:
   
   
   • From the left-hand navigation pane or home screen, click on Data map.
     
     
3. Register the Data Source:
   
   
   • Go to Data sources (or Sources) and click the + Register button at the top.
     
     
   • Select your database type from the list (e.g., Azure SQL Database or SQL Server) and click Continue.
     
     
   • Fill out the registration form:
     
     
     • Name: Provide a friendly identifier (e.g., Prod_Customer_DB).
       
       
     • Server name / Endpoint: Specify the connection host path.
       
       
     • Collection: Choose the target collection hierarchy where the asset will live.
       
       
   • Click Register.
     
     
4. Configure Authentication & Credentials:
   
   
   • Set up permissions (e.g., granting a managed identity or service principal db_datareader access on the target database).
     
     
   • Store credentials securely in an Azure Key Vault connected to Purview.
     
     
5. Run the Scan to Generate Metadata:
   
   
   • Locate your newly registered database source in the Data Map, hover over it, and click the New Scan icon (or select New Scan under the source details).
     
     
   • Provide a Scan name (e.g., Full-Schema-Scan) and select your configured Credential.
     
     
   • Scope the scan to specific schemas, tables, or select the entire database.
     
     
   • Choose a Scan rule set (system defaults handle standard structural extraction and system classifications like PII).
     
     
   • Configure a trigger schedule (e.g., run weekly or run once).
     
     
   • Click Save and run.
     
     
   • Result: Purview automatically parses technical database metadata (tables, columns, data types, and foreign key relationships) into the Data Map.
     
     

2. Collibra Data Intelligence Cloud

Collibra manages database metadata either via automated integration/connectors or manual top-down modeling within the operating model structure (Database asset $\rightarrow$ Schema asset $\rightarrow$ Table asset $\rightarrow$ Column asset).

Step-by-Step Instructions (Manual Asset Creation):

1. Open the Global Navigation Toolbar:
   
   
   • Log into Collibra and look at the main navigation bar at the top of the screen.
     
     
2. Initiate Creation:
   
   
   • Click the Create button (represented by the + plus icon) on the main toolbar.
     
     
   • The Create dialog box will appear.
     
     
3. Select Asset Type:
   
   
   • Click on the Assets tab within the dialog box.
     
     
   • In the Asset type field, search for and select Database (subsequent levels use Schema, Table, or Column).
     
     
4. Define Asset Parameters:
   
   
   • Domain: Choose the designated Community and Domain where this database technical asset should reside (e.g., Technical Metadata Domain).
     
     
   • Asset Name: Type the precise name of the database (e.g., Sales_Analytics_DB).
     
     
5. Save the Asset:
   
   
   • Click Create.
     
     
6. Establish Relations (Nesting Schema and Tables):
   
   
   • Open the newly created Sales_Analytics_DB asset page.
     
     
   • Navigate to the Relations section of the asset.
     
     
   • Add a relation to connect child components using contextual commands: click + Add Relation, select the relation type (e.g., Database contains / is part of Schema), and link or create the child Schema asset.
     
     
   • Repeat the downward hierarchy steps to link schemas to Tables and tables to Columns, allowing you to curate both automated technical imports and rich business context descriptions manually.