Amundsen: Navigating the Ocean of Enterprise Data

In today’s data-driven landscape, organizations face a paradoxical challenge: they possess vast oceans of data but struggle to help their teams find the right information when needed. As data ecosystems grow increasingly complex—spanning multiple databases, data lakes, and analytics platforms—the ability to discover and understand relevant data becomes a critical bottleneck. Lyft recognized this challenge and created an innovative solution that has since become one of the most popular open-source data discovery platforms: Amundsen.

Named after the Norwegian explorer Roald Amundsen, who was famous for navigating the Northwest Passage and reaching the South Pole, this platform helps users navigate their own complex data environments. Amundsen combines search capabilities, metadata enrichment, and community features to transform how teams discover, understand, and trust their data assets.

Before diving into Amundsen’s capabilities, it’s worth understanding the challenges it was designed to solve:

Many organizations have successfully built data lakes and warehouses, but without proper metadata management and discovery tools, these can quickly become “data swamps”—vast repositories of information that are difficult to navigate. Data analysts often spend 50-80% of their time simply trying to find the right data rather than analyzing it.

In many organizations, data knowledge exists as tribal wisdom spread across teams. When someone needs to find a specific dataset, they rely on asking colleagues rather than using systematic tools. This approach breaks down as organizations scale and as team members change roles.

Even when users find potentially relevant data, they often struggle to determine if it’s trustworthy, current, and appropriate for their use case. Without context about data quality, freshness, and lineage, users hesitate to base important decisions on the data they discover.

Amundsen tackles these challenges with a microservice-based architecture built around three core components:

At its heart, Amundsen is a search engine for data. The search service, powered by Elasticsearch, enables users to quickly find datasets, dashboards, users, and other resources through a Google-like interface. This component indexes metadata from across the organization’s data ecosystem, making it searchable through a unified interface.

The metadata service acts as the foundation of Amundsen, storing and serving comprehensive information about data assets. This service is built on Neo4j, a graph database that excels at storing the complex relationships between different data entities. This graph-based approach allows Amundsen to understand and visualize how data resources are connected.

// Example Neo4j query showing how Amundsen models data relationships
MATCH (table:Table)-[:HAS_COLUMN]->(column:Column),
      (table)-[:IN_DATABASE]->(db:Database),
      (table)-[:IN_SCHEMA]->(schema:Schema)
WHERE db.name = 'hive' AND schema.name = 'sales'
RETURN table.name, collect(column.name) as columns

The frontend service provides an intuitive interface for users to interact with Amundsen. Built with React, it offers a clean, modern UI that presents search results, detailed metadata, and collaboration features. The interface is designed for both technical and non-technical users, emphasizing simplicity and ease of use.

Beyond these core services, Amundsen includes a flexible data ingestion framework that connects to various metadata sources through extractors. These extractors pull information from data sources like Hive, Presto, Snowflake, Redshift, BigQuery, and many others. This framework allows Amundsen to build a comprehensive view of an organization’s data landscape.

Unlike basic metadata repositories, Amundsen brings Google-like search capabilities to data discovery. Its search algorithm accounts for various factors to rank results by relevance:

• Usage-based ranking: Frequently used datasets appear higher in results
• Field-level search: Find tables based on column names and descriptions
• Description completeness: Better-documented resources rank higher
• PageRank-inspired algorithms: Tables connected to important resources receive higher rankings

This approach helps users quickly find the most relevant resources rather than wading through long lists of potential matches.

Amundsen provides comprehensive profiles for data assets that include:

• Schema information: Detailed column definitions, types, and descriptions
• Sample data: Preview actual values to understand content
• Usage statistics: How frequently the data is accessed and by whom
• Data quality metrics: Quality scores and validation results
• Freshness indicators: When the data was last updated

These profiles give users the context they need to understand if a dataset is appropriate for their needs without having to load and explore the data themselves.

Understanding where data comes from and how it transforms is crucial for building trust. Amundsen provides lineage visualization that shows:

• Upstream sources: What raw data feeds into this table
• Transformation processes: How the data is processed and changed
• Downstream dependencies: What reports and dashboards use this data

This feature helps users understand the complete lifecycle of data, enhancing trust and enabling impact analysis for potential changes.

Amundsen recognizes that data discovery is not just a technical challenge but also a social one. It includes features that promote collaboration:

• Dataset owners: Clear identification of who owns and understands each resource
• User bookmarks: Ability to save frequently used resources
• Usage insights: Information about which teams and individuals use specific data
• Tribal knowledge capture: Ways to document and share institutional knowledge

These social features help transform organizational data knowledge from tribal wisdom to systematic documentation.

Amundsen can be deployed in several ways, depending on organizational needs:

1. Docker Compose: The simplest option for testing and small deployments git clone https://github.com/amundsen-io/amundsen.git cd amundsen docker-compose -f docker-amundsen.yml up
2. Kubernetes: For production-grade deployments helm repo add amundsen https://amundsen-io.github.io/amundsen/ helm install amundsen amundsen/amundsen
3. Cloud-Native Services: Leverage AWS Neptune for the graph database and Amazon Elasticsearch Service for search

Successfully implementing Amundsen requires a thoughtful approach to metadata ingestion:

1. Identify key data sources: Start with the most widely used databases and data warehouses
2. Configure extractors: Set up the appropriate connectors for each source
3. Establish ingestion schedules: Determine how frequently metadata should be refreshed
4. Enrich with business metadata: Add descriptions, owners, and tags for context
5. Integrate with data quality tools: Connect with tools like Great Expectations or dbt for quality metrics

Most organizations begin with a focused approach, integrating their most critical data sources first and then expanding coverage over time.

Amundsen is designed to be extensible and customizable:

• Custom metadata models: Extend the metadata schema for specific needs
• UI customization: Adapt the interface to match organizational branding
• Integration with internal tools: Connect with proprietary systems and workflows
• Custom search ranking: Adjust relevance algorithms to fit specific use cases

This flexibility allows organizations to tailor Amundsen to their unique data environments and discovery needs.

As the birthplace of Amundsen, Lyft provides the most well-documented implementation:

• Challenge: Data scientists spent 60% of their time finding and validating data
• Implementation: Built Amundsen to index 100% of their data warehouse tables
• Result: Reduced data discovery time by 70%, with thousands of weekly active users
• Impact: Accelerated decision-making and analytical workflows across the company

A mid-market e-commerce business implemented Amundsen to solve cross-team data sharing:

• Challenge: Siloed data knowledge across product, marketing, and analytics teams
• Implementation: Deployed Amundsen with integrations to Snowflake, Redshift, and Tableau
• Result: Created a unified data catalog with clear ownership and documentation
• Impact: Improved cross-functional collaboration and reduced duplicate data pipelines by 40%

A large bank implemented Amundsen as part of their regulatory compliance initiative:

• Challenge: Needed to document data lineage and usage for regulatory reporting
• Implementation: Integrated Amundsen with their existing governance framework
• Result: Comprehensive metadata repository with clear lineage documentation
• Impact: Streamlined audit processes and improved regulatory compliance reporting

Both originated from tech companies (Lyft and LinkedIn) but took different approaches:

• Architecture: Amundsen uses Neo4j as its graph database, while DataHub has a more flexible storage layer
• UI Focus: Amundsen emphasizes simplicity and search, while DataHub offers more extensive governance features
• Implementation Complexity: Amundsen is generally considered easier to deploy initially

While both address metadata management, they serve different primary use cases:

• Origin & Focus: Atlas was designed primarily for Hadoop governance, while Amundsen focuses on data discovery
• User Experience: Amundsen provides a more consumer-like search experience
• Technical Requirements: Atlas has a heavier infrastructure footprint
• Governance Depth: Atlas offers more comprehensive governance capabilities

Compared to proprietary platforms like Alation or Collibra:

• Cost: Amundsen is open-source with no licensing fees
• Implementation Speed: Can be deployed more quickly for basic discovery needs
• Enterprise Features: May require more customization for advanced governance requirements
• Support: Relies on community support rather than dedicated vendor assistance

The most successful Amundsen deployments begin by addressing specific pain points:

1. Identify a high-value use case: Focus on a particular team or dataset that would benefit most
2. Measure current state: Document how long users currently spend finding data
3. Implement targeted solution: Deploy Amundsen with relevant metadata for this specific case
4. Measure improvements: Track changes in data discovery time and user satisfaction
5. Expand incrementally: Add more data sources based on demonstrated success

This approach builds momentum through quick wins rather than attempting a massive enterprise-wide deployment initially.

Technology alone won’t solve data discovery challenges—organizational culture matters too:

• Assign data stewards: Designate individuals responsible for metadata quality
• Create incentives: Recognize teams that maintain high-quality metadata
• Include in workflows: Make metadata updates part of regular data processes
• Lead by example: Ensure leadership uses and references the platform

Organizations that succeed with Amundsen establish metadata quality as a shared responsibility rather than an afterthought.

Amundsen delivers the most value when integrated with complementary data tools:

• Data quality platforms: Connect with tools like Great Expectations or dbt
• BI and visualization tools: Link Tableau, Looker, or PowerBI dashboards
• Data catalogs and governance: Integrate with broader governance platforms
• Data pipelines: Capture lineage from workflow tools like Airflow

These integrations create a comprehensive data management environment rather than a standalone discovery tool.

The Amundsen project continues to evolve with several exciting developments:

Machine learning is being applied to improve discovery:

• Personalized recommendations: Suggesting relevant datasets based on user behavior
• Automatic tagging: Using ML to classify and tag datasets
• Anomaly detection: Identifying potential metadata issues
• Usage prediction: Forecasting which datasets will become important

The platform is expanding beyond traditional data tables to include:

• Machine learning features and models: Discovery for AI/ML assets
• APIs and data services: Finding programmatic data access points
• Data applications: Discovering data-powered applications
• Query snippets: Finding and reusing common analytical queries

As data governance becomes increasingly important, Amundsen is enhancing its capabilities:

• Policy management: Tracking data usage policies and constraints
• Compliance metadata: Documenting regulatory requirements
• Access request workflows: Streamlining data access processes
• Audit trails: Tracking who accesses what data and why

In a world drowning in data but starving for insights, Amundsen provides a lighthouse for organizations navigating their complex data landscapes. By combining powerful search capabilities, rich metadata context, and collaborative features, it transforms how teams discover and understand their data assets.

The platform’s open-source nature, combined with its focus on user experience, has made it a popular choice for organizations seeking to accelerate their data workflows. Whether you’re a large enterprise with petabytes of data or a growing company building your data practice, Amundsen offers a proven approach to solving the data discovery challenge.

As data continues to grow in both volume and strategic importance, tools like Amundsen will become increasingly essential components of the modern data stack. By helping users find the right data quickly and understand it thoroughly, Amundsen doesn’t just solve a technical problem—it transforms how organizations create value from their data assets.

#Amundsen #DataDiscovery #MetadataManagement #DataCatalog #OpenSource #DataGovernance #DataLineage #Neo4j #Elasticsearch #LyftEngineering #DataSearch #ModernDataStack #DataScience #KnowledgeGraph #DataQuality #SnowflakeTools #RedshiftMetadata #BigQueryCatalog #DataMesh #SelfServiceAnalytics