LinkedIn Open Sources OpenHouse Data Lakehouse Control Plane
Managing data lakehouses isn’t easy, so LinkedIn created and now has released as open source, OpenHouse, a control plane and interface for supervising a wide variety of data lakehouses.
It all starts with data lakes. These are cheap, open storage systems for any data type — CSV, JSON, tabular data, text, images, audio, video, JSON, CSV, etc. A Data Lakehouse, as defined by Databricks, is an architecture that enables efficient and secure Artificial Intelligence (AI) and Business Intelligence (BI) analysis on a data lake’s data. LinkedIn‘s OpenHouse provides an open source control plane to manage tables within open data lakehouse deployments.
This control plane is made up of a declarative catalog and a suite of data services. Users can seamlessly define tables, their schemas, and associated metadata declaratively within the catalog. OpenHouse reconciles the observed state of tables with the desired state by orchestrating various data services.
LinkedIn built this because there were no other tools available that could address its issues. Its open source data lakehouse deployments are built on the foundations of compute engines such as Apache Spark, Trino, and Apache Flink; distributed storage; and metadata catalogs/table formats, like Apache Iceberg, Delta, Hudi, Apache Hive Metastore.” That’s a lot of data in a wide variety of formats and architectures.
As LinkedIn admitted, “While functional, our current setup for managing tables is fragmented. The individual building blocks of compute engines, distributed storage, and metadata catalogs operate independently as part of an overall data plane.”
How LinkedIn Uses OpenHouse
OpenHouse was the answer. Since its inception last year, OpenHouse has been a cornerstone of LinkedIn’s data infrastructure, managing over 3,500 tables and serving more than 550 daily active users. Its impact has been profound, notably slashing the time-to-market for LinkedIn’s data build tool (dbt) implementation on managed tables by over six months and halving the end-user toil associated with data sharing. Integrating over 1,000 datasets, including those from AI and Large Language Models (LLMs), into OpenHouse.
The inspiration behind OpenHouse stemmed from the perennial struggle between control and flexibility in big data management. Traditional cloud data warehouse solutions, while ensuring governance and performance, often lack the scalability and adaptability offered by open source data lakehouse systems. OpenHouse emerges as a solution to this dilemma, providing a managed experience that liberates end-users from the intricacies of infrastructure management while empowering data infrastructure teams with enhanced control and governance capabilities.
At the heart of OpenHouse lies its Catalog, a RESTful table service that offers secure and scalable table provisioning alongside declarative metadata management. This is complemented by Data Services, which facilitates seamless table maintenance.
OpenHouse’s key features include fundamental catalog operations, retention management, governance through column tagging, and comprehensive observability tools. These features are seamlessly integrated with Apache Spark. This enables standard engine syntax, SQL queries, and the DataFrame API to execute operations efficiently.
Moreover, OpenHouse introduces advanced replication capabilities by extending the Apache Gobblin framework, ensuring high availability and consistency across geographies. Its support for Apache Iceberg as a table format further underscores its commitment to compliance and optimal performance through regular maintenance tasks.
Recognizing the importance of adaptability, OpenHouse was designed with pluggability in mind, offering interfaces for storage, authentication, authorization, database management, and job submission. This design philosophy ensures that OpenHouse can be customized to fit diverse environments, from cloud infrastructures to specific table formats.
As OpenHouse embarks on this new chapter as a BSD 2-Clause license open source project, LinkedIn invites the global community to explore its capabilities, contribute to its development, and provide feedback. The company is particularly focused on understanding how OpenHouse performs in various settings and is committed to addressing technical challenges as it transitions from Apache Hive to OpenHouse.
