Tobiko Launches Its SQLMesh-Based Cloud Service Into GA
Tobiko is the company behind the open source SQLMesh project, a data transformation framework that aims to allow developers to more easily and efficiently ship data. Tobiko is making its enterprise-ready cloud service Tobiko Cloud generally available today.
The new service extends the SQLMesh project with enterprise features like single sign-on and role-based access controls on top of the existing scheduling and observability features that are already part of Apache 2.0-licensed SQLMesh.
In some ways, SQLMesh and Tobiko Cloud are direct competitors to dbt and dbt Labs. The team is not shying away from the dbt comparison. Indeed, on its homepage, Tobiko argues that the new service is “everything dbt wishes it was.” Tobiko Cloud, the team argues, can scale well beyond what dbt is currently capable of.
The idea at the core of SQLMesh is to bring modern DevOps practices to building data pipelines. SQLMesh has a semantic understanding of SQL, allowing it to effectively parse SQL, track changes, run unit tests and debug transformation errors. It also allows users to create virtual data environments during the development process. Developers can write their pipelines in SQL or Python.
Tobiko Cloud’s scheduler.
“SQLMesh has already done a lot, really,” Tobiko co-founder and CEO Tyson Mao said. “It’s already shown to impact teams in terms of cutting down costs, automating processes and just saving time. Time spent waiting around for the warehouse to compute things is time wasted and money spent on the warehouse. We’ve shown that we can really improve those types of things.”
As for where the team draws the line between the open source project and its commercial service, Tobiko co-founder Toby Mao noted that the team focused squarely on enterprise features. The two core features that are included in Tobiko Cloud that aren’t available in the core open source project are role-based access control and governance. But the commercial offering also includes an improved scheduler which the company claims is more performant than the one in the open source version.
“With Tobiko Cloud, we can have an understanding of who the users are, and so we can do things like role-based access control and other things for security and for data deployments, where you don’t want to necessarily have to give your developer access to your entire warehouse,” Toby Mao said.
Mao credits dbt for setting the standard for data transformations to a large audience. But he also notes that the team saw a 9x improvement in speed and cost efficiency over dbt when it benchmarks its commercial offering with the proprietary scheduler against dbt. During a time when enterprises are increasingly mindful about how much money they are spending on their cloud services, that’s a potential advantage for Tobiko. Already, half of the company’s customers previously used dbt (while others often come from Airflow, Datafrom or even home-built systems).
SQLMesh model blueprinting.
On top of its commercial offering, Tobiko also continues to drive the development of the open source version. The community around SQLMesh has grown significantly in the last year. Tobiko co-founder and chief architect Iaroslav Zeigerman stressed that any changes to SQLMesh are driven by the community. Some of the latest additions there include a linter and model blueprints that help developers reduce boilerplate by creating templatized model definitions.
Looking ahead, the team is now working on a VS Code extension to help streamline the developer experience, but unsurprisingly, it is also looking at how it can use AI to further improve its service.
