Fireproof Uses Merkle CRDTs to Cut Database Latency
Fireproof — the developer of a real-time, open source document database that’s lightweight enough to run almost anywhere, including on mobile and edge devices — launched in February of 2023. Eighteen months later, it’s got a thriving community of open source users and contributors. It recently completed its pre-seed funding, led by Engineering Capital, and is set to unveil the paid version of the platform, Fireproof Cloud, in November.
The database’s lightweight imprint, real-time applicability, and interactive support for collaborative, user-generated transactional data is greatly attributed to its reliance on Merkle CRDTs. This data structure pairs cryptographic approaches alongside those for linked data (i.e. semantic knowledge graphs) for a local-first experience, in which users can write applications first, and then choose their backend later.
“You’ll be familiar with the experience in any collaborative online app that has even a modicum of offline support,” said Chris Anderson, Fireproof CEO. “If it’s fast, it’s got something like a CRDT powering it.”
Coupling the advantages of CRDTs with those supplied by Merkle — which “is what powers the blockchain,” Anderson noted — enables Fireproof to offer flawless data provenance for expedient, interactive applications for GenAI chatbots, financial trading, helpdesks, and more.
The Might of CRDTs
Conflict-Free Replicated Data Types (CRDT) were largely designed to simplify some of the complications of distributed computing. According to Anderson, CRDTs “are all about collaboration and the ability to have a document or dataset shared across multiple devices or users.” The chief value proposition of CRDTs is they can be combined after being distributed so that the sum of individual efforts is reaped by the underlying system.
This paradigm allows Fireproof to distribute its workloads and computations among different parties while availing itself of the collective efforts involved in performing them. “You can take any two copies of CRDTs and hand it out to four friends, and they can do some stuff on it and you can always merge it later,” Anderson explained.
Fireproof’s CRDTs are noteworthy because they also involve the Merkle paradigm, which enables them to provide the data provenance for which blockchain is known. The combination is ideal for any number of real-time applications in which such data provenance is desirable, like “in any industry, if you’ve got a call center or a management desk that’s documenting events in real-time and making decisions, and needs to be able to look back on what happened with an audit level of integrity,” Anderson said.
Immutable Data Provenance
Although Fireproof is not expressly powered by blockchain, the data provenance of that technology is one of its critical enablers. This fact is particularly true for transactional applications. “Each transaction in Fireproof corresponds to a snapshot that has a cryptographic hash associated with it for data provenance,” Anderson revealed. “If you’ve got the hash you can load the root of the snapshot and, from there, the whole data structure is tamper-proof, or self-validating.” Users can employ that data provenance however it best suits their specific application. According to Anderson, it’s possible to facilitate it at granular levels, like for the touch ID or facial ID of a user performing work on a device.
However, it’s the low latency implications of such data provenance that, when combined with Fireproof’s CRDT methodology, are most impressive. One of the early contributors to Fireproof’s open source project employed the technology for event data recording for autonomous vehicles.
When an event occurs, such as airbags deploying, it’s necessary to upload the most recent video and telemetry data in what Anderson termed “priority order”. In that case, “they can send, every second or however often they want, in a cookie to the cloud that’s a proof of the provenance of all the data that would come in and upload later,” Anderson said. “That way, the proof is going along and, when it’s time to snapshot, you can prove the snapshot integrity corresponds to the event.”
Linked Data
Whereas other document stores typically represent data via the JSON format, Fireproof’s data is represented via the InterPlanetary Linked Data (IPLD) model. As the name of this data model implies, it utilizes concepts associated with linked data or semantic knowledge graphs, including the tenet of providing unique identifiers to data objects.
According to Anderson, IPLD is “a binary object representation that’s a little more expressive than JSON.” This data model is characterized by a universal identifier for data at rest and inherent support for provenance regarding the order in which respective objects have been changed. “What the deal with it is, it’s all made with these immutable references, because the identifiers of everything in IPLD is the hash of its content,” Anderson observed.
This data model complements Fireproof’s Merkle CRTD structure by enabling the latter to contribute to the former’s data provenance. This provenance reinforces the overall integrity of the data contained in Fireproof and that of the myriad applications it supports.
“We’re extending [IPLD] to add to that provenance the signature of the past key style, the key from the end device that made the update,” Anderson said. “That will give you the full picture.”
Fireproof’s immutable data provenance is optimal for use cases involving insurance contracts, including those for rental cars. For “the part where you decline the insurance at the rental car company, you can have a cookie that you can put into a pdf and take away a signature, if you wanted to,” Anderson said.
Fireproof Cloud
The projected November release of Fireproof Cloud represents the grand vision of the company — for the time being. It will contain additional features, like capabilities for vectorizing user text snippets, as well as best practices for “sharing [data], security access control stuff, and designing your own schema in a traditional app,” Anderson added. “You can piggyback on our best practices.” Plus it will retain Fireproof’s core characteristics of allowing developers to quickly set up a database, that operates on almost any platform while synchronizing data with cryptographic proofs for immutable data provenance, ready collaboration, and timely responsiveness.
