Sign in
Dapp

A Developer's Tale: Building a Database with IPFS

In this blog, explore the fascinating journey of building the dapp "Polis" and its IPFS-powered database inspired by ancient Greece's Polis city.

Infura

• 8 min read
A Developer's Tale: Building a Database with IPFS

In this blog, we explore the fascinating journey of building the dapp "Polis" and its IPFS-powered database inspired by ancient Greece's Polis city.

In ancient Greece, there was a city called "Polis" where people gathered to discuss a wide range of topics, including politics, governance, and laws. Unfortunately, they never delved into the art of building a database in IPFS.

As a result, when we embarked on building the dapp Polis that necessitated an IPFS-powered database, we faced numerous challenges, occasionally succumbing to bouts of frustration. 

In this developer's tale, I'll explore our journey, highlighting our thought processes, unforeseen discoveries, and key moments. Although this isn't a comprehensive guide and may not represent the perfect solution, it stands as a valuable resource for those aiming to create something similar or explore deeper experimentation.

What is web3 dapp, Polis?

Polis is a knowledge base for all things dapps, its primary function is to be a collection of dapp templates and essential tools, curated to empower developers in their quest to create outstanding projects. For now, anyone can submit their entry on Polis in a few simple clicks and plans are on the way to better streamline this process. The code for Polis is also open-sourced so feel free to follow the repository and give it a star!

Choosing the stack for Polis with Infura and IPFS

In choosing the stack for Polis, we had two key requirements: using Polis should be free for all users and Polis must be developed using a decentralized database so as to reduce dependence on centralized providers. With these requirements set, we embarked on a quest for the Polis-stack.

Database

In choosing a database, the first option we ruled out was the blockchain, primarily because of gas fees. Considering the vast amount of data we planned to gather, including descriptions, links, and images, writing to the blockchain would be costly.

Other database solutions like orbit-db were available, but we sought something straightforward, minimalistic, and something we could easily integrate with our existing knowledge.

This lead us to IPFS, IPFS is a protocol designed for organizing and transferring data using content addressing rather than location addressing, complemented by peer-to-peer networking.

However, a significant drawback of IPFS is its emphasis on data immutability. Traditional databases allow for easy updates and modifications, while IPFS enshrines immutability. Once data is committed to IPFS, it remains unchanged. To update data, one typically generates a new version with a different content hash.

Our next challenge was finding a method to create mutable data storage with IPFS. One solution is to implement a mutable file system (MFS). Though a promising idea, we realized we weren't looking for file storage per se, but rather a database that might be encompassed in a single file.

Our research then led us to IPNS, and it appeared to be the perfect complement to IPFS. With their powers combined, we were set on the database end.

Infura IPFS

For easy access to IPFS, we chose Infura. Infura is both reliable and intuitive and with the Infura IPFS offering, you can be assured that your IPFS files are always stored securely, preventing the feeling that you're sending files into the Bermuda Triangle.

To interact with Infura, you can utilize the API endpoints or use the JavaScript client library for IPFS, ipfs-http-client or alternatively, you can use kubo-rpc-client. However, we chose to use the HTTP endpoints that Infura provides. For a comprehensive introduction to Infura IPFS, you might find this blog post helpful.

Frontend

For our frontend, we enlisted Next.js because, let's face it, it's the cool kid in town. To save ourselves from CSS-induced headaches, we brought in Tailwind CSS – because, frankly, we're not CSS superheroes.

w3name

To address content addressing and naming, we went with w3name. This is a service and client library that implements IPNS, - a protocol that uses public key cryptography to allow for updatable naming in an atomically verifiable way.

Now let’s dive into how we tied all these together.

IPFS as a database

For the sake of simplicity, we are creating a minimal JSON database named applications.json.

This JSON file consists of an array of objects, each representing an application submitted by a fellow web3 enthusiast. Here's an example featuring just one application.

[
	{
			"id": "oMObNGIHGupfgKkwUnK-W", // random string
			"user": "0xc12ca5A8c4726ed7e00aE6De2b068C3c48fA6570", // user's wallet address
			"createdAt": "9/1/2023, 11:34:27 PM",
			"category": [ "Security"],
			"title": "MobyMask ",
			"description": "This snap warns you when interacting with a contract that has been identified as a phisher in the MobyMask Phisher Registry.  ",
			"applicationUrl": "<https://montoya.github.io/mobymask-snap/>",
			"repoUrl": "<https://github.com/Montoya/mobymask-snap>",
			"screenshots": "QmddFhP3od4qRyLpzVQkoCuoGp3XRxHEeS1kckW3uHfiqF",
			"logo": "QmdiM7AnXtShBznk6HbHrTBttbetRYbeixmDjaypreuPCV",
			"isEditorsPick": false
	}

]

While searching for ways to build a simple database with IPFS, we came across an awesome blog post by Radovan Stevanovic. Based on this post, we built our own implementation, considering the data structure we wanted. We didn't implement all the database functionalities since we aimed to test an MVP of Polis first. 

Here are some primary functions in the database, sufficient to understand our direction, located in `lib/database.ts`. As we continue to receive more applications, both the JSON object and file grow in size. While this growth might pose challenges in the future, we're not overly concerned currently. 

We can tackle scalability issues later. Although IPFS excels in its present role, we must think about how we'll manage updates, like adding new applications or editing existing ones. Get ready for some tech magic as we mould IPFS's rigidity into a more adaptable dance, all thanks to IPNS!

Compensating for IPFS's shortcomings using IPNS

The primary challenge with our current approach is that any update to the JSON file results in a new file with a different CID. This isn't the desired outcome.

How does IPNS help address this issue? IPNS allows users to associate a mutable, human-readable name with a specific IPFS content address. Even when the content linked to the IPNS name changes, we can still access the updated content.

Here's a brief overview of how IPNS works:

  • An IPFS user generates a cryptographic key pair. The private key remains secret, while the public key is used to create an IPNS namespace.
  • The user can then publish an IPNS record, signing it with their private key. This record contains the mutable name and the corresponding IPFS content address (usually a hash).
  • The IPNS record is distributed across the IPFS network via the Distributed Hash Table (DHT), making it available to other users.
  • When someone wishes to access content associated with the mutable name, they look up the IPNS record using that name and retrieve the current content address.

Tying it all together with a frontend

In the 'Submit Application' modal, once the submit button is pressed, a brief pause ensues, highlighted by a continuously spinning loader icon on the button. A significant amount of data, including images, is then relayed to the submitApplication function. Here’s what goes on inside that function:

export const submitApplication = async ({
  images,
  data,
}: {
  images: FormData;
  data: string;
}) => {
  let logoHash, screenshotsHash;

  const logo = images.get("logo") as File;
  const screenshots = images.getAll("screenshots") as File[];

// adding the logo to ipfs and get back the hash of the file
  if (logo) {
    logoHash = await add({ file: logo });
  }

// adding all screenshots to ipfs as a directory and get back the hash of the directory
  if (screenshots.length > 0) {
    screenshotsHash = await add({ files: screenshots });
  }

  const application = JSON.parse(data) as Omit<
    IApplication,
    "id" | "screenshots" | "logo"
  >;

// retrive the latest database file. 
  const state = await retrieveDatabase();

  const newState = addNode(state, {
    id: nanoid(),
    ...application,
    screenshots: screenshotsHash,
    logo: logoHash,
    createdAt: new Date().toLocaleString("en-US", { timeZone: "UTC" }),
    isEditorsPick: false,
  });

// creating and add a new file to ipfs, then publish its hash to ipns
  await storeDatabase(newState);

// nextjs cache revalidation to show the newly added application
  revalidatePath("/");
};

Here’s a high-level description of what this function does:

  • Extracts image files: It extracts the logo and screenshots from the provided images FormData.
  • Uploads to IPFS: If provided, the logo and screenshots are uploaded to IPFS, and their respective IPFS hash values are stored.
  • Parses application data: The provided data string is converted to a JSON object.
  • Retrieves current database: It fetches the current state of the database.
  • Adds new application entry: A new application entry, including the uploaded image hashes, is added to the database.
  • Updates database on IPFS: The updated database is stored on IPFS and its hash is published to IPNS for mutable referencing.
  • Cache revalidation: It triggers a cache update to reflect the new addition in the user interface.

Let’s take a closer look at what happens inside the retrieveDatabase function referenced above:

const retrieveDatabase = async () => {
// getting the latest db hash
  const hash = await getcurrentHash();

  const json = await cat(hash);
  return deserializeDatabase(json);
};

Here, IPFS doesn't know the location of our database file, so it reaches out to IPNS and waits for a reply, IPNS finally responds with a hash, here’s a closer look at the getCurrentHash function:

const getcurrentHash = async () => {
const nameServiceUrl = process.env.WEB3_NAME_SERVICE_URL;//https://name.web3.storage/name

  if (!nameServiceUrl) {
    throw new Error("WEB3_NAME_SERVICE_URL is not set");
  }

  const res = await fetch(`${nameServiceUrl}/${process.env.DB_HASH}`);
	
  return (await res.json()).value; // hash of the latest db file
};

Here’s a closer look at the addNode function:

const addNode = (
  state: Map<string, ApplicationNode>,
  node: ApplicationNode
) => {
  return new Map(state).set(node.id, node);
};

Here’s a closer look at the storeDatabase function:

const storeDatabase = async (state: Map<string, ApplicationNode>) => {
  const json = serializeDatabase(state);
  const jsonDataBlob = new Blob([json], { type: "application/json" });
// adding the file to IPFS
  const hash = await add({ blob: jsonDataBlob, fileName: "db.json" });

  if (!hash) {
    throw new Error("couldn't store database");
  }
// updating the ipns record with the latest hash: refer the next act
  await update(process.env.DB_HASH!, process.env.DB_KEY!, hash);
};

A closer look at what goes on in the add function:

export const add = async ({ blob: Blob, fileName: string}) => {
  // append ?wrap-with-directory=true to the baseUrl to upload a directory
  const baseUrl = `${process.env.INFURA_IPFS_ENDPOINT}/api/v0/add`;
  const formData = new FormData();
  formData.append("data", data.blob, data.fileName);
  try {
    const response = await fetch(baseUrl, {
      method: "POST",
      headers: {
        Authorization:
          "Basic " +
          Buffer.from(
            process.env.INFURA_IPFS_KEY + ":" + process.env.INFURA_IPFS_SECRET
          ).toString("base64"),
      },
      body: formData,
    });
    const hash = (await response.json()).Hash;
    return hash;
  } catch (error) {
    console.error("Error adding file", error);
  }
};

Now, it's IPNS's turn to shine. It holds critical secrets, and only it can access and modify them using its private key, serving as the guardian of the name:

import * as Name from "w3name";
const update = async (ipns: string, keyStr: string, content: string) => {
	if (!keyStr) {
    	return;
  }

  const name = Name.parse(ipns);
  const revision = await Name.resolve(name);
  const nextRevision = await Name.increment(revision, content);
  const key = new Uint8Array(Buffer.from(keyStr, "base64"));
  const name2 = await Name.from(key);
// publishing the next version of the db with the private key
  await Name.publish(nextRevision, name2.key);
};

Wrapping Up

Though it might seem like we reached a seamless conclusion, the journey to this endpoint wasn't always smooth, as those who build often understand. Here are some trials, errors, and potential future challenges:

  • We toyed with the idea of merging traditional methods with newer ones. What if we saved application data in IPFS but stored user details in a web2 database like Supabase? This would simplify authentication and authorization. However, it felt incongruous, akin to powering a Tesla using a diesel generator while believing we were eco-friendly.
  • In our dapp's early version, we aimed to store data for each application in a unique IPFS directory covering data, screenshots, and logos. We'd then publish each to IPFS individually. This strategy resulted in numerous IPFS files. Plus, users had to oversee their IPNS private keys independently.

    The intent was to permit updates to a specific application by its creator exclusively. Yet, this approach made it challenging to search for applications, and when we did find them, the process was slow. Managing private keys presented another hurdle. We considered using MetaMask Snaps, which seemed promising, but we shelved the idea due to other prevailing issues.
  • We also experimented with Gun, OrbitDb, and a few other database solutions.

So, my friend, that's how my developer tale comes to a close. I hope you've enjoyed it. If you’re curious to learn more about IPFS and how Infura makes it easy to access IPFS, check out the Infura website and the developer’s guide to IPFS.

Sign up for more like this.

Enter your email
Subscribe