A Beginner’s Guide to Decentralized Data Layers

Imagine you’ve written a document, and you’d like to share it with some people you know. You have a few options for how you could go about that: you can store it locally, store it using the cloud, or store it using a decentralized method. They all have pros and cons.

Which one should you pick?

The Problem With Centralized Data Stores

Pre-internet, you might have a hard drive where you saved the document. This is inconvenient because you’d have to share the physical object in order to share the document.

These days, you’re much more likely to share a document with “cloud” storage; that is, storage options that require an internet connection. We think of things like Google Docs or Dropbox as examples of cloud storage. With cloud storage, you use your internet connection to access a server housed somewhere else. Ultimately, the data is still present in one physical location, but you don’t need to be physically present to access the information.

Cloud storage is a huge technological advancement from local storage, like hard drives, because it allows you to access information from just about anywhere, even using devices that wouldn’t be powerful enough to hold all that data locally.

However, both of these are still examples of centralized storage. At the end of the day, the data is housed in one real, physical location. And that comes with some risks. For instance, if a data server is hacked, all the information stored there can be taken down. If the location is compromised by a disaster like a flood or a fire, there goes all the data.

Furthermore, while you might own your own hard drive, cloud servers are usually owned by a company. If Google or Dropbox decide that they don’t want to host your data anymore, they can take it down and remove all ability to access it. Or the companies themselves may be hacked or served cease-and-desist orders, forcing them to take down your information.

With all centralized data storage methods, you’re at risk of losing access to your information at any time. It sounds dire, but it happens every day around the world in cases ranging from hacks to accidents to censorship.

But you have an important document you need to share. How can you avoid these issues?

The Promise of Decentralization

In contrast to more legacy solutions, decentralized data storage does not rely on information being stored in a single physical spot. Instead, the information is shared in many copies stored on many computers across the world using blockchain technology.

Because the information is distributed over a blockchain network, you can avoid many of the problems inherent to centralized storage:

• An accident or disaster can’t take down all the information, or even result in downtime, even if one computer is compromised.
• You don’t need to put all your trust in a single company and hope they won’t censor, remove, or charge you more to host your data.
• Cryptographic proofs let you verify the integrity yourself, and know that your data hasn’t been tampered with.
• With many hosts, it’s practically impossible to suppress a file once it’s widely replicated.

With decentralized data layers, you have a system that’s more trustworthy, reliable, censorship-resistant, and often less expensive. So your document can be saved and shared in a significantly more controlled and secure way.

Why is it called a “layer”? Just like layers of a cake, software layers are each unique parts that make up a better whole. By calling it a “layer,” you’re emphasizing that data storage and retrieval are decoupled from other tasks like processing transactions or running smart-contract code. This makes each layer simpler to design, optimize, and swap out without breaking the rest of the system.

How do decentralized data layers work?

At its core, a decentralized data layer is a peer-to-peer network whose sole job is to store, replicate, and serve data without any single central operator. It consists of a few parts.

Nodes and Network Topology

Every participant that runs the software is a node. They contribute disk space, bandwidth, and CPU to the network. (Often, but not always, this is just the computer on a network.)

Nodes connect into a structured peer-to-peer overlay (often using a Distributed Hash Table or DHT). Each node maintains a small routing table of neighbors. New nodes “bootstrap” (a term that refers to a node joining the wider network) by asking known seeds for peer addresses, then join the DHT.

Imagine you’re joining a new club but don’t know anyone. The club hands you a list of a few “welcome ambassadors” you can talk to. You text one of them, they introduce you to several other members, and pretty soon you’ve met enough people to feel fully integrated into the club’s social circle. That initial intro with the ambassador is the “bootstrap” step.

Storing and Replicating Data

When we talk about storing and replicating data in a decentralized data layer, we have two goals:

• Break the file into manageable pieces, each uniquely identifiable
• Ensure multiple peers hold copies, so that no single outage or censorship attempt can make it disappear

To achieve this, data gets chopped into shards and given a “cryptographic hash” to produce a unique content address. That hash is like a barcode that you can scan to know what information is in a particular packet without needing to go through all the information inside.

Before or after storing your data, nodes may submit cryptographic proofs to show they indeed hold the chunks they promised. Some examples include Heartbeat Checks, lightweight tests where a node proves periodic possession, and Data Availability Sampling, where clients randomly sample pieces of information to ensure the entire block’s data is available.

Retrieval & Indexing

Ready to access the information you’ve saved in a decentralized data layer? Your node will first ask the nodes around it where the information you want is stored, which helps develop a roadmap to reach the information.

Once you learn which peer(s) hold the chunk, you open a connection and download it. For especially large files, you can make parallel requests, where you pull parts of that data from different nodes at the same time to speed up the process.

When you’re done, your node will check the hash on the information you’ve accessed to make sure it matches the hash for the information you asked for, ensuring you have the right data and it hasn’t been changed.

Economic Incentives

Storing information, of course, takes processing power from the nodes in a network. Generally, networks like Golem Base pay for this shared power with cryptocurrency tokens.

When someone’s node successfully stores and serves data, they are rewarded with tokens (GLM) that can be used like any other currency. Some networks have a bidding system while others use fixed-rate markets.

Learn more about the economics of Golem Network.

This layered, modular approach lets developers focus on their app logic while the decentralized data layer guarantees their data stays available, verifiable, and censorship-resistant.

Using a Decentralized Data Layer: End to End

Let’s say you have a photo you’d like to share with your friends.

You use your network app to upload the photo. The app splits the photo into 20 shards that are each 1MB large. Each shard is given a hash to help identify what it is.

Your node announces to the other nodes in the network that it has a shard of this photo.

The shards are then shared with other nodes, effectively replicating your photo across all the nodes, adding to its durability in the event of catastrophe. (In an ideal scenario, the nodes are spread out across the entire planet.)

Next, when your friend wants to see the image, they can pull it down. The software then reassembles the different shards, verifying the hashes as it goes to ensure it hasn’t been tampered with.

New Developments for Decentralized Data Layers

This technology is still fairly new, and it’s expanding rapidly as more and more people get involved with Web3.

While it’s good to understand how decentralized data layers work and why they’re so powerful, you don’t need to be an expert in Web3 to be able to get something out of it. More and more businesses, researchers, developers, and private people are using decentralized data tech to store and share information in much more powerful, controllable ways.

Networks like Golem Base help people who are new to this area harness the power of decentralized data layers by combining this tech with straightforward Web2 interfaces you already know how to use.

Read our litepaper to learn more about how it works, or explore these other articles.