Skip to content

Commit

Permalink
Update documentation from web3.storage to storacha
Browse files Browse the repository at this point in the history
  • Loading branch information
hakierka committed Aug 6, 2024
1 parent dea470b commit 5049121
Show file tree
Hide file tree
Showing 31 changed files with 209 additions and 209 deletions.
4 changes: 2 additions & 2 deletions src/pages/docs/concepts/_meta.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"ucans-and-web3storage": "UCANs and web3.storage",
"ucans-and-storacha": "UCANs and Storacha",
"content-addressing": "Content-addressing",
"upload-vs-store": "Upload vs. Store",
"architecture-options": "Architecture options",
Expand All @@ -13,4 +13,4 @@
"ucan": {
"display": "hidden"
}
}
}
20 changes: 10 additions & 10 deletions src/pages/docs/concepts/architecture-options.mdx
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
import { Callout } from 'nextra/components'

# Possible architectures using web3.storage to upload
# Possible architectures using Storacha to upload

UCAN opens up a number of options in how to integrate with w3up: Should you, the developer, own the Space? Should you delegate permissions to your users? Or should your user own their own Space? Broadly, there are three ways to integrate:

- **Client-server**: You (the developer) own the space, your users upload data to your application server, and you upload it to web3.storage from there.
- **Client-server**: You (the developer) own the space, your users upload data to your application server, and you upload it to Storacha from there.
- **Delegated**: You own the Space, but you provide your user with a delegation that allows them to upload directly to the service. This avoids proxying the upload through your backend (no egress from your infrastructure).
- **User-owned**: Your user owns the Space and registers it and they use it to upload directly with the service; if you want to instrument visibility into what they're uploading, you'll have to write separate code in your app for it.

In the How-tos section of the docs, we focused on the first two options, as they are the most typical today. However, you can implement each of these in a number of ways, but we talk through some considerations when implementing a given option.

## Client-server

In this set up, you (the developer) own the space, your users upload data to your application server, and you upload it to web3.storage from there.
In this set up, you (the developer) own the space, your users upload data to your application server, and you upload it to Storacha from there.

```mermaid
sequenceDiagram
participant User
w3up-client in backend-\>\>w3up-client in backend: Client set with Agent with delegation from Space
User-\>\>w3up-client in backend: Upload data
w3up-client in backend-\>\>web3.storage w3up service: Upload data
w3up-client in backend-\>\>storacha w3up service: Upload data
```

You'll need a registered Space, and your client in the backend to have a delegation to use the Space.
Expand All @@ -40,12 +40,12 @@ In this set up, you own the Space, but you provide your user with a delegation t
sequenceDiagram
participant w3up-client in user
participant w3up-client in backend
participant web3.storage w3up service
participant storacha w3up service
w3up-client in backend-\>\>w3up-client in backend: Client created with Agent and delegation from Space
w3up-client in user-\>\>w3up-client in user: Client instantiated with default Agent
w3up-client in user-\>\>w3up-client in backend: Request delegation with user's Agent DID
w3up-client in backend-\>\>w3up-client in user: Send delegation from Space to user's Agent DID
w3up-client in user-\>\>web3.storage w3up service: Upload data
w3up-client in user-\>\>storacha w3up service: Upload data
```

Typically `w3up-client` will be running in your end-user's client code, as well as backend code that's able to generate UCANs that delegate the ability to upload and pass them to your users (e.g., `w3up-client` running in a serverless worker).
Expand Down Expand Up @@ -144,11 +144,11 @@ In this set up your user owns the Space and registers it and they use it to uplo
sequenceDiagram
participant User
participant Application backend
participant web3.storage w3up service
participant storacha w3up service
Application backend-\>\>User: Front end code that includes w3up-client
User-\>\>web3.storage w3up service: (If needed) Create Space and register it
User-\>\>web3.storage w3up service: (If needed) Use Agent email verification to "log in" to Space
User-\>\>web3.storage w3up service: Upload data using w3up-client
User-\>\>storacha w3up service: (If needed) Create Space and register it
User-\>\>storacha w3up service: (If needed) Use Agent email verification to "log in" to Space
User-\>\>storacha w3up service: Upload data using w3up-client
```

If you want your user to own their own Space, you'll typically be relying on the `w3up-client` methods to create a Space, authorize the Space, and authorize the Agent on the user-side; afterwards they can run any of the upload methods.
Expand Down
24 changes: 12 additions & 12 deletions src/pages/docs/concepts/car.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,28 @@
# Working with Content Archive (CAR) files

When you upload files to web3.storage using the client library, your data is converted into a graph of data structures, which are then packed into a format called a Content Archive (CAR) before being sent to the web3.storage service.
When you upload files to Storacha using the client library, your data is converted into a graph of data structures, which are then packed into a format called a Content Archive (CAR) before being sent to the Storacha service.

For most use cases, you never need to know about this process, as the conversion happens behind the scenes when using the client library. However, you might want to know more about how to manipulate CAR files directly, especially if you're using `upload` and `store` separately in web3.storage.
For most use cases, you never need to know about this process, as the conversion happens behind the scenes when using the client library. However, you might want to know more about how to manipulate CAR files directly, especially if you're using `upload` and `store` separately in Storacha.

## What is a Content Archive?

The [Content Archive format](https://ipld.io/specs/transport/car/) is a way of packaging up [content addressed data](https://web3.storage/docs/concepts/content-addressing/) into archive files that can be easily stored and transferred. You can think of them like [TAR files](https://en.wikipedia.org/wiki/Tar_\(computing\)) that are designed for storing collections of content addressed data.
The [Content Archive format](https://ipld.io/specs/transport/car/) is a way of packaging up [content addressed data](https://storacha.network/docs/concepts/content-addressing/) into archive files that can be easily stored and transferred. You can think of them like [TAR files](https://en.wikipedia.org/wiki/Tar_\(computing\)) that are designed for storing collections of content addressed data.

The type of data stored in CARs is defined by [IPLD](https://ipld.io/), or InterPlanetary Linked Data. IPLD is a specification and set of implementations for structured data types that can link to each other using a hash-based Content Identifier (CID). Data linked in this way forms a Directed Acyclic Graph, or DAG, and you'll likely see a few references to DAGs in the documentation for IPLD and IPFS.

IPFS files are one example of IPLD data, but IPLD can also be used to access data from Ethereum, Git, and other hash-addressed systems. You can also use IPLD as a general purpose format for your structured data, sort of like a Web3-flavored JSON. See Advanced IPLD formats below for more information.

## CARs and web3.storage
## CARs and Storacha

When the web3.storage client and CLI pack up regular files into a CAR to store on IPFS, the CAR contains data encoded in the same format used by IPFS when importing files using the command line or other IPFS APIs.
When the Storacha client and CLI pack up regular files into a CAR to store on IPFS, the CAR contains data encoded in the same format used by IPFS when importing files using the command line or other IPFS APIs.

This format uses an IPLD "codec" called [dag-pb](https://ipld.io/docs/codecs/known/dag-pb/), which uses [Protocol Buffers](https://developers.google.com/protocol-buffers) to encode an object graph. Inside the graph are [UnixFS objects](https://docs.ipfs.io/concepts/file-systems/#unix-file-system-unixfs) that describe the files and their contents. We do this client-side for a few reasons.

First, formatting everything on the client allows us to calculate the root Content Identifier for the data you're uploading before we send any data to the remote service. This means that you can compare the CID returned by the web3.storage service to the one you calculated locally, and you don't have to trust the service to do the right thing.
First, formatting everything on the client allows us to calculate the root Content Identifier for the data you're uploading before we send any data to the remote service. This means that you can compare the CID returned by the Storacha service to the one you calculated locally, and you don't have to trust the service to do the right thing.

Another reason to use CARs is to support large files, which would otherwise hit size limits on the web3.storage backend platform. The data in a CAR is already chunked into small blocks, which makes CARs easy to shard into small pieces that can be uploaded in batches. This also enables the web3.storage platform to get larger content into Filecoin deals.
Another reason to use CARs is to support large files, which would otherwise hit size limits on the Storacha backend platform. The data in a CAR is already chunked into small blocks, which makes CARs easy to shard into small pieces that can be uploaded in batches. This also enables the Storacha platform to get larger content into Filecoin deals.

CAR files are a format that pretty much any IPFS tool or implementation can interact with. You can export data from your personal IPFS node into a CAR file and upload it to web3.storage using `w3 up --car` or `client.uploadCAR`. As a result, we dive into the various ways you might interact with CAR files.
CAR files are a format that pretty much any IPFS tool or implementation can interact with. You can export data from your personal IPFS node into a CAR file and upload it to Storacha using `w3 up --car` or `client.uploadCAR`. As a result, we dive into the various ways you might interact with CAR files.

## Command line tools

Expand Down Expand Up @@ -85,17 +85,17 @@ ipfs dag import path/to/input.car

### `ipfs-car`

The [`ipfs-car`](https://github.com/web3-storage/ipfs-car) package includes library functions for packing and unpacking files into CARs, using the IPFS UnixFs data model. The library includes the same functionality as the ipfs-car command line utility [described above](https://web3.storage/docs/how-tos/work-with-car-files/#ipfs-car).
The [`ipfs-car`](https://github.com/web3-storage/ipfs-car) package includes library functions for packing and unpacking files into CARs, using the IPFS UnixFs data model. The library includes the same functionality as the ipfs-car command line utility [described above](https://storacha.network/docs/how-tos/work-with-car-files/#ipfs-car).

See the `ipfs-car` [README](https://github.com/web3-storage/ipfs-car#api) for API documentation and usage examples.

### `@ipld/car`

The [`@ipld/car`](https://github.com/ipld/js-car) package contains the main JavaScript implementation of the CAR specification and is used by ipfs-car under the hood. If you want to store non-file data using [advanced IPLD formats](https://web3.storage/docs/how-tos/work-with-car-files/#advanced-ipld-formats), you should use @ipld/car directly.
The [`@ipld/car`](https://github.com/ipld/js-car) package contains the main JavaScript implementation of the CAR specification and is used by ipfs-car under the hood. If you want to store non-file data using [advanced IPLD formats](https://storacha.network/docs/how-tos/work-with-car-files/#advanced-ipld-formats), you should use @ipld/car directly.

@ipld/car also provides the CarReader interface used by the web3.storage client's [putCar](https://web3.storage/docs/reference/js-client-library/#store-car-files)[method](https://web3.storage/docs/reference/js-client-library/#store-car-files).
@ipld/car also provides the CarReader interface used by the Storacha client's [putCar](https://storacha.network/docs/reference/js-client-library/#store-car-files)[method](https://storacha.network/docs/reference/js-client-library/#store-car-files).

Here's a simple example of loading a CAR file from a Node.js stream and storing it with web3.storage:
Here's a simple example of loading a CAR file from a Node.js stream and storing it with Storacha:

```js
import fs from 'node:fs'
Expand Down
22 changes: 11 additions & 11 deletions src/pages/docs/concepts/content-addressing.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,33 @@
# Content addressing

web3.storage's decentralized file storage relies on _content addressing_ to find, reference, and retrieve your files on the network. Content addressing is a technique for organizing and locating data in a system in which the key used to locate content is derived from the content itself, rather than its location. While you don't need to understand content addressing to be able to incorporate web3.storage in your apps and services, if you're curious about what's going on under the hood, read on.
Storacha's decentralized file storage relies on _content addressing_ to find, reference, and retrieve your files on the network. Content addressing is a technique for organizing and locating data in a system in which the key used to locate content is derived from the content itself, rather than its location. While you don't need to understand content addressing to be able to incorporate sSoracha in your apps and services, if you're curious about what's going on under the hood, read on.

## The basic problem

Consider what happens when you resolve a link like `web3.storage/docs/concepts/content-addressing`. First, your operating system queries a global shared key-value store, split into many domains — you may know this as the Domain Name System (DNS). The DNS returns an IP address that your network card can use to send HTTP requests over the network, where this site's naming conventions turn the key `/docs/concepts/content-addressing` into a response payload.
Consider what happens when you resolve a link like `storacha.network/docs/concepts/content-addressing`. First, your operating system queries a global shared key-value store, split into many domains — you may know this as the Domain Name System (DNS). The DNS returns an IP address that your network card can use to send HTTP requests over the network, where this site's naming conventions turn the key `/docs/concepts/content-addressing` into a response payload.

The problem is, components of an address like `web3.storage/docs/concepts/content-addressing` are _mutable_, meaning they can change over time. In the context of the web, where _everything_ is mutable and dynamic, this is just the way it's always been. As a result, [link rot](https://en.wikipedia-on-ipfs.org/wiki/Link_rot) is just something we've all learned to live with.
The problem is, components of an address like `storacha.network/docs/concepts/content-addressing` are _mutable_, meaning they can change over time. In the context of the web, where _everything_ is mutable and dynamic, this is just the way it's always been. As a result, [link rot](https://en.wikipedia-on-ipfs.org/wiki/Link_rot) is just something we've all learned to live with.

## CIDs: Location-independent, globally unique keys

However, thanks to content addressing, link rot may become a thing of the past. A content-addressed system such as web3.storage is like our key-value-based DNS, with one significant difference: You no longer get to choose the keys. Instead, the keys are derived directly from the file contents using an algorithm that will always generate the same key for the same content.
However, thanks to content addressing, link rot may become a thing of the past. A content-addressed system such as Storacha is like our key-value-based DNS, with one significant difference: You no longer get to choose the keys. Instead, the keys are derived directly from the file contents using an algorithm that will always generate the same key for the same content.

As a result, we no longer need to coordinate among multiple writers to our store by splitting the key space into domains and locations on file systems. There's now one universal domain: the domain of all possible values. If multiple people add the same value, there's no collision in the key space. They just each get the same key back from the put method, with one additional benefit: The availability and performance of retrievals on the network is increased. This gives our keys _location independence_. There's one other important result: Each individual key is a unique signature for the data itself, ensuring _verifiability_ that the key matches the content and the content hasn't been altered.

This type of key is called a _content identifier (CID)_. Once you know the CID of a file on the web3.storage network, you have all you need for the network to locate and return the file back to you.
This type of key is called a _content identifier (CID)_. Once you know the CID of a file on the Storacha network, you have all you need for the network to locate and return the file back to you.

## web3.storage CIDs under the hood
## Storacha CIDs under the hood

web3.storage uses CIDs to make its decentralized file storage work, with help from [IPFS](https://ipfs.io/) for locating files and making sure they're always available.
Storacha uses CIDs to make its decentralized file storage work, with help from [IPFS](https://ipfs.io/) for locating files and making sure they're always available.

Content addressing is the basis of the peer-to-peer hypermedia protocol IPFS (the InterPlanetary File System), which web3.storage uses to locate files. When web3.storage stores your data on IPFS, it can be retrieved from any IPFS node that has a copy of that data. This can make data transfers more efficient and reduce the load on any single node. As each user fetches a piece of data, they keep a local copy around to help other users who might request it later.
Content addressing is the basis of the peer-to-peer hypermedia protocol IPFS (the InterPlanetary File System), which Storacha uses to locate files. When Storacha stores your data on IPFS, it can be retrieved from any IPFS node that has a copy of that data. This can make data transfers more efficient and reduce the load on any single node. As each user fetches a piece of data, they keep a local copy around to help other users who might request it later.

In addition to web3.storage making it easy to get your data onto the content-addressed IPFS network, it also provides long-term persistence for your files using the decentralized Filecoin storage network. The Filecoin network incentivizes participants to provide storage space for files on the network. By combining IPFS and Filecoin storage into one easy-to-use service, web3.storage makes it simple to store, locate, and retrieve your files on the decentralized web.
In addition to Storacha making it easy to get your data onto the content-addressed IPFS network, it also provides long-term persistence for your files using the decentralized Filecoin storage network. The Filecoin network incentivizes participants to provide storage space for files on the network. By combining IPFS and Filecoin storage into one easy-to-use service, Storacha makes it simple to store, locate, and retrieve your files on the decentralized web.

There are two types of CIDs that web3.storage interacts with:
There are two types of CIDs that Storacha interacts with:

- Content CIDs: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are generally prefixed by `bafy…`. This is what the vast majority of IPFS users will be using.
- Shard CIDs: The CID of the serialized shards of data itself that are produced client-side, sent to web3.storage, and stored. These CIDs are generally prefixed by `bag…`.
- Shard CIDs: The CID of the serialized shards of data itself that are produced client-side, sent to Storacha, and stored. These CIDs are generally prefixed by `bag…`.

In the vast majority of cases, users should focus on content CIDs, as this is what they'll be using to fetch their content. If you stick with using the recommended client and CLI methods, then you won't really have to ever worry about the shard CIDs.

Expand Down
Loading

0 comments on commit 5049121

Please sign in to comment.