Skip to content

Introduction

Threads are the backbone of Textile's encrypted, recoverable, schema-based, and cross-application data storage. Think of a thread as a decentralized database of encrypted files and messages, shared between specific participants.

At the core of every thread is a secret. Only peers that possess the secret can decrypt thread content or follow linkages.

Hint

Unlike a blockchain, threads are not based around the idea of consensus. Instead, they follow an agent-centric approach similar to holochain. Each peer has authority over thread access-control and storage.

Because threads are simply a hash-chain of update messages, or blocks, they can represent any type of dataset. Some blocks point to off-chain data stored on IPFS. For example, a set of photos, a PDF, or even an entire website. Application developers are able to add structure to threads and make them interoperable with other applications by using schemas.

Threads are auto-magically synced with other account peers. For example, you may have one peer on your phone, and another on your laptop, both with access to the same account seed (more about accounts here). Threads can also be shared with other non-account peers (other users). In any case, each peer maintains a copy of its threads. A p2p messaging protocol keeps all the copies in sync.

The special hash-chain or graph structure of a thread allows them to be easily shared with other peers and backed-up to cafes. Given one message, you can find all the others.

Requirements

Threads are supposed to serve a decentralized cloud-like function for safely storing and retrieving data generated by applications for users, i.e., photos, messages, contacts, health data, etc. They were designed with the following requirements in mind:

  • Conflict resistant: Similar to ipfs-log, a thread should facilitate a resilient, distributed state, shared among multiple members (and/or devices).
  • Mud puddle resistant: There should be a way to safely backup a thread's state such that the owner can recover it.
  • Offline-first: Because most people access the internet primarily from mobile devices, threads must enable a UX that works well in scenarios where connectivity is spotty, and peers are continually coming and going from the network.
  • Secure: Peers must sign updates and encrypted with the shared key. Ideally, linkages should also be obscured by encryption.

Implementation

On the one hand, threads is a data model for representing a dataset as a hash-chain of updates. On the other hand, it's a protocol for orchestrating that state between peers.

Info

Threads aim to be language and platform agnostic. For this reason, Textile uses protocol buffers extensively because they are a "language-neutral, platform-neutral, extensible mechanism for serializing structured data".

Model

message Thread {
    string id                 = 1;
    string key                = 2;
    bytes sk                  = 3;
    string name               = 4;
    string schema             = 5;
    string initiator          = 6;
    Type type                 = 7;
    Sharing sharing           = 8;
    repeated string whitelist = 9;
    State state               = 10;
    string head               = 11;

    // Type controls read (R), annotate (A), and write (W) access
    enum Type {
        PRIVATE   = 0; // initiator: RAW, whitelist:
        READ_ONLY = 1; // initiator: RAW, whitelist: R
        PUBLIC    = 2; // initiator: RAW, whitelist: RA
        OPEN      = 3; // initiator: RAW, whitelist: RAW
    }

    // Sharing controls if (Y/N) a thread can be shared
    enum Sharing {
        NOT_SHARED  = 0; // initiator: N, whitelist: N
        INVITE_ONLY = 1; // initiator: Y, whitelist: N
        SHARED      = 2; // initiator: Y, whitelist: Y
    }

    // State indicates the loading state
    enum State {
        LOADING_TAIL = 0; // tail blocks are being loaded
        LOADED       = 1; // blocks are all loaded / paused
        LOADING_HEAD = 2; // head block is being loaded
    }

    // view info
    Block head_block  = 101;
    Node schema_node  = 102;
    int32 block_count = 103;
    int32 peer_count  = 104;
}

Orchestration

The orchestration of thread state between peers can be thought of as syncing a graph of blocks and files, which involves sending outbound updates and reading inbound updates. In practice, this is handled by a libp2p service.

Info

The thread service's protocol ID is /textile/threads/<version>.

Outbound updates

  1. A block describing an update to a thread is signed with the author's private key and encrypted with the thread secret.
  2. The resulting message is then sent over directly to known participants.
  3. If a peer is offline, the message is again encrypted with their public key and delivered to their cafe inbox(es).

Inbound updates

  1. Upon receiving an update, either directly or from a cafe inbox, a peer will decrypt, verify, and locally index the inner block.
  2. Depending on thread behavior, the peer will follow the graph, downloading linked messages from the network until it no longer finds new blocks.
  3. Ideally, the update can be fast-forwarded merged, but a deterministic 3-way merge may be needed.
  4. Files DAG node referenced by the graph also need to be synced. Sync is handled by first downloading the thread's schema, which contains pinning (storage) instructions for the DAG's inner files.
  5. Finally, a peer may send back an acknowledgment to the author containing its latest graph tip (or HEAD in git terminology) if it believes they have diverged.

Account Threads

Account peers can instruct each other to create and delete threads by communicating over a special internal account thread. An additional signature is used to handle this on outbound updates, which differentiates account and non-account peer messages.

The account thread is used to track account peers, profile information, and known contacts. Like normal threads, it is kept in sync between account peers. Read more about account sync here.

Snapshots

Thread snapshots enable account sync, recovery, and login from new devices. Snapshots are an encrypted representation of a thread. They contain metadata and the latest update hash, which is usually stored by cafes. Account peers will continuously search for, decrypt, and apply one another's snapshots to their local thread state.

Access Control

Control over thread access and sharing is handled by a combination of the type and sharing settings. An immutable member address "whitelist" gives the initiator fine-grained control. The table below outlines access patterns for the thread initiator and the whitelist members. An empty whitelist is taken to be "everyone", which is the default.

Thread type controls read (R), annotate (A), and write (W) access:

private   --> initiator: RAW, whitelist:
read_only --> initiator: RAW, whitelist: R
public    --> initiator: RAW, whitelist: RA
open      --> initiator: RAW, whitelist: RAW

Thread sharing style controls if (Y/N) a thread can be shared:

not_shared  --> initiator: N, whitelist: N
invite_only --> initiator: Y, whitelist: N
shared      --> initiator: Y, whitelist: Y`

Info

Access control will be moving to a more familiar, roll-based design in a future release. See this GitHub issue for more.

Blocks

Blocks are the raw components of a thread. Think of them as an append-only log of thread updates where each one is hash-linked to its parent(s), forming a tree. New / recovering peers can sync history by merely traversing the hash tree.

In practice, blocks are small (encrypted) protocol buffers, linked together by their IPFS CID (content id or hash). You can explore the protocol buffer definitions here.

There are several block types:

-  MERGE:    3-way merge added.
-  IGNORE:   A block was ignored.
-  FLAG:     A block was flagged.
-  JOIN:     Peer joined.
-  ANNOUNCE: Peer set username / avatar / inbox addresses
-  LEAVE:    Peer left.
-  TEXT:     Text message added.
-  FILES:    File(s) added.
-  COMMENT:  Comment added to another block.
-  LIKE:     Like added to another block.

Tip

The Blocks API can serve Graphviz dot files which can be converted to an image as seen below. See textile blocks --help for more)

A chat thread

An account thread shared between eight peers

Files

The FILES block type points to off-thread data in the form of an IPLD merkle DAG. These DAGs are described by and validated against the thread's schema.

Hint

Any data added to a thread ends up as a file, regardless of whether or not the source data was an actual file. For example, echoing a string into a thread results in a "file" containing that string.

Read more about files in the next section.

V2 Roadmap

Work on the next version of threads (v2) has started. The goals and individual issues for the next version can be seen here. The high-level goals have been copied below:

  • Move to a standalone GitHub repository.
  • Add the notion of "intent" to the data model so that other applications can understand a threads intended purpose.
  • Replace the protocol buffer based linkages with IPLD so other programs can more easily traverse threads.
  • Use a hybrid p2p and gossipsub orchestration pattern (details to come).
  • Break HEAD into multiple categories (peers, content, annotations)
  • Move to a role-based access model for accounts and applications.
  • Encrypt thread snapshots with one-off keys that can be provisioned w/ rolls.
  • Allow threads to be searchable and "followed" from a gateway.
  • Support a more traditional "feed" mechanism (1->many vs. some->some).
  • Use a single AES key as the thread secret, hash of key as ID.
  • In addition to being governed by access role, peers should not be required to download thread history.
  • Local thread interactions should use the more human-readable key and not id.

Feedback welcome!

Please let us know if you want to weigh in on the next version of threads.