forked from PigeonProtocolConsortium/Protocol-Spec
Compare commits
21 Commits
| Author | SHA1 | Date |
|---|---|---|
 Netscape Navigator
|
9eb575f54f | |
|
Netscape Navigator
|
031a7fdd9e | |
|
Netscape Navigator
|
e875873ec9 | |
|
Netscape Navigator
|
7a45cc7bbd | |
|
Netscape Navigator
|
a4a80f9b94 | |
 MIIRC
|
4a10d39d83 | |
|
MIIRC
|
75e1952013 | |
|
Netscape Navigator
|
4f12ceab81 | |
|
Netscape Navigator
|
683f869e8d | |
|
Netscape Navigator
|
f3f94d969a | |
|
Netscape Navigator
|
d4be66d381 | |
|
Netscape Navigator
|
7d7edecb5c | |
|
Netscape Navigator
|
989809bfe9 | |
|
Netscape Navigator
|
606eedf8af | |
|
Netscape Navigator
|
cda89e9530 | |
|
Netscape Navigator
|
5d433b078c | |
|
Netscape Navigator
|
863b1cf181 | |
|
Netscape Navigator
|
dd3c4614d6 | |
|
Netscape Navigator
|
e7bbdf8e90 | |
|
Netscape Navigator
|
47b01a2d65 | |
Jesse Laprade
|
202e07f573 |
416
README.md
416
README.md
|
|
@ -1,138 +1,199 @@
|
|||
|
||||
|
||||
## Pigeon - The Off Grid Peer-to-Peer Protocol
|
||||
# The Pigeon protocol
|
||||
|
||||
* Overview (You are Here)
|
||||
* [Frequently Asked Questions](faq.md)
|
||||
* [Roadmap](roadmap.md)
|
||||
* [Developer Docs and Specification](dev_docs.md)
|
||||
* [Ideas and Features](ideas.md)
|
||||
The Pigeon protocol is an off-grid, serverless, peer-to-peer protocol
|
||||
for building software that works on poor internet connections, or
|
||||
entirely offline.
|
||||
|
||||
# Who Uses Pigeon? Why?
|
||||
|
||||
Pigeon is used by software developers building peer-to-peer apps. It is particularly useful for peer-to-peer apps that have poor internet connectivity or no connectivity at all. Pigeon provides these developers a way to store and replicate (backup) their nodes data while eliminating the need for networks like the internet and preventing interference from malicious third parties.
|
||||
# Project Status
|
||||
|
||||
Every user in the mesh has a local database. Entries in the database are cryptographically signed. Because the entries are signed, you can share your database entries anywhere (CD-R, email, public forums, [USB dead drops](https://en.wikipedia.org/wiki/USB_dead_drop)) knowing that the message has not been altered by malicious third parties.
|
||||
HIBERNATION. Pigeon was an exploration of ideas that I embarked on in 2020. Now (2021) I am changing my focus. If more people are interested in the project I might start work on it again. Please let me know by raising an issue. For now, I am shifting my focus to other areas unless interest in the project changes.
|
||||
|
||||
When you "follow" a peer, you keep a backup copy of their database on your machine and update the data occasionally via the use of "bundle" files. This replicated data can then be "gossiped" to peers that need to read the data when the other peer is not available.
|
||||
# Table of contents
|
||||
|
||||
In summary, Pigeon is:
|
||||
- [Introduction](#introduction)
|
||||
- [Why build software that follows the Pigeon protocol?](#why-build-software-that-follows-the-pigeon-protocol)
|
||||
- [How does the Pigeon protocol benefit users?](#how-does-the-pigeon-protocol-benefit-users)
|
||||
- [Data is backed up by default](#data-is-backed-up-by-default)
|
||||
- [Data can be shared anywhere](#data-can-be-shared-anywhere)
|
||||
- [Data is tamper-resistant](#data-is-tamper-resistant)
|
||||
- [Data is always available](#data-is-always-available)
|
||||
- [Use-case scenarios](#use-case-scenarios)
|
||||
- [Software implementation ideas](#software-implementation-ideas)
|
||||
- [Implementations](#implementations)
|
||||
- [How does the Pigeon protocol work?](#how-does-the-pigeon-protocol-work)
|
||||
- [What do Pigeon messages look like as data?](#what-do-pigeon-messages-look-like-as-data)
|
||||
- [Constraints and Design Philosophy](#constraints-and-design-philosophy)
|
||||
- [What the Pigeon protocol is not trying to address](#what-the-pigeon-protocol-is-not-trying-to-address)
|
||||
- [How the Pigeon protocol differs from Sneakernet](#how-the-pigeon-protocol-differs-from-sneakernet)
|
||||
- [How the Pigeon protocol differs From Secure Scuttlebutt](#how-the-pigeon-protocol-differs-from-secure-scuttlebutt)
|
||||
- [Influences](#influences)
|
||||
- [Getting involved with the Pigeon protocol development](#getting-involved-with-the-pigeon-protocol-development)
|
||||
- [For more information](#for-more-information)
|
||||
|
||||
* decentralized (peer-to-peer)
|
||||
* replicated by default
|
||||
* tamper and forgery resistant
|
||||
* delay tolerant and offline-first
|
||||
* built for [sneakernet](https://en.wikipedia.org/wiki/Sneakernet) from the ground up
|
||||
# Introduction
|
||||
|
||||
# Use Cases and Real-World Applications
|
||||
This document will describe why you may want to build software that
|
||||
follows the Pigeon protocol, how the protocol works, and how you can
|
||||
write software that follows this protocol.
|
||||
|
||||
Pigeon can serve a number of use cases. Below are some examples:
|
||||
This document provides a reference to the Pigeon protocol
|
||||
specifications, and is written for developers.
|
||||
|
||||
* Systems with low connectivity or uptime such as remote sensor logging, maritime systems, solar systems with intermittent power, IoT systems with poor network connectivity.
|
||||
* Store-and-forward message gateways, such as a [data mule](https://en.wikipedia.org/wiki/Data_mule).
|
||||
* Censorship resistant applications, such as peer-to-peer messaging and blogging.
|
||||
* [Delay tolerant networking](https://en.wikipedia.org/wiki/Delay-tolerant_networking)
|
||||
* Applications that require a high level of data-integrity or auditing.
|
||||
* Delay-tolerant peer-to-peer social networks, games, file sharing etc...
|
||||
* Time series data storage
|
||||
# Why build software that follows the Pigeon protocol?
|
||||
|
||||
Framed in a more applied sense, Pigeon could theoretically support applications and use cases such as:
|
||||
Software that follows the Pigeon protocol can operate online or
|
||||
offline, regardless of connectivity quality. This allows you to focus
|
||||
on software development, instead of worrying about network types and
|
||||
quality, or third-party services.
|
||||
|
||||
* A messenger app
|
||||
* An IoT data logger
|
||||
* A newsgroup / [NNTP](https://en.wikipedia.org/wiki/Network_News_Transfer_Protocol) analog
|
||||
* A turn-based board game
|
||||
* A microblogging social network
|
||||
* An e-commerce application for P2P cryptocurrency users
|
||||
* Secure Scuttlebutt import / export / gateway tool
|
||||
* A social mapping / point-of-interest sharing site
|
||||
* Bluetooth file sharing app
|
||||
* A GUI database browser for developers that wish to use the protocol for log storage or as a time series DB
|
||||
* Sync files over actual pigeons, possibly soliciting help from world famous boxer and pigeon racing enthusiast Mike Tyson
|
||||
# How does the Pigeon protocol benefit users?
|
||||
|
||||
## Help Wanted
|
||||
The Pigeon protocol outlines several ways users can benefit from using
|
||||
software that follows this protocol.
|
||||
|
||||
If you wish to become involved with protocol development, there are a few areas we need help in currently. Email us for more information.
|
||||
This section consists of the following subsections:
|
||||
|
||||
* Documentation editors, proof readers and feedback. The feedback really matters. Please email me your opinions and ideas. I will reply to all messages.
|
||||
* We need real-world applications to be built using the protocol! We are happy to assist you along the way.
|
||||
* We need a BNF grammar for Pigeon messages.
|
||||
- [Data is backed up by default](#data-is-backed-up-by-default)
|
||||
- [Data can be shared anywhere](#data-can-be-shared-anywhere)
|
||||
- [Data is tamper-resistant](#data-is-tamper-resistant)
|
||||
- [Data is always available](#data-is-always-available)
|
||||
|
||||
## Implementations
|
||||
## Data is backed up by default
|
||||
|
||||
There is currently one working implementation available [here](https://tildegit.org/PigeonProtocolConsortium/pigeon_ruby).
|
||||
Software that follows the Pigeon protocol allows users to store data
|
||||
on their nodes, and copy their data to other nodes. By copying data to
|
||||
other nodes, this not only backs up data, but also provides users with
|
||||
data they can access offline.
|
||||
|
||||
## Data can be shared anywhere
|
||||
|
||||
Every user in the mesh has a local database. You can share your
|
||||
database entries anywhere, such as CD-R, email, public forums, and
|
||||
[USB dead drops](https://en.wikipedia.org/wiki/USB_dead_drop).
|
||||
|
||||
## Data is tamper-resistant
|
||||
|
||||
Entries in the database are cryptographically signed. Because the
|
||||
entries are signed, you don't have to worry about malicious third
|
||||
parties tampering with your data.
|
||||
|
||||
## Data is always available
|
||||
|
||||
Any data the user is currently accessing is already available offline,
|
||||
and in case of a sudden internet outage, or poor connectivity issues,
|
||||
the access to the data is not affected, because the data doesn't rely
|
||||
on an internet connection.
|
||||
|
||||
You can "follow" peers, by copying their database to your
|
||||
machine. Updates are performed by retrieving newer bundles from your
|
||||
peers. This allows you to share your peers' bundles on behalf of
|
||||
them. This is useful when the peer is unavailable and another peer
|
||||
wants their bundle.
|
||||
|
||||
The act of sharing peers' data on behalf of other peers is called
|
||||
"gossiping".
|
||||
|
||||
# Use-case scenarios
|
||||
|
||||
The Pigeon protocol may be useful in scenarios such as:
|
||||
|
||||
* Systems with low connectivity or uptime such as remote sensor
|
||||
logging, maritime systems, solar systems with intermittent power,
|
||||
IoT systems with poor network connectivity.
|
||||
* Store-and-forward message gateways, such as a
|
||||
[data mule](https://en.wikipedia.org/wiki/Data_mule).
|
||||
* Censorship resistant applications, such as peer-to-peer messaging
|
||||
and blogging.
|
||||
* [Delay tolerant networking](https://en.wikipedia.org/wiki/Delay-tolerant_networking)
|
||||
* Applications that require a high level of data-integrity or
|
||||
auditing.
|
||||
* Delay-tolerant peer-to-peer social networks, games, file sharing
|
||||
etc...
|
||||
* Time series data storage
|
||||
|
||||
# Software implementation ideas
|
||||
|
||||
* A messenger application
|
||||
* An IoT data logger
|
||||
* A newsgroup or [NNTP](https://en.wikipedia.org/wiki/Network_News_Transfer_Protocol) analog
|
||||
* A turn-based board game
|
||||
* A microblogging social network
|
||||
* An e-commerce application for P2P cryptocurrency users
|
||||
* A Secure Scuttlebutt import / export / gateway tool
|
||||
* A social mapping / point-of-interest sharing site
|
||||
* A Bluetooth file sharing application
|
||||
* A graphical database browser for developers that want to use the
|
||||
protocol for log storage or as a time series database
|
||||
* A synchronization system over actual pigeons, possibly soliciting
|
||||
help from the world-famous boxer and pigeon-racing enthusiast, Mike
|
||||
Tyson
|
||||
|
||||
# Implementations
|
||||
|
||||
There is currently one working implementation available
|
||||
[here](https://github.com/PigeonProtocolConsortium/pigeon-cli-ruby).
|
||||
A long term goal is to support all major platforms and languages.
|
||||
|
||||
## Prior Art
|
||||
# How does the Pigeon protocol work?
|
||||
|
||||
Pigeon borrows many of the ideas set forth by the [Secure Scuttlebutt protocol](https://ssbc.github.io/scuttlebutt-protocol-guide/). It is my opinion that SSB is one of the most innovative protocols created in recent years. Without the research and efforts of the [Secure Scuttlebutt Consortium](https://github.com/ssbc), this project would not be possible, so a big thanks goes out to all the people who make SSB possible.
|
||||
Each node in a social network of peers has a local "log". The log is an
|
||||
append-only feed of messages written in an ASCII-based serialization
|
||||
format.
|
||||
|
||||
I've also been inspired by the compactness and minimalism of [SQLite, which should serve as a role model for all of us](https://www.sqlite.org/talks/wroclaw-20090310.pdf).
|
||||
Messages are signed with a secret key to validate a message's
|
||||
integrity and to prevent tampering by malicious peers.
|
||||
|
||||
In many ways, this protocol can be considered an amalgam of the best ideas from both SQLite and Secure Scuttlebutt.
|
||||
Nodes in the social network "follow" other logs from peers. Nodes replicate the
|
||||
logs of their peers and "gossip" information about peers across the
|
||||
social network.
|
||||
|
||||
Pigeon also borrows the [Lipmaa link concept seen in the Bamboo protocol](https://github.com/AljoschaMeyer/bamboo) to allow for partial verification of message feeds.
|
||||
Gossip information is packaged into "bundles" which contain backups of
|
||||
peer logs in a format that can be transmitted through sneakernet,
|
||||
direct serial connection, or any high-throughput medium, regardless of
|
||||
latency.
|
||||
|
||||
## How Pigeon Differs from Traditional Sneakernet
|
||||
|
||||
[Sneakernet](https://en.wikipedia.org/wiki/Sneakernet) is a protocol used by ancient civilizations to exchange files between computers with limited internet connectivity. Although Pigeon protocol messages can be exchanged over sneakernet, Pigeon is _not_ sneakernet. Sneakernet messages by themselves are not tamper resistant, nor do they provide redundant backup via peers. In contrast, a Pigeon protocol message is redundantly replicated _beyond_ its intended recipient to neighboring peers ("friend of a friend") via gossip and uses cryptography to guarantee that a message's content has not been altered by a third party.
|
||||
|
||||
In summary, Pigeon protocol offers benefits above what a traditional sneakernet can provide. A Pigeon protocol message:
|
||||
|
||||
* Is automatically backed up by peers and peers-of-peers (gossip).
|
||||
* Cannot be forged by malicious parties.
|
||||
* Cannot be altered by anyone except the author.
|
||||
|
||||
## How Pigeon Differs From Secure Scuttlebutt (SSB)
|
||||
|
||||
As mentioned, Pigeon was inspired mostly by the work of Secure Scuttlebutt. Pigeon takes a different approach in a few areas, however.
|
||||
|
||||
1. No reliance on networking in the core library. Although SSB is theoretically able to support Sneakernet-only operation, it is difficult in practice due to reliance on UDP, TCP, and in the case of pubs, DNS.
|
||||
1. Pigeon uses a custom key/value serialization format instead of JSON. This has two benefits:
|
||||
* Serialization and signing is much simpler. Indentation and whitespace are less likely to cause verification problems.
|
||||
* Unlike JSON, pigeon messages do not allow nesting, which promotes simplified message schemas.
|
||||
1. Pigeon uses Crockford flavored Base32 rather than URL safe Base64. This makes it easier to support old or low powered systems and is easier to support FAT16 / embedded systems.
|
||||
1. Pigeon was designed for portability from the beginning. It has a small enough conceptual overhead that it will actually be possible to support platforms other than NodeJS. Complicated features (like network support) are ignored in favor of an easy-to-implement standard
|
||||
1. It uses Lipmaa links, so you can verify a feed without downloading all messages. This was inspired by the work of the Bamboo protocol.
|
||||
|
||||
|
||||
## How Does It Work?
|
||||
|
||||
Each node in a swarm of peers has a local "log". The log is an append only feed of messages written in an ASCII-based serialization format. Messages are signed with a secret key to validate a message's integrity and to prevent tampering by malicious peers. Nodes in the swarm "follow" other logs from peers of interest. Nodes always replicate the logs of their peers and "gossip" information about peers across the swarm. Gossip information is packaged into "bundles" which contain backups of peer logs in an efficient format that can be easily transmitted via sneakernet, direct serial connection, or any high throughput medium, regardless of latency.
|
||||
|
||||
Log synchronization via Sneakernet is the main use case for Pigeon messages to be transmitted. SD Cards sent via postal mail offer an excellent medium for transmission of Pigeon messages, although any data transfer medium is theoretically possible.
|
||||
Log synchronization through Sneakernet is the main use case for Pigeon
|
||||
messages to be transmitted. SD Cards sent via postal mail offer an
|
||||
excellent medium for transmission of Pigeon messages, although any
|
||||
data transfer medium is possible.
|
||||
|
||||
|
||||
|
||||
## What Messages Look Like
|
||||
# What do Pigeon messages look like as data?
|
||||
|
||||
Messages use a custom ASCII-based encoding scheme (shown below). Although they are mostly human readable, they are intended to be parsed by third party applications rather than written by hand.
|
||||
Pigeon messages use a custom ASCII-based encoding scheme, as seen in
|
||||
**Example 1** and **Example 2** below.
|
||||
|
||||
Example 1:
|
||||
**Note**: Although the message encoding scheme is human-readable the
|
||||
messages should be parsed by third-party applications rather than
|
||||
written by hand.
|
||||
|
||||
**Example 1**:
|
||||
|
||||
```
|
||||
author USER.R68Q26P1GEFC0SNVVQ9S29SWCVVRGCYRV7D96GAN3XVQE3F9AZJ0
|
||||
depth 0
|
||||
kind my_example
|
||||
lipmaa NONE
|
||||
kind my_blog
|
||||
prev NONE
|
||||
|
||||
foo:"bar"
|
||||
current_status:"Another great day. Nice weather, too!"
|
||||
|
||||
signature 2VMAG4SCX5RHVBKCB1RNZCB0AJN4WN6FEMS7W9FM1CVYSZXMX7CPQFCDPYEKCTGG91Y1YSGY4G5K8XAGQ67HEPDFRMRYQHWQBATAC2R
|
||||
|
||||
```
|
||||
|
||||
Example 2:
|
||||
**Example 2**:
|
||||
|
||||
```
|
||||
author USER.R68Q26P1GEFC0SNVVQ9S29SWCVVRGCYRV7D96GAN3XVQE3F9AZJ0
|
||||
depth 1
|
||||
kind another_example
|
||||
lipmaa NONE
|
||||
kind like
|
||||
prev TEXT.6CBA4J3756A5SNM1W1GHNCTT9EG95ZP3ZMAT5Z1EJP7TXMNNVZC0
|
||||
|
||||
bar:"baz"
|
||||
liked_post_multihash:TEXT.6CBA4J3756A5SNM1W1GHNCTT9EG95ZP3ZMAT5Z1EJP7TXMNNVZC0
|
||||
|
||||
signature Y34Q47V0BY370RM5KWGRJRN9HFNGJN0C3DEYVB2V2476CW9RN5HD4XD7KMQ6T4T42N36R5P3XX6E3FYEWVZR25AVCF6KQPZHJP6EM10
|
||||
|
||||
|
|
@ -140,36 +201,173 @@ signature Y34Q47V0BY370RM5KWGRJRN9HFNGJN0C3DEYVB2V2476CW9RN5HD4XD7KMQ6T4T42N36R5
|
|||
|
||||
|
||||
|
||||
## Protocol Maturity
|
||||
For examples of how messages are handled, see the Ruby-implementation tutorial
|
||||
[here](https://github.com/PigeonProtocolConsortium/pigeon-cli-ruby/blob/main/ruby_tutorial.md)
|
||||
|
||||
The [first working implementation of a Pigeon protocol client](https://tildegit.org/PigeonProtocolConsortium/pigeon_ruby) is complete. We are temporarily halting feature development to focus on documentation, bug fixes and outreach. Contact us to get involved.
|
||||
# Constraints and Design Philosophy
|
||||
|
||||
## Constraints and Design Philosophy
|
||||
* Offline-first means offline-only. Never incorporate TCP or UDP
|
||||
features. TCP- or UDP-related concerns must be handled by
|
||||
higher-level protocols or by application developers. This is to
|
||||
ensure the protocol is always an option for off-grid usage.
|
||||
|
||||
<!-- I wasn't sure what the "do not make ... unless it would hurt portability" meant below -->
|
||||
* Natural is better than simple. Convention over configuration. Do
|
||||
not make plugins for common use cases unless it would hurt
|
||||
portability.
|
||||
|
||||
* Prefer a monolithic internal structure. Avoid external dependencies
|
||||
except for limited use cases (Eg: crypto libs). Do not break things
|
||||
into smaller pieces until there are at least three real-world
|
||||
reasons to do so. Decoupling a library into a package for only 2
|
||||
use cases is not acceptable.
|
||||
|
||||
* Maintain ecosystem diversity by having a protocol that can be
|
||||
easily and entirely ported to new languages and platforms.
|
||||
|
||||
* No singletons. No signing authorities, no servers of any kind, even
|
||||
locally, no differentiation between peers (eg: no "super peers").
|
||||
|
||||
* Configuration is always a design compromise. We will allow a limit
|
||||
of 10 configuration options for all eternity. These are simple
|
||||
key/value pairs. No nesting, no namespacing, no dots, no dashes, no
|
||||
nested config names, no arrays, none of that crap. Seriously, I'm
|
||||
watching you.
|
||||
|
||||
* Offline-first means offline-only. Never incorporate TCP or UDP features ever. Such concerns must be handled by higher-level protocols or by application developers. This is to ensure that the protocol is always a viable option for off-grid use cases.
|
||||
* Natural is better than simple. Convention over configuration. Do not make plugins for common use cases unless it would hurt portability.
|
||||
* Prefer a monolithic internal structure. Avoid external dependencies except for limited use cases (Eg: crypto libs). Do not break things into smaller pieces until there are at least three real-world reasons to do so. Decoupling a library into a package for only 2 use cases is not acceptable.
|
||||
* Maintain ecosystem diversity by having a protocol that can be easily and entirely ported to new languages and platforms.
|
||||
* No singletons. No signing authorities, no servers of any kind, even locally, no differentiation between peers (eg: no "super peers").
|
||||
* Configuration is always a design compromise. We will allow a limit of 10 configuration options for all eternity. These are simple key/value pairs. No nesting, no namespacing, no dots, no dashes, no nested config names, no arrays, none of that crap. Seriously, I'm watching you.
|
||||
* Assume CPU and RAM are not plentiful.
|
||||
* Assume platform has no networking support. No servers. No hooks for startup, shutdown, or reboot.
|
||||
* Assume block storage is plentiful when making resource allocation tradeoffs.
|
||||
* Files are better than sessions, but be filesystem agnostic. Persistence mechanisms are implementation-specific.
|
||||
|
||||
* Assume platform has no networking support. No servers. No hooks for
|
||||
startup, shutdown, or reboot.
|
||||
|
||||
* Assume block storage is plentiful when making resource allocation
|
||||
trade-offs.
|
||||
|
||||
* Files are better than sessions, but be filesystem
|
||||
agnostic. Persistence mechanisms are implementation-specific.
|
||||
|
||||
* Provide tamper resistance. Privacy features will be added later.
|
||||
|
||||
* Enable "Free listening"
|
||||
|
||||
* Have a formal specification (reference implementations are not OK).
|
||||
* Minimize conceptual overhead (If it's not needed at least 80% of the time, don't add it).
|
||||
* Use a serialization format that is deterministic and easy to parse on constrained devices.
|
||||
* Backwards compatibility. Numerous compromises have been made to support legacy systems, such as devices that lack network support and FAT16 file systems.
|
||||
|
||||
## Non-Goals
|
||||
* Minimize conceptual overhead (If it's not needed at least 80% of
|
||||
the time, don't add it).
|
||||
|
||||
* Networking support: The protocol will make no mention of TCP, UDP, SSH, HTTP, Etc..
|
||||
* Configurability: Configuration is the root of all evil. Instead of trying to create something that serves every use case, focus on >80% use cases and allow edge cases to be handled by application developers.
|
||||
* Anonymity: Though pseudonymity is possible via the use of multiple identities, the concept of anonymity is not well suited to this protocol. Every message is signed by a known identity, though identities are easily created and can be backed by pseudonyms for privacy.
|
||||
* (OPEN TO DISCUSSION) Encryption. Like many internet protocols, encryption concerns are offloaded to the transport layer. Consider using an encrypted file system or encrypted compression application to store data. If you have ideas on how we could support encryption, please contact us. Please keep in mind that anonymity is not necessarily a project goal.
|
||||
* Use a serialization format that is deterministic and easy to parse
|
||||
on constrained devices.
|
||||
|
||||
# Up Next
|
||||
* Backwards compatibility. Numerous compromises have been made to
|
||||
support legacy systems, such as devices that use FAT16 file systems or lack network support.
|
||||
|
||||
This concludes the overview. To learn more, continue to [frequently asked questions](faq.md)
|
||||
# What the Pigeon protocol is not trying to address
|
||||
|
||||
* Networking support: The protocol will make no mention of TCP, UDP,
|
||||
SSH, HTTP, Etc..
|
||||
* Configurability: Configuration is the root of all evil. Instead of
|
||||
trying to create something that serves every use case, focus on
|
||||
over 80% use cases and allow edge cases to be handled by
|
||||
application developers.
|
||||
* Anonymity: Though pseudonymity is possible via the use of multiple
|
||||
identities, the concept of anonymity is not well suited to this
|
||||
protocol. Every message is signed by a known identity, though
|
||||
identities are easily created and can be backed by pseudonyms for
|
||||
privacy.
|
||||
* OPEN TO DISCUSSION: Encryption. Like many internet protocols,
|
||||
encryption concerns are offloaded to the transport layer. Consider
|
||||
using an encrypted file system or encrypted compression application
|
||||
to store data. If you have ideas on how we could support encryption,
|
||||
please contact us. Please keep in mind that anonymity is not
|
||||
necessarily a project goal.
|
||||
|
||||
# How the Pigeon protocol differs from Sneakernet
|
||||
|
||||
[Sneakernet](https://en.wikipedia.org/wiki/Sneakernet) is a protocol
|
||||
used by ancient civilizations to exchange files between computers with
|
||||
limited internet connectivity.
|
||||
|
||||
Although the Pigeon protocol messages can be exchanged over
|
||||
sneakernet, the Pigeon protocol is _not_ sneakernet. Sneakernet
|
||||
messages by themselves are not tamper-resistant, nor does it provide
|
||||
redundant backup using peers.
|
||||
|
||||
In contrast, a Pigeon protocol message is redundantly replicated
|
||||
_beyond_ its intended recipient to neighboring peers, "a friend of a
|
||||
friend", through gossip. Unlike sneakernet, the Pigeon protocol also
|
||||
uses cryptography to guarantee that a message's content has not been
|
||||
altered by a third party.
|
||||
|
||||
# How the Pigeon protocol differs From Secure Scuttlebutt
|
||||
|
||||
[Secure Scuttlebut](https://scuttlebutt.nz/) is a decentralized,
|
||||
peer-to-peer protocol that also works offline.
|
||||
|
||||
The Pigeon protocol does not rely on networking in the core
|
||||
library. Although Secure Scuttlebutt is able to support
|
||||
Sneakernet-only operation, it is difficult in practice due to reliance
|
||||
on UDP, TCP, and in the case of
|
||||
[pubs](https://github.com/ssbc/ssb-server/wiki/Pub-Servers), DNS.
|
||||
|
||||
The Pigeon protocol uses a custom key/value serialization format
|
||||
instead of JSON. This has two benefits:
|
||||
|
||||
* Serialization and signing is more simple, because indentation and
|
||||
whitespace are less likely to cause verification problems.
|
||||
* Unlike JSON, pigeon messages do not allow nesting, which promotes
|
||||
simplified message schemas.
|
||||
|
||||
The Pigeon protocol uses Crockford-flavored Base32 rather than URL
|
||||
safe Base64. This makes it easier to support old or low-powered
|
||||
systems and is easier to support FAT16 or embedded systems.
|
||||
|
||||
Unlike Secure Scuttlebutt, the Pigeon protocol was designed for
|
||||
portability and simplicity. It allows support for platforms that
|
||||
aren't limited to NodeJS.
|
||||
|
||||
Complicated features, such as network support, are ignored in favor of
|
||||
an easy-to-implement standard.
|
||||
|
||||
# Influences
|
||||
|
||||
Pigeon borrows many of the ideas set forth by the
|
||||
[Secure Scuttlebutt protocol](https://ssbc.github.io/scuttlebutt-protocol-guide/).
|
||||
It is our opinion that SSB is one of the most innovative protocols
|
||||
created in recent years. Without the research and efforts of the
|
||||
[Secure Scuttlebutt Consortium](https://github.com/ssbc)
|
||||
, this project would not be possible, so a big thanks goes out to all
|
||||
the people who make SSB possible.
|
||||
|
||||
We've also been inspired by the compactness and minimalism of
|
||||
[SQLite, which should serve as a role model for all of us](https://www.sqlite.org/talks/wroclaw-20090310.pdf).
|
||||
|
||||
In many ways, this protocol can be considered an amalgam of the best
|
||||
ideas from both SQLite and Secure Scuttlebutt.
|
||||
|
||||
# Getting involved with the Pigeon protocol development
|
||||
|
||||
If you want to be involved with the Pigeon protocol development, there
|
||||
are areas we need help in currently:
|
||||
|
||||
* Documentation, editors, and feedback.
|
||||
* We need applications to be built using the protocol! We are happy to
|
||||
assist you along the way.
|
||||
* We need a BNF grammar for Pigeon messages.
|
||||
|
||||
Your feedback really helps. Feel free to raise an issue with your opinions and
|
||||
ideas. We will reply to all messages.
|
||||
|
||||
# For more information
|
||||
|
||||
Check out the following links:
|
||||
|
||||
* [Message Format Specification](message_format.md)
|
||||
* [Bundle Format Specification](dev_docs.md)
|
||||
* [Frequently Asked Questions](faq.md)
|
||||
* [Roadmap](roadmap.md)
|
||||
* [Ideas and Features](ideas.md)
|
||||
|
||||
TODO:
|
||||
|
||||
* Explain what forking is and why it is not supported.
|
||||
* Explain how verification happens (Soapdog)
|
||||
* Question: "How do apps find what they need in the Pigeon feeds and/or accumulate state forward? Is this the responsibility of the Pigeon libraries, or do apps do it themselves?"
|
||||
|
|
|
|||
|
|
@ -1,5 +0,0 @@
|
|||
# Bundle File Specification (WIP)
|
||||
|
||||
We will draft this specification after the first client is finished- we've found that too many things have changed since the start of the project.
|
||||
|
||||
Please see this [Lexer](https://tildegit.org/PigeonProtocolConsortium/pigeon_ruby/src/branch/master/dist/pigeon/lexer.rb), written in Ruby, that can help you build your own bundle-consuming code:
|
||||
155
dev_docs.md
155
dev_docs.md
|
|
@ -1,26 +1,48 @@
|
|||
# Fundamental Concept: Free Listening
|
||||
|
||||
The concept of "Free Listening" is mentioned often when developing the protocol. The original concept is borrowed from the Secure Scuttlebutt protocol and is well summarized in [a 2018 work by Andre Staltz](https://staltz.com/reinvent-the-social-web.html):
|
||||
|
||||
```
|
||||
Pull is the opposite of push. Instead of realtime push notifications, you choose when you want to get updates about new content. Instead of free speech, as in the right to broadcast, we value free listening, which is the right not to be shouted at. Most social networks allow people to basically insert—push—themselves into a community. But Scuttlebutt invites are the other way around: someone in the community has to pull you in.
|
||||
|
||||
We are not proponents of total self-reliance. With full independence, you’d just be a lonely node in your social graph. The real value of the social graph is not the amount of nodes, it’s the amount of edges. They represent dependence on each other.
|
||||
|
||||
Your friend is your backup, literally. If you happen to lose all your data because your computer explodes, all you need is your crypto key pair, your identity, which is a small file. Then you can re-download everything you said and liked and did from a friend nearby. Nothing will be lost. This happened to someone in the community, and we were pleasantly surprised how well it worked.
|
||||
```
|
||||
|
||||
Pigeon has many of the same goals as Secure Scuttlebutt, including the user's right to Free Listening. Keep this in mind as you navigate the various design decisions and compromises of the protocol.
|
||||
|
||||
# Fundamental Concept: Content Addressing
|
||||
|
||||
Pigeon Protocol relies heavily on content addressing and the SHA-256 hashing algorithm. This means that instead of assigning arbitrary or user-generated names to resources (as is the case with web addresses), names are assigned using the SHA-256 hash algorithm.
|
||||
|
||||
Please watch [this video] if you are unfamiliar with hashing algorithms.
|
||||
|
||||
# Fundamental Concept: Digital Signatures
|
||||
|
||||
Pigeon uses [ED25519](https://ed25519.cr.yp.to/) to generate and verify message signatures. An understanding of digital signing algorithms is assumed. If you need help understanding this concept, please contact us on the mailing list. Due to time constraints, I will defer to existing resources that exist online.
|
||||
|
||||
# Messages: The Basic Building Block
|
||||
|
||||
The most important protocol concept is that of the "message".
|
||||
In their most simple form, Pigeon protocol messages are just ASCII text documents. They are human readable and can even be created by hand in a text editor, though most clients will provide better means of authoring messages.
|
||||
In their most simple form, Pigeon protocol messages are just text documents. Text encoding is determined by the client, but UTF-8 is highly encouraged for maximum interoperability. They are human readable and can even be created by hand in a text editor, though most clients will provide better means of authoring messages.
|
||||
|
||||
Below is an example of such a message:
|
||||
|
||||
```
|
||||
author @MF312A76JV8S1XWCHV1XR6ANRDMPAT2G5K8PZTGKWV354PR82CD0.ed25519
|
||||
author USER.4CZHSZAH8473YPHP1F1DR5ZVCRKEA4Q0BY18NMXYE14NZ0XV2PGG
|
||||
depth 123
|
||||
kind weather_report
|
||||
prev %ZV85NQS8B1BWQN7YAME1GB0G6XS2AVN610RQTME507DN5ASP2S6G.sha256
|
||||
depth 3
|
||||
lipmaa 2
|
||||
prev TEXT.E90DY6RABDQ2CJPVQHYQDYH6N7Q46SZKQ0AQ76J6D684HYBRKE4G
|
||||
|
||||
temperature:"22.0C"
|
||||
webcam_photo:&FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG.sha256
|
||||
weather_reported_by:@0DC253VW8RP4KGTZP8K5G2TAPMDRNA6RX1VHCWX1S8VJ67A213FM.ed25519
|
||||
webcam_photo:FILE.FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG
|
||||
weather_reported_by:USER.GGP2VX0ZN41EYXMN81YB0Q4AEKRCVZ5RD1F1PHPY3748HAZSHZC4
|
||||
|
||||
signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08.sig.ed25519
|
||||
signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08
|
||||
```
|
||||
|
||||
The specifics of the message format are explained line-by-line [in the message format explanation document](message_format.md).
|
||||
Please read this document before continuing.
|
||||
The specifics of the message format are explained line-by-line [in the message format explanation document](message_format.md). **Please read the message format explanation document before continuing.**
|
||||
|
||||
# Bundles: The Transmission Medium
|
||||
|
||||
|
|
@ -29,7 +51,9 @@ A message without a receiver is not very useful. Any peer in a Pigeon cluster ha
|
|||
* Messages, created by the peer or forwarded on behalf of a "peer of a peer"
|
||||
* Files, in the form of blobs
|
||||
|
||||
Pigeon sends messages and files to peers through the use of "bundles", which are simply a filesystem directories that is arranged in a predictable layout. The directories are transmitted over any medium that supports file transfer (the protocol does not involve itself with such concerns).
|
||||
Pigeon sends messages and files to peers through the use of "bundles". Bundles are a file directory arranged in a predictable layout that is specified by the protocol. Peers share bundles with each other through a variety of means. When a peer unpacks a bundle into their local database, the database contents are updated with new information from peers such as messages and blobs (files).
|
||||
|
||||
The directories are transmitted over any medium that supports file transfer (the protocol does not involve itself with such concerns).
|
||||
|
||||
Pigeon follows the philosophy of "offline-first means offline-only". Bundle files could theoretically be:
|
||||
|
||||
|
|
@ -40,57 +64,45 @@ Pigeon follows the philosophy of "offline-first means offline-only". Bundle file
|
|||
|
||||
The protocol does not place any restrictions on how bundles are handled, transported, compressed or encrypted. It does, however, dictate the internal layout of the bundle. Authors of protocol clients must pay special attention to how bundles are created to ensure security and interoperability between client implementations.
|
||||
|
||||
## The Structure of a Bundle: Messages
|
||||
## Where Do Messages Go in a Bundle?
|
||||
|
||||
A bundle's message payload is contained in a plaintext file at the root bundle directory. The file is named `messages.pgn`. Messages are joined together via carriage return (`\n`). Typically, `messages.pgn` will contain messages created by the bundle's author, plus messages created by the author's peers.
|
||||
A bundle's [message](message_format.md) payload (as opposed to its blob payload) is contained in a plaintext file at the root bundle directory. The file is named `messages.pgn`. Messages are joined together via carriage return (`\n`).
|
||||
|
||||
Typically, `messages.pgn` will contain messages created by the bundle's author, plus messages created by the author's peers.
|
||||
|
||||
Here is an example of the contents of a `messages.pgn` file:
|
||||
|
||||
```
|
||||
author @RZW2HE8MQFRH93NP3YKWC1QGZ6VWDZW1WZPMEKQ5MP0NM6TE54W0.ed25519
|
||||
kind &ET9C7B9N82XR0F021CXEWSPDH23H4CHMX866WWA3R2PXEFZM67PG.sha256
|
||||
author USER.R68Q26P1GEFC0SNVVQ9S29SWCVVRGCYRV7D96GAN3XVQE3F9AZJ0
|
||||
depth 0
|
||||
kind chat_message
|
||||
prev NONE
|
||||
depth 0
|
||||
lipmaa 0
|
||||
|
||||
a:"b"
|
||||
content:"Hello, world!"
|
||||
|
||||
signature MYTASNHSPJCV8GH59EEBX27CJHW2N6K02A3MTNFHWHSPDSQKHTMKR23WQS71MTQY0ED4CSK88XNJJ8PV5W9F1BREDR0NZ2CMMRRFT20.sig.ed25519
|
||||
signature 2VMAG4SCX5RHVBKCB1RNZCB0AJN4WN6FEMS7W9FM1CVYSZXMX7CPQFCDPYEKCTGG91Y1YSGY4G5K8XAGQ67HEPDFRMRYQHWQBATAC2R
|
||||
|
||||
author @RZW2HE8MQFRH93NP3YKWC1QGZ6VWDZW1WZPMEKQ5MP0NM6TE54W0.ed25519
|
||||
kind a
|
||||
prev %XKB8MVA3AZZAG1D29AHPG33G2T8BWAGRKJ0DJ4H01MGPM5R9Y3RG.sha256
|
||||
author USER.R68Q26P1GEFC0SNVVQ9S29SWCVVRGCYRV7D96GAN3XVQE3F9AZJ0
|
||||
depth 1
|
||||
lipmaa 0
|
||||
kind chat_message
|
||||
prev TEXT.6CBA4J3756A5SNM1W1GHNCTT9EG95ZP3ZMAT5Z1EJP7TXMNNVZC0
|
||||
|
||||
&7Z2CSZKMB1RE5G6SKXRZ63ZGCNP8VVEM3K0XFMYKETRDQSM5WBSG.sha256:"b"
|
||||
content:"Good morning!"
|
||||
|
||||
signature 5Y8KQVMJ0BADEPWZEXSMYYVMVE5JYQ0069RQ4S80CT9GA64KCRVTWYSK3V728J024916P9SVZ62W9HVZ189C6PANHWD23T07C779P2R.sig.ed25519
|
||||
signature Y34Q47V0BY370RM5KWGRJRN9HFNGJN0C3DEYVB2V2476CW9RN5HD4XD7KMQ6T4T42N36R5P3XX6E3FYEWVZR25AVCF6KQPZHJP6EM10
|
||||
|
||||
author @RZW2HE8MQFRH93NP3YKWC1QGZ6VWDZW1WZPMEKQ5MP0NM6TE54W0.ed25519
|
||||
kind a
|
||||
prev %5N0KJN0TSWRVMXY3JF0FAJRXXDB1AHT9YXBTKE2V7H98GKQND4PG.sha256
|
||||
author USER.0JZA9F3EQVX3NAG69D7VRYCGRVVCWS92S9QVVNS0CFEG1P62Q86R
|
||||
depth 2
|
||||
lipmaa 1
|
||||
kind like
|
||||
prev TEXT.GAP6NJ21K6N75RAEJQ10C2QHFXNRHVPMC54FMGVA77CDJ8AVZQB5
|
||||
|
||||
b:&HDDSVC617PS44NP856N3CJN91HPJXEHHHE93595BJC9VJN6KANFG.sha256
|
||||
|
||||
signature 7XSFRS28B5T6GN0XEYAASFQQAJG9YTVNSXQY0JD8XHYHPQVNK2VSH081PS9CNPNKEAEJGEPXZR6GSZ21SV1HTKQ7R3SZ49P8PHRER18.sig.ed25519
|
||||
|
||||
author @ET6MN0PM5WQKEMPZ3PN39HRFQM8EH2WZRW10W45ZDWV6ZGQ1CWKY.ed25519
|
||||
kind ab606fa8-958e-45e6-9856-0103217ce0a9
|
||||
prev %HT23KER8VQJFMFDAWX7RX6CCXVVCZ2H2C9H1HEA5ZNC9A6MAFXG0.sha256
|
||||
depth 0
|
||||
lipmaa 2
|
||||
|
||||
foo:"f6a627ce-9e0d-4faa-9146-bd4e56b61811"
|
||||
|
||||
signature 3NQ3P4J3TJSWH7WTJ0T4WCTVDVX1QAVW31G0K5T59F3X1DRYE3ECD1YVFJ0PT85RTRPD2GG8H091F8TG2A7CV36J8N5Y69RYGTQJE08.sig.ed25519
|
||||
target:TEXT.5BQZVA8JDC77AVGMF45CMPVHRNXFHQ2C01QJEAR57N6K12JN6PAG
|
||||
|
||||
signature W68NWDQB2WTZ8T1RHP5BZA4N1STVKV16K0PXH10MZVR3XTF8HC7T8646X7SAKP5DFZ5K74QEKE3T2K6V0EST50YQQD7FD2PT0H8J62G
|
||||
```
|
||||
|
||||
The example above is multiple pigeon messages joined together with a carraige return ("\n").
|
||||
The file contains the messages of two authors (`@ET6MN...`, `@RZW2H...`).
|
||||
The example above is multiple pigeon messages joined together with a carriage return ("\n").
|
||||
**Notice that not all messages were created by the same author.**
|
||||
|
||||
A peer can use the file above to update their local database. It is important to note that **a client will always reject a message that it cannot verify**. Below is an example of a refused update:
|
||||
|
||||
|
|
@ -99,52 +111,17 @@ A peer can use the file above to update their local database. It is important to
|
|||
* The bundle contains messages authored by peer B, starting at `depth 8`.
|
||||
* Since Peer A cannot verify message at `depth 8` until it receives `depth 7` and `depth 6`, the messages are rejected by peer A.
|
||||
|
||||
* Clients always reject message that cannot be verified. For example, if a `messages.pgn` file starts at `depth 6` for a peer `@ABC`, and the local client has only verified `@ABC`s feed up to `depth 3`, the messages in the bundle will be rejected by the local client because it does not have enough information available to verify the authenticity of the message.
|
||||
* Messages must be ordered `depth` for a particular author.
|
||||
* Messages
|
||||
* Clients always reject messages that cannot be verified. For example, if a `messages.pgn` file starts at `depth 6` for a peer `USER.ABC`, and the local client has only verified `USER.ABC`s feed up to `depth 3`, the messages in the bundle will be rejected by the local client because it does not have enough information available to verify the authenticity of the message.
|
||||
* Messages must be ordered by `depth` for a particular author.
|
||||
|
||||
Some things for Pigeon implementors to note about bundles:
|
||||
# Where Do Files Go in a Bundle?
|
||||
|
||||
* It can contain other peoples messages
|
||||
* It is advisable to only ingest blobs that are referenced
|
||||
Files ("blobs") are transferred alongside the `*.pgn` message bundle.
|
||||
It is the responsibility of the _receiver_ (not the sender) to calculate the multihash of an incoming file. The ensures that files are not misrepresented or tampered with.
|
||||
|
||||
Example:
|
||||
Files added to a blob must follow these naming rules:
|
||||
|
||||
a user exports a bundle that contains a few messages and the following blobs:
|
||||
|
||||
```
|
||||
&622PRNJ7C0S05XR2AHDPKWMG051B1QW5SXMN2RQHF2AND6J8VGPG.sha256
|
||||
&FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG.sha256
|
||||
&YPF11E5N9JFVB6KB1N1WDVVT9DXMCHE0XJWBZHT2CQ29S5SEPCSG.sha256
|
||||
```
|
||||
|
||||
Assuming the user's client follows this specification, the exported bundles strucutre would be laid out as follows on the local filesystem:
|
||||
|
||||
```
|
||||
├── messages.pgn
|
||||
├── 622PRNJ
|
||||
│ └── 7C0S05X
|
||||
│ └── R2AHDPK
|
||||
│ └── WMG051B
|
||||
│ └── 1QW5SXM
|
||||
│ └── N2RQHF2
|
||||
│ └── AND6J8V.GPG
|
||||
├── FV0FJ0Y
|
||||
│ └── ZADY7C5
|
||||
│ └── JTTFYPK
|
||||
│ └── DBHTZJ5
|
||||
│ └── JVVP5TC
|
||||
│ └── KP0605W
|
||||
│ └── WXYJG4V.MRG
|
||||
└── YPF11E5
|
||||
└── N9JFVB6
|
||||
└── KB1N1WD
|
||||
└── VVT9DXM
|
||||
└── CHE0XJW
|
||||
└── BZHT2CQ
|
||||
└── 29S5SEP.CSG
|
||||
```
|
||||
|
||||
# Up Next
|
||||
|
||||
This concludes the developer documentation. Please email us with questions. To learn more, continue to the [idea bin](IDEAS.md).
|
||||
* The [filename extension](https://en.wikipedia.org/wiki/Filename_extension) is located in the same directory as the `messages.pgn` file.
|
||||
* The filename must follow [8.3 filename conventions](https://en.wikipedia.org/wiki/8.3_filename).
|
||||
* The file extension must be `.blb`.
|
||||
* The filename cannot be longer than 8 chars.
|
||||
|
|
|
|||
22
diagram.md
22
diagram.md
|
|
@ -9,26 +9,24 @@ classDiagram
|
|||
}
|
||||
|
||||
class Example1{
|
||||
author USER.R68Q2...9AZJ0
|
||||
author USER.R68...ZJ0
|
||||
depth 0
|
||||
kind my_example
|
||||
lipmaa NONE
|
||||
kind my_blog
|
||||
prev NONE
|
||||
|
||||
foo:"bar"
|
||||
current_status:"Another great day. Nice weather, too!"
|
||||
|
||||
signature 2VMAG4...TAC2R
|
||||
signature 2VMAG...TAC2R
|
||||
}
|
||||
|
||||
class Example2{
|
||||
author USER.R68Q2...F9AZJ0
|
||||
author USER.R68...ZJ0
|
||||
depth 1
|
||||
kind another_example
|
||||
lipmaa NONE
|
||||
prev TEXT.6CBA4...NVZC0
|
||||
kind like_a_post
|
||||
prev TEXT.6CB...ZC0
|
||||
|
||||
bar:"baz"
|
||||
liked_post_multihash:TEXT.6CB...ZC0
|
||||
|
||||
signature Y34Q4...6EM10
|
||||
signature Y34Q47...6EM10
|
||||
}
|
||||
```
|
||||
```
|
||||
|
|
|
|||
BIN
diagram1.png
BIN
diagram1.png
Binary file not shown.
|
Before Width: | Height: | Size: 41 KiB After Width: | Height: | Size: 18 KiB |
4
faq.md
4
faq.md
|
|
@ -25,7 +25,3 @@
|
|||
# Is This a Blockchain?
|
||||
|
||||
It's different than a block chain despite some similarity. A global blockchain is a singleton, and Pigeon has no singletons. Each Pigeon node maintains their own feed of messages rather than sharing a global feed. When you trust a peer, you agree to replicate their feed. You also replicate the feed of their peers. Unlike a blockchain, there are many feeds (rather than one) and they do not require consensus or coordination to coexist. Additionally, forking of feeds is not supported by Pigeon.
|
||||
|
||||
# Up Next
|
||||
|
||||
Continue to [Developer Docs and Specification](DEV_DOCS.md)
|
||||
|
|
@ -5,38 +5,82 @@ In the test that follows, we will explore a pigeon message line-by-line.
|
|||
The example message is shown in its entirety below:
|
||||
|
||||
```
|
||||
author @MF312A76JV8S1XWCHV1XR6ANRDMPAT2G5K8PZTGKWV354PR82CD0.ed25519
|
||||
author USER.4CZHSZAH8473YPHP1F1DR5ZVCRKEA4Q0BY18NMXYE14NZ0XV2PGG
|
||||
depth 123
|
||||
kind weather_report
|
||||
prev %ZV85NQS8B1BWQN7YAME1GB0G6XS2AVN610RQTME507DN5ASP2S6G.sha256
|
||||
depth 3
|
||||
lipmaa 2
|
||||
prev TEXT.E90DY6RABDQ2CJPVQHYQDYH6N7Q46SZKQ0AQ76J6D684HYBRKE4G
|
||||
|
||||
temperature:"22.0C"
|
||||
webcam_photo:&FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG.sha256
|
||||
weather_reported_by:@0DC253VW8RP4KGTZP8K5G2TAPMDRNA6RX1VHCWX1S8VJ67A213FM.ed25519
|
||||
webcam_photo:FILE.FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG
|
||||
weather_reported_by:USER.GGP2VX0ZN41EYXMN81YB0Q4AEKRCVZ5RD1F1PHPY3748HAZSHZC4
|
||||
|
||||
signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08.sig.ed25519
|
||||
signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08
|
||||
```
|
||||
|
||||
### Pigeon Multihashes and Data Types
|
||||
|
||||
Pigeon has 4 data types:
|
||||
|
||||
* Blob multihash: Used to identify arbitrary binary files mentioned in a feed. Blob multihashes start with the word `FILE.`, followed by a Base32 hash (SHA256).
|
||||
* Message multihash: Used to reference a message in someone's feed. Starts with the word `TEXT.` followed by a Base32 SHA256 checksum.
|
||||
* User multihash: Used to reference a particular feed. Starts with the word `USER.` followed by a base32 encoded ED25519 public key.
|
||||
* String: Information shorter than 128 characters can be wrapped in "double quotes" and placed directly into messages. Larger strings must be converted to blobs. The byte length of a character is determined by the client's chosen text encoding scheme. Pigeon has no official text encoding, but UTF-8 is highly encouraged in scenarios where interoperability is important. One character always one character, even if it is multi-byte. Strings cannot contain whitespace characters (encoding dependent). The protocol does not implement character escaping (eg: `"\n"`). Please start a discussion on the mailing list if your application requires this feature.
|
||||
* None: The word "NONE" is used to indicate the absence of data, similarly to `null` or `nil` seen in some programming languages.
|
||||
|
||||
### Parts of a Message
|
||||
|
||||
The three parts of a Pigeon message are:
|
||||
|
||||
1. Header: Data that is used by the _protocol_
|
||||
2. Body: Data that is used by the _user or application_
|
||||
3. Footer: Cryptographic signature to prevent tampering or forgery.
|
||||
|
||||
**The parts of a message must follow the order specified in this document.** The order of headers is not user definable. The fact that headers are alphabetical is a coincidence and future versions of the protocol might not be alphabetized.
|
||||
|
||||
### Parts of a Header
|
||||
|
||||
A header is the first part of a message and contains 5 subsections. Each of these sections will be explained in further detail in the sections that follow:
|
||||
|
||||
1. `author`: A user multihash indicating the author of the message. The public key will be used to verify the signature (found in the footer)
|
||||
1. `depth`: The order number of the current message. Since feeds are append-only, this number will only increase. The `depth` field ensures unique message signatures, even for successive duplicate messages.
|
||||
1. `kind`: Used by applications to determine the intent or "shape" of a message.
|
||||
1. `prev`: The multihash of the previous message in the feed. Required for verifying the authenticity of a feed.
|
||||
|
||||
|
||||
**Header entries must follow the order specified above.**
|
||||
|
||||
### Line 1: `Author`
|
||||
|
||||
EXAMPLE:
|
||||
|
||||
```
|
||||
author @MF312A76JV8S1XWCHV1XR6ANRDMPAT2G5K8PZTGKWV354PR82CD0.ed25519
|
||||
author USER.4CZHSZAH8473YPHP1F1DR5ZVCRKEA4Q0BY18NMXYE14NZ0XV2PGG
|
||||
```
|
||||
|
||||
The first line of a Pigeon message header is the `author` entry.
|
||||
|
||||
Every Pigeon database has an "identity". An identity is an ED25519 key pair that prevents tampering by parties other than the database owner. An identity is publicly referenced using a "multihash". In the example above, the identity multihash was `@MF312A76JV8S1XWCHV1XR6ANRDMPAT2G5K8PZTGKWV354PR82CD0.ed25519`.
|
||||
Every Pigeon database has an "identity". An identity is an ED25519 key pair that prevents tampering by parties other than the database owner. In the example above, the identity multihash was `USER.4CZHSZAH8473YPHP1F1DR5ZVCRKEA4Q0BY18NMXYE14NZ0XV2PGG`.
|
||||
|
||||
The steps to generate a valid identity are:
|
||||
|
||||
1. Perform [Crockford Base32 encoding](https://www.crockford.com/base32.html) on an ED25519 public key.
|
||||
2. Add an `@` symbol to the beginning of the string from step 1.
|
||||
3. Add a `.ed25519` string to the end of the string from step 2.
|
||||
2. Concatenate the characters `USER.` to the beginning of the string from step 1.
|
||||
|
||||
### Line 2: `Kind`
|
||||
### Line 2: `Depth`
|
||||
|
||||
EXAMPLE:
|
||||
|
||||
```
|
||||
depth 3
|
||||
```
|
||||
|
||||
Pigeon messages exist in a linear sequence which only moves forward and never "forks", skips or moves backward.
|
||||
Every message has a `depth` field to indicate its "place in line".
|
||||
Because every message has an ever-increasing integer that never duplicates, every message in a Pigeon feed will have a unique hash. This is true even if messages have identical body content.
|
||||
|
||||
The `depth` count always starts at 0 and icreases by 1 every time a new message is added to the feeds. Feeds that assign `depth` values in a non-sequential order are invalid. Eg: No gaps, no skipping, etc..
|
||||
|
||||
### Line 3: `Kind`
|
||||
|
||||
EXAMPLE:
|
||||
|
||||
|
|
@ -52,86 +96,31 @@ It must meet the following criteria:
|
|||
* Cannot contain whitespace or control characters
|
||||
* May contain any of the following characters:
|
||||
* alphanumeric characters
|
||||
* dashes (`-`)
|
||||
* underscores (`_`)
|
||||
* Symbols used for multihashes, such as `@`, `&` and `%` (covered later).
|
||||
|
||||
### Line 3: `Prev`
|
||||
|
||||
EXAMPLE:
|
||||
|
||||
```
|
||||
prev %ZV85NQS8B1BWQN7YAME1GB0G6XS2AVN610RQTME507DN5ASP2S6G.sha256
|
||||
```
|
||||
|
||||
A Pigeon message feed is a unidirectional chain of documents where the newest document points back to the document that came before it in the chain ([example diagram](diagram1.png)).
|
||||
|
||||
To create this chain, a Pigeon message uses the `prev` field. The `prev` field contains a message multihash. In this case, the multihash is `%ZV85NQS8B1BWQN7YAME1GB0G6XS2AVN610RQTME507DN5ASP2S6G.sha256`.
|
||||
|
||||
Messages are content addressed. This is in contrast to protocols such as HTTP which use names to identify resources. Because Pigeon messages are addressed by content rather than by name, changing a message's content, even by just one character, has the effect of completely changing the message's multihash.
|
||||
* dashes (`-`), underscores (`_`) and dots (`.`)
|
||||
|
||||
**For the first message of a feed, this value is set to `NONE`.**
|
||||
|
||||
Message multihashes are calculated as follows:
|
||||
|
||||
1. The first character is a `%` symbol, indicating that it is a `message` rather than an `identity`, `blob` or `string`.
|
||||
2. The next 52 characters are a [Crockford base 32](https://www.crockford.com/base32.html) SHA512 hash of the previous message's content.
|
||||
3. The message multihash ends in `.sha512`.
|
||||
1. Create a [Crockford base 32](https://www.crockford.com/base32.html) sha256 hash of the message's content.
|
||||
2. Append the string `TEXT.` to the front of the checksum created in step 1.
|
||||
|
||||
### Line 4: `Depth`
|
||||
### Line 4: `Prev`
|
||||
|
||||
EXAMPLE:
|
||||
|
||||
```
|
||||
depth 3
|
||||
prev TEXT.E90DY6RABDQ2CJPVQHYQDYH6N7Q46SZKQ0AQ76J6D684HYBRKE4G
|
||||
```
|
||||
|
||||
Pigeon messages exist in a linear sequence which only moves forward and never "forks".
|
||||
Every message has a `depth` field to indicate its "place in line".
|
||||
Because every message has an ever-increasing integer that never duplicates, every message in a Pigeon feed will have a unique hash. This is true even if messages have identical body content.
|
||||
A Pigeon message feed is a unidirectional chain of documents where the newest document points back to the document that came before it in the chain ([example diagram](diagram1.png)).
|
||||
|
||||
### Line 5: `Lipmaa`
|
||||
To create this chain, a Pigeon message uses the `prev` field. The `prev` field contains a message multihash. In this case, the multihash is `TEXT.E90DY6RABDQ2CJPVQHYQDYH6N7Q46SZKQ0AQ76J6D684HYBRKE4G`.
|
||||
|
||||
**THIS FIELD WAS WRITTEN INCORRECTLY. THIS WILL CHANGE SOON. YOU CAN SAFELY MOVE TO THE NEXT SECTION OF THE DOCS**
|
||||
Messages are content addressed. This is in contrast to protocols such as HTTP which use names to identify resources. Because Pigeon messages are addressed by content rather than by name, changing a message's content, even by just one character, has the effect of completely changing the message's multihash.
|
||||
|
||||
This concept was borrowed from the [Bamboo protocol](https://github.com/AljoschaMeyer/bamboo#links-and-entry-verification) and [Helger Lipmaa's thesis](https://kodu.ut.ee/~lipmaa/papers/thesis/thesis.pdf).
|
||||
|
||||
The `lipmaa` field (often called a "Lipmaa Link") is a special kind of `prev` field that allows partial verification of feeds. This field makes it possible to verify a single message (or subset of messages) without downloading the entire chain of messages.
|
||||
|
||||
|
||||
|
||||
The `lipmaa` field is calculated as follows:
|
||||
|
||||
```ruby
|
||||
def lipmaa(n)
|
||||
# The original lipmaa function returns -1 for 0
|
||||
# but that does not mesh well with our serialization
|
||||
# scheme. Comments welcome on this one.
|
||||
return 0 if n < 1 # Prevent -1, division by zero etc..
|
||||
|
||||
m, po3, x = 1, 3, n
|
||||
# find k such that (3^k - 1)/2 >= n
|
||||
while (m < n)
|
||||
po3 *= 3
|
||||
m = (po3 - 1) / 2
|
||||
end
|
||||
po3 /= 3
|
||||
# find longest possible back-jump
|
||||
if (m != n)
|
||||
while x != 0
|
||||
m = (po3 - 1) / 2
|
||||
po3 /= 3
|
||||
x %= m
|
||||
end
|
||||
if (m != po3)
|
||||
po3 = m
|
||||
end
|
||||
end
|
||||
return n - po3
|
||||
end
|
||||
```
|
||||
|
||||
### Line 6: Body Start (Empty Line)
|
||||
### Line 5: Body Start (Empty Line)
|
||||
|
||||
Once all headers are added, a client must place an empty line (`\n`) after the header.
|
||||
The empty line signifies the start of the message body.
|
||||
|
|
@ -140,18 +129,19 @@ Some notes about body entries:
|
|||
|
||||
* The body of a message starts and ends with an empty line (`\n`).
|
||||
* Every body entry is a key value pair. Keys and values are separated by a `:` character (no spaces).
|
||||
* A key must be 1-90 characters in length
|
||||
* A message may not exceed 128 key/value pairs.
|
||||
* A key must be 1-90 characters in length.
|
||||
* A key cannot contain whitespace or control characters
|
||||
* A key may contain any of the following characters:
|
||||
* A key only contains the following characters (`[A-Z|a-z|\-|\_|\.|0-9]{1,90}`):
|
||||
* alphanumeric characters (a-z, A-Z, 0-9)
|
||||
* dots (`.`)
|
||||
* dashes (`-`)
|
||||
* underscores (`_`)
|
||||
* Symbols used for multihashes, such as `@`, `&` and `%` (covered later).
|
||||
* A value may be a:
|
||||
* A string (128 characters or less)
|
||||
* A multihash referencing an identity (`@`), a message (`%`) or a blob (`&`).
|
||||
* A multihash referencing an identity (`USER.`), a message (`TEXT.`) or a blob (`FILE.`).
|
||||
|
||||
### Lines 7: Entry Containing a String
|
||||
### Lines 6: Example Entry Containing a String
|
||||
|
||||
EXAMPLE:
|
||||
|
||||
|
|
@ -159,54 +149,52 @@ EXAMPLE:
|
|||
temperature:"22.0C"
|
||||
```
|
||||
|
||||
Body entries are defined by user and contain key/value pairs of application-specific data.
|
||||
Body entries are defined by the user and contain key/value pairs of application-specific data.
|
||||
When a key/value pair represents something other than an identity, blob or message ID, a string is used.
|
||||
Strings can be used for any type of data that does not fit into the other three categories.
|
||||
Strings must be less than or equal to 128 characters in length.
|
||||
The example above is the most simple kind of body entry. It specifies an arbitrary string representing the current temperature.
|
||||
|
||||
### Lines 8: Entry Referencing a Blob
|
||||
The protocol does not dictate the format of strings (ie: there is no 1st class JSON support). The meaning and formatting of a string is the responsibility of the application.
|
||||
|
||||
### Lines 7: Entry Referencing a Blob
|
||||
|
||||
EXAMPLE:
|
||||
|
||||
```
|
||||
webcam_photo:&FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG.sha256
|
||||
webcam_photo:FILE.FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG
|
||||
```
|
||||
|
||||
Applications may attach files to messages in the form of blobs. Blobs are referenced using a blob multihash.
|
||||
|
||||
* Starts with a `&` character.
|
||||
* Ends with `.sha256`
|
||||
* Contains exactly 52 characters between the `&` and `.sha256` parts. This is a SHA256 hash of the blob's content, represented in Crockford Base 32 encoding.
|
||||
A blob is referenced in a message's key or value. A client will include a blob's content in a "bundle" (explained later). This ensures that a feed's peers get a copy of the file that a message references.
|
||||
|
||||
A blob is referenced in a message's key or value. A client will include a blob's content in a "bundle" (explained later).
|
||||
|
||||
### Lines 9: Entry Referencing a Peer's Identity
|
||||
### Lines 8: Entry Referencing a Peer's Identity
|
||||
|
||||
EXAMPLE:
|
||||
|
||||
```
|
||||
weather_reported_by:@0DC253VW8RP4KGTZP8K5G2TAPMDRNA6RX1VHCWX1S8VJ67A213FM.ed25519
|
||||
weather_reported_by:USER.GGP2VX0ZN41EYXMN81YB0Q4AEKRCVZ5RD1F1PHPY3748HAZSHZC4
|
||||
```
|
||||
|
||||
A message may reference other identities (or its own identity) by using an identity sigil either in the key or value portion of the entry.
|
||||
A message may reference other identities (or its own identity) by using an identity multihash either in the key or value portion of the entry.
|
||||
|
||||
This is analogous to "social tagging" seen in many social networks.
|
||||
|
||||
### Lines 10: Empty Carriage Return (Footer Start)
|
||||
### Lines 9: Empty Carriage Return (Footer Start)
|
||||
|
||||
The last part of a message is the footer. Like a message body, a message footer starts and ends with an empty line.
|
||||
The footer is essential for ensuring the tamper resistant properties of a Pigeon message.
|
||||
|
||||
### Lines 11: Signature Line
|
||||
### Lines 10: Signature Line
|
||||
|
||||
EXAMPLE:
|
||||
|
||||
```
|
||||
signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08.sig.ed25519
|
||||
signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08
|
||||
```
|
||||
|
||||
A signature starts with the word `signature` followed by a space.
|
||||
After that, the body (including the trailing `\n`) is signed using the author's ED25519 key.
|
||||
After that, the body and header (including the trailing `\n`) are signed using the author's ED25519 key.
|
||||
The signature is encoded with Crockford base 32.
|
||||
The signature ends with `.sig.ed25519`.
|
||||
An empty carraige return is added after the signature line.
|
||||
An empty carriage return is added after the signature line.
|
||||
|
|
|
|||
|
|
@ -47,8 +47,3 @@ With a finalized spec and a portable client library, the next goal is to promote
|
|||
This phase will be considered complete when there are three production-scale apps using the libraries authored. By this point, we've hopefully made a difference and helped people regain control of their data and find a new alternative to the current status-quo of "online only" computer applications.
|
||||
|
||||
After that, I might rename the project so that we are not tied to the legacy baggage of the prototype phase. It might be fun to apply to a grant for continued maintenance (or just lock down the feature set- it's too early to say).
|
||||
|
||||
|
||||
# Up Next
|
||||
|
||||
Continue to the [idea bin](IDEAS.md).
|
||||
Loading…
Reference in New Issue