PigeonProtocolConsortium/Protocol-Spec
PigeonProtocolConsortium
/
Protocol-Spec
5
4
Fork 1
Code Issues Pull Requests Releases Wiki Activity

Draft a project roadmap.

This commit is contained in:
Netscape Navigator 2020-03-21 09:49:22 -05:00
parent 8a2dee4266
commit b3de0742a2
1 changed files with 134 additions and 102 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

236
README.md
View File

@ -1,6 +1,6 @@
![](logo.png)
# Pigeon
# Pigeon - The Off Grid Peer-to-Peer Protocol™
A synchronizing peer-to-peer messaging protocol that is:
@ -10,75 +10,24 @@ A synchronizing peer-to-peer messaging protocol that is:
* delay tolerant
* built for [sneakernet](https://en.wikipedia.org/wiki/Sneakernet) from the ground up
The document below describes a protocol as it _should be_ rather than as it is. This document does not describe a working protocol. It is a planning document for a protocol and the first software packages that will implement the protocol.
# Protocol Maturity
The document below describes a protocol as it _should be_ rather than as it is. Although the [first implementation of a Pigeon protocol client](https://tildegit.org/PigeonProtocolConsortium/pigeon_ruby) is nearly complete, it is still a work in progress.
This document does not yet describe a working protocol. It is a planning document for a protocol and the first software packages that will implement the protocol.
# Why?
Pigeon can serve a number of use cases. Below are some examples:
* Systems with low connectivity or uptime (remote sensor logging, maritime systems, solar systems with intermittent power, IoT systems with poor network connectivity)
* Systems with low connectivity or uptime such as remote sensor logging, maritime systems, solar systems with intermittent power, IoT systems with poor network connectivity.
* Store-and-forward message gateways, such as a [data mule](https://en.wikipedia.org/wiki/Data_mule).
* Censorship resistant communications
* Censorship resistant applications, such as peer-to-peer messaging and blogging.
* [Delay tolerant networking](https://en.wikipedia.org/wiki/Delay-tolerant_networking)
* Applications that require a high level of data-integrity or auditing.
* delay-tolerant peer-to-peer social networks, games, file sharing etc...
* Delay-tolerant peer-to-peer social networks, games, file sharing etc...
* Time series data storage
# How Pigeon Differs from Traditional Sneakernet
Sneakernet is a protocol used by ancient civilization to exchange files between computers with limited internet connectivity. Although Pigeon protocol messages can be exchanged over sneakernet, Pigeon is _not_ sneakernet. Sneakernet messages by themselves are not tamper resistant, nor are they replicated by peers other than the recipient. In contrast, a Pigeon protocol message is replicated _beyond_ its intended recipient to neighboring peers ("friend of a friend") via gossip and uses cryptography to guarantee that a message's content has not been altered by a third party.
# I Have Internet Access. Why Should I Care?
* [How Iran Turned Off the Internet](https://thewire.in/tech/how-iran-turned-off-the-internet)
* [Building Internet-connected things seems obvious today, but what about when there’s no Internet?](https://back7.co/home/raspberry-pi-recovery-kit)
* [‘Nobody’s got to use the Internet’: A GOP lawmaker’s response to concerns about Web privacy](https://www.washingtonpost.com/news/powerpost/wp/2017/04/15/nobodys-got-to-use-the-internet-a-gop-lawmakers-response-to-concerns-about-web-privacy/)
* [The death of America's net neutrality and how it affects you](https://www.dw.com/en/the-death-of-americas-net-neutrality-and-how-it-affects-you/a-43934099)
* [YouTube and Facebook Are Removing Evidence of Atrocities, Jeopardizing Cases Against War Criminals](https://theintercept.com/2017/11/02/war-crimes-youtube-facebook-syria-rohingya/)
* [Encryption is Not Preventing Law Enforcement from Investigating Crime ](https://www.alec.org/article/encryption-is-not-preventing-law-enforcement-from-investigating-crime/)
* [Iraq introduces nightly internet curfew](https://netblocks.org/reports/iraq-introduces-nightly-internet-curfew-JAp1DKBd)
* [Building a Low-Tech Internet](https://www.lowtechmagazine.com/2015/10/how-to-build-a-low-tech-internet.html)
* [Inside Cuba's massive, weekly, human-curated sneakernet](https://boingboing.net/2018/05/03/inside-cubas-massive-weekly.html)
* [CollapseOS](https://collapseos.org/)
* [Russian Law Takes Effect that Gives Government Sweeping Power Over Internet](https://www.npr.org/2019/11/01/775366588/russian-law-takes-effect-that-gives-government-sweeping-power-over-internet)
* [Indian Internet shut down as protests rage against citizenship bill](https://edition.cnn.com/2019/12/12/asia/india-shutdown-citizenship-bill-intl-hnk/index.html)
* [Google goes offline after fibre cables cut](https://www.bbc.com/news/technology-50851420)
* [Authoritarian Nations Are Turning the Internet Into a Weapon](https://onezero.medium.com/authoritarian-nations-are-turning-the-internet-into-a-weapon-10119d4e9992)
# What Does It Do?
Each node in a swarm of peers has a local "log". The log is an append only feed of messages written in an ASCII-based serialization format. Messages are signed with a secret key to validate a message's integrity and to prevent tampering by untrusty peers. Nodes in the swarm "follow" other logs from peers of interest. Nodes always replicate the logs of their peers and "gossip" information about peers across the swarm. Gossip information is packaged into "bundles" which contain backups of peer logs in an efficient binary format that can be easily transmitted via sneakernet, direct serial connection, or any other high-latency, high throughput medium.
# How It Works
Sneakernet is the main use case for Pigeon messages to be transmitted. Transmission of SD Cards via postal mail offer an excellent medium for transmission of Pigeon messages, although any data transfer medium is theoretically possible.
![](sync.png)
# Prior Art
This is an exploration of ideas set forth by the Secure Scuttlebutt protocol. It is my opinion that SSB is one of the most innovative protocols created in recent years. Without the research and efforts of the SSBC, this project would not be possible, so a big thanks goes out to all the people who make SSB possible.
I've also been inspired by the compactness and minimalism of [SQLite, which should serve as a role model for all of us](https://www.sqlite.org/talks/wroclaw-20090310.pdf).
In many ways, this protocol can be considered an amalgam of the best ideas from both SQLite and Secure Scuttlebutt.
# Hard Constraints
* Have [near] zero config. We will allow a limit of 10 configuration options for all eternity. These are simple key/value pairs. No nesting, no namespacing, no dots, no dashes, no nested config names, no arrays, none of that crap. Seriously, I'm watching you.
* Offer a portable and interchangeable format for application developers to synchronize data between peers.
* No singletons. No servers, no client/server differentiation.
# Protocol Vs. Implementation Vs. Application
In the specification below, the terms "protocol", "implementation" and "application" will be used frequently.
The **protocol** is a document that specifies a set of guidelines that implementors must follow when authoring a Pigeon Protocol implementation. Analogy: HTTP is a protocol specified by [RFC 2616](https://tools.ietf.org/html/rfc2616).
An **implementation** is a software library (not an application for end users) that implements the Pigeon protocol specification. Example: LibCurl and LibHTTP are HTTP implementations.
An **application** is software that uses a Pigeon Protocol implementation, possibly augmenting the protocol with additional functionality. Example: Netscape Navigator is an HTTP application.
# What is Possible?
Below are some possible use cases to illustrate real-world applications. Once protocol implementations exist, the ideas below should be possible.
@ -95,30 +44,23 @@ Below are some possible use cases to illustrate real-world applications. Once pr
* Sync a feed over bluetooth (via external tool)
* Sync a feed over actual pigeons, possibly soliciting help from world famous boxer and pigeon racing enthusiast Mike Tyson.
# Unanswered Questions
# How Pigeon Differs from Traditional Sneakernet
* Ephemeral key exchange
* Merkle tree vs. hash chain
* Standardized general purpose message schemas (follow, unfollow, redirect, etc.)
# Concepts
[Sneakernet](https://en.wikipedia.org/wiki/Sneakernet) is a protocol used by ancient civilizations to exchange files between computers with limited internet connectivity. Although Pigeon protocol messages can be exchanged over sneakernet, Pigeon is _not_ sneakernet. Sneakernet messages by themselves are not tamper resistant, nor do they provide redundant backup via peers. In contrast, a Pigeon protocol message is redundantly replicated _beyond_ its intended recipient to neighboring peers ("friend of a friend") via gossip and uses cryptography to guarantee that a message's content has not been altered by a third party.
* Base64: The protocol will use Base 64 Encoding with URL and Filename Safe Alphabet as specified in [RFC 4648](https://tools.ietf.org/html/rfc4648). The protocol always uses the standard "=" character for padding. Deviations from this will be explicitly noted.
* Identity: A base64 `ed25519` public key string starting with `@` and ending with `.ed25519`.
* Message Signature: An ED25519 signature starting with a `%` and end with `.sha256`. Messages (covered later) are referenced by a signature.
* String: A 1..62 byte list of ASCII characters, Starting and ending with `"`.
* Blob: Arbitrary binary data, with a current max size of 1.4 MB.
* Blob Hash: A base64'ed SHA256 of a Blob, starting with `&` and ending with `.sha256`.
* Pair: One `string` and on of the following: `Blob Hash|Signature|Identity|String`, joined with a `:` character between the two. See "key" and "value" below.
* Header: A reserved pair of information that is required by the protocol for internal reasons. Not to be confused by a `Pair`, which is user definable. The only Headers the protocol currently uses are `author`, `depth`, `kind`, `prev`. Headers do not have "string quotes" around the key, and the value is delimited by a space (` `) character. Example: `sequence 46`.
* Key: The value to the left of a `:` in a pair. Always a string.
* Value: The value to the right of a `:` in a pair. It is always one of the following: `Blob Hash, Signature, Identity, String`.
* Message: A document with an author, prev, depth, kind, an arbitrary set of attribute pairs and a footer.
* Footer: ??? TODO ????
* Signature: ??? TODO ???
* Kind: A string at the top of a message indicating the message's purpose. Example `"private_message"`, `"mention"`, `"share"`. Kinds may be namespaced by applications using the `.` character.
* Feed: A linked collection of messages.
* Null Signature: An ASCII `0` character, used to indicate the first message in a feed (discussed later)
* Bundle: A specially crafted text archive sent from one peer directly to another peer for the sake of synchronizing and gossiping feeds. Bundles are intricate and require their own document, found [here](bundles.md)
In summary, Pigeon protocol offers benefits above what a traditional sneakernet can provide. A Pigeon protocol message:
* Is automatiaclly backed up by peers and peers-of-peers (gossip).
* Cannot be forged by malicious parties.
* Cannot be altered by anyone except the author.
# How Does It Work?
Each node in a swarm of peers has a local "log". The log is an append only feed of messages written in an ASCII-based serialization format. Messages are signed with a secret key to validate a message's integrity and to prevent tampering by untrusted peers. Nodes in the swarm "follow" other logs from peers of interest. Nodes always replicate the logs of their peers and "gossip" information about peers across the swarm. Gossip information is packaged into "bundles" which contain backups of peer logs in an efficient binary format that can be easily transmitted via sneakernet, direct serial connection, or any high throughput medium, regardless of latency.
Log synchronization via Sneakernet is the main use case for Pigeon messages to be transmitted. Transmission of SD Cards via postal mail offer an excellent medium for transmission of Pigeon messages, although any data transfer medium is theoretically possible.
![](sync.png)
# What a Message Looks Like
@ -155,7 +97,116 @@ signature AerpDKbKRrcaM9wihwFsPC4YRAfYWie5XFEKAdnxQom7MTvsXd9W39AvHfljJnEePZpsQV
```
# How It Works
# I Have Internet Access. Why Should I Care?
* [How Iran Turned Off the Internet](https://thewire.in/tech/how-iran-turned-off-the-internet)
* [Building Internet-connected things seems obvious today, but what about when there’s no Internet?](https://back7.co/home/raspberry-pi-recovery-kit)
* [‘Nobody’s got to use the Internet’: A GOP lawmaker’s response to concerns about Web privacy](https://www.washingtonpost.com/news/powerpost/wp/2017/04/15/nobodys-got-to-use-the-internet-a-gop-lawmakers-response-to-concerns-about-web-privacy/)
* [The death of America's net neutrality and how it affects you](https://www.dw.com/en/the-death-of-americas-net-neutrality-and-how-it-affects-you/a-43934099)
* [YouTube and Facebook Are Removing Evidence of Atrocities, Jeopardizing Cases Against War Criminals](https://theintercept.com/2017/11/02/war-crimes-youtube-facebook-syria-rohingya/)
* [Encryption is Not Preventing Law Enforcement from Investigating Crime](https://www.alec.org/article/encryption-is-not-preventing-law-enforcement-from-investigating-crime/)
* [Iraq introduces nightly internet curfew](https://netblocks.org/reports/iraq-introduces-nightly-internet-curfew-JAp1DKBd)
* [Building a Low-Tech Internet](https://www.lowtechmagazine.com/2015/10/how-to-build-a-low-tech-internet.html)
* [Inside Cuba's massive, weekly, human-curated sneakernet](https://boingboing.net/2018/05/03/inside-cubas-massive-weekly.html)
* [CollapseOS](https://collapseos.org/)
* [Russian Law Takes Effect that Gives Government Sweeping Power Over Internet](https://www.npr.org/2019/11/01/775366588/russian-law-takes-effect-that-gives-government-sweeping-power-over-internet)
* [Indian Internet shut down as protests rage against citizenship bill](https://edition.cnn.com/2019/12/12/asia/india-shutdown-citizenship-bill-intl-hnk/index.html)
* [Google goes offline after fibre cables cut](https://www.bbc.com/news/technology-50851420)
* [Authoritarian Nations Are Turning the Internet Into a Weapon](https://onezero.medium.com/authoritarian-nations-are-turning-the-internet-into-a-weapon-10119d4e9992)
# Prior Art
Pigeon borrows many of the ideas set forth by the Secure Scuttlebutt protocol. It is my opinion that SSB is one of the most innovative protocols created in recent years. Without the research and efforts of the SSBC, this project would not be possible, so a big thanks goes out to all the people who make SSB possible.
I've also been inspired by the compactness and minimalism of [SQLite, which should serve as a role model for all of us](https://www.sqlite.org/talks/wroclaw-20090310.pdf).
In many ways, this protocol can be considered an amalgam of the best ideas from both SQLite and Secure Scuttlebutt.
# Constraints and Design Philosophy
* Configuration is bad and should be considered a design comprise in nearly all situations. We will allow a limit of 10 configuration options for all eternity. These are simple key/value pairs. No nesting, no namespacing, no dots, no dashes, no nested config names, no arrays, none of that crap. Seriously, I'm watching you.
* No singletons. No signing authorities, no servers of any kind, even locally, no differentiation between peers (eg: no "super peers").
* Offline-first. Never incorporate TCP or UDP features ever. Such concerns must be handled by application developers. This is to ensure that the protocol is always a viable option for off-grid use cases.
# Roadmap
## Phase I (You Are Here): Build a Working Client
This is the brainstorming phase where the initial proof-of-concept clients will be written. The first protocol client will be slow and may not be suitable for embedded use within a larger application.
## Phase II: Build a Working Application
Using the protocol client from phase I, build an application which uses Pigeon for simulated real-world conditions. This phase will allow for discovery of problems with the draft specification and the first client implementations.
Please see the "What's Possible" section for a list of applications that may be published.
## Phase III: Client Improvements
Once a gauntlet of applications have been built and outstanding design problems have been addressed, re-write existing client libraries. Unlike the client built in Phase I, the clients built in this phase will have a focus on:
* Production-scale performance
* Stability
* Portability to targets like WASM, embedded systems, Windows, etc..
* Ability to be embedded into existing applications.
## Phase IV: Finalize v1 Spec
Once a production-grade client exists, the focus will then become documentation. Using the knowledge gained from phases I-III, we will re-write all documentation, possibly using Gitbook or similar services.
Version 1 of the protocol will be considered complete at this phase and the protocol will be considered "ready for production use".
## Phase V: Stabilize, Maintain, Proliferate
With a finalized spec and a portable client library, the next goal is to promote the product to as many developers as possible and continue to author software that is well suited to the protocol.
# Unanswered Questions
* Ephemeral key exchange
* Standardized general purpose message schemas (follow, unfollow, same_as, etc.)
# The Initial Implementation Should...
* Allow for importing/exporting to SSB via plugins.
* Prefer a monolithic internal structure. Avoid external dependencies except for limited use cases (Eg: crypto libs). Do not break things into smaller pieces until there are at least three real-world reasons to do so. Decoupling a library into a package for only 2 use cases is not acceptable.
* Assume CPU and RAM are not plentiful.
* Assume platform has no networking support. No servers. No hooks for startups, shutdowns, or reboots.
* Assume CPU resources and memory are limited.
* Assume block storage is plentiful when making resource allocation tradeoffs.
* Files are better than sessions.
* ...but be filesystem agnostic. Persistence mechanisms are implementation-specific.
* Provide tamper resistance. Privacy features will be added in v2.
* Be easily ported to new platforms and languages.
* Enable "Free listening"
* Have a formal specification (reference implementations are not OK).
* Minimize conceptual overhead (If it's not needed at least 80% of the time, don't add it).
* Use a serialization format that is deterministic and easy to parse on constrained devices.
# Concepts
**This list is out of date.** Numerous changes and problems were addressed in the implementation of a client. We will update this list when the v0 client is released.
* Base64: The protocol will use Base 64 Encoding with URL and Filename Safe Alphabet as specified in [RFC 4648](https://tools.ietf.org/html/rfc4648). The protocol always uses the standard "=" character for padding. Deviations from this will be explicitly noted.
* Identity: A base64 `ed25519` public key string starting with `@` and ending with `.ed25519`.
* Message Signature: An ED25519 signature starting with a `%` and end with `.sha256`. Messages (covered later) are referenced by a signature.
* String: A 1..62 byte list of ASCII characters, Starting and ending with `"`.
* Blob: Arbitrary binary data, with a current max size of 1.4 MB.
* Blob Hash: A base64'ed SHA256 of a Blob, starting with `&` and ending with `.sha256`.
* Pair: One `string` and on of the following: `Blob Hash|Signature|Identity|String`, joined with a `:` character between the two. See "key" and "value" below.
* Header: A reserved pair of information that is required by the protocol for internal reasons. Not to be confused by a `Pair`, which is user definable. The only Headers the protocol currently uses are `author`, `depth`, `kind`, `prev`. Headers do not have "string quotes" around the key, and the value is delimited by a space (` `) character. Example: `sequence 46`.
* Key: The value to the left of a `:` in a pair. Always a string.
* Value: The value to the right of a `:` in a pair. It is always one of the following: `Blob Hash, Signature, Identity, String`.
* Message: A document with an author, prev, depth, kind, an arbitrary set of attribute pairs and a footer.
* Footer: ??? TODO ????
* Signature: ??? TODO ???
* Kind: A string at the top of a message indicating the message's purpose. Example `"private_message"`, `"mention"`, `"share"`. Kinds may be namespaced by applications using the `.` character.
* Feed: A linked collection of messages.
* Null Signature: An ASCII `0` character, used to indicate the first message in a feed (discussed later)
* Bundle: A specially crafted text archive sent from one peer directly to another peer for the sake of synchronizing and gossiping feeds. Bundles are intricate and require their own document, found [here](bundles.md)
# Running a CLI Client
**NOTE:** Some of the output examples may have changes. This section will be updated upon completion of the [first implementation of a Pigeon protocol client](https://tildegit.org/PigeonProtocolConsortium/pigeon_ruby).
```bash
@ -259,22 +310,3 @@ pigeon bundle consume @GOl+398b2kWeLi6+DCcU0i3AWD6vWmUtocBVYbpkpNk=.ed25519.pige
# =>
```
# The Initial Implementation Should...
* Allow for importing/exporting to SSB via plugins.
* Assume CPU and RAM are not plentiful.
* Assume platform has no networking support. No servers. No hooks for startups, shutdowns, or reboots.
* Assume CPU resources and memory are limited.
* Assume block storage is plentiful when making resource allocation tradeoffs.
* Files are better than sessions.
* ...but be filesystem agnostic. Persistence mechanisms are implementation-specific.
* Provide tamper resistance. Privacy features will be added in v2.
* Be easily ported to new platforms and languages.
* Enable "Free listening"
* Have a formal specification (reference implementations are not OK).
* Have no singletons (no signing authorities, no servers of any kind, even locally)
* Minimize conceptual overhead (If it's not needed at least 80% of the time, don't add it).
* Offline-first. Never incorporate TCP or UDP features ever. Such concerns must be handled by application developers.
* Prefer a monolithic internal structure. Avoid external dependencies except for limited use cases (Eg: crypto libs). Do not break things into smaller pieces until there are at least three real-world reasons to do so. Decoupling a library into a package for only 2 use cases is not acceptable.
* Use a serialization format that is deterministic and easy to parse on constrained devices.