7.8 KiB
Raw Permalink Blame History

Fundamental Concept: Free Listening

The concept of "Free Listening" is mentioned often when developing the protocol. The original concept is borrowed from the Secure Scuttlebutt protocol and is well summarized in a 2018 work by Andre Staltz:

Pull is the opposite of push. Instead of realtime push notifications, you choose when you want to get updates about new content. Instead of free speech, as in the right to broadcast, we value free listening, which is the right not to be shouted at. Most social networks allow people to basically insert—push—themselves into a community. But Scuttlebutt invites are the other way around: someone in the community has to pull you in.

We are not proponents of total self-reliance. With full independence, youd just be a lonely node in your social graph. The real value of the social graph is not the amount of nodes, its the amount of edges. They represent dependence on each other.

Your friend is your backup, literally. If you happen to lose all your data because your computer explodes, all you need is your crypto key pair, your identity, which is a small file. Then you can re-download everything you said and liked and did from a friend nearby. Nothing will be lost. This happened to someone in the community, and we were pleasantly surprised how well it worked.

Pigeon has many of the same goals as Secure Scuttlebutt, including the user's right to Free Listening. Keep this in mind as you navigate the various design decisions and compromises of the protocol.

Fundamental Concept: Content Addressing

Pigeon Protocol relies heavily on content addressing and the SHA-256 hashing algorithm. This means that instead of assigning arbitrary or user-generated names to resources (as is the case with web addresses), names are assigned using the SHA-256 hash algorithm.

Please watch [this video] if you are unfamiliar with hashing algorithms.

Fundamental Concept: Digital Signatures

Pigeon uses ED25519 to generate and verify message signatures. An understanding of digital signing algorithms is assumed. If you need help understanding this concept, please contact us on the mailing list. Due to time constraints, I will defer to existing resources that exist online.

Messages: The Basic Building Block

The most important protocol concept is that of the "message". In their most simple form, Pigeon protocol messages are just text documents. Text encoding is determined by the client, but UTF-8 is highly encouraged for maximum interoperability. They are human readable and can even be created by hand in a text editor, though most clients will provide better means of authoring messages.

Below is an example of such a message:

depth 123
kind weather_report



The specifics of the message format are explained line-by-line in the message format explanation document. Please read the message format explanation document before continuing.

Bundles: The Transmission Medium

A message without a receiver is not very useful. Any peer in a Pigeon cluster has two things it needs to share with its peers:

  • Messages, created by the peer or forwarded on behalf of a "peer of a peer"
  • Files, in the form of blobs

Pigeon sends messages and files to peers through the use of "bundles". Bundles are a file directory arranged in a predictable layout that is specified by the protocol. Peers share bundles with each other through a variety of means. When a peer unpacks a bundle into their local database, the database contents are updated with new information from peers such as messages and blobs (files).

The directories are transmitted over any medium that supports file transfer (the protocol does not involve itself with such concerns).

Pigeon follows the philosophy of "offline-first means offline-only". Bundle files could theoretically be:

  • Transmitted over removable media such as USB drives or optical media (CD).
  • Compressed using zip compression to increase storage capacity.
  • Encrypted using tools like PGP
  • Hosted and dynamically updated on a web server for network retrieval.

The protocol does not place any restrictions on how bundles are handled, transported, compressed or encrypted. It does, however, dictate the internal layout of the bundle. Authors of protocol clients must pay special attention to how bundles are created to ensure security and interoperability between client implementations.

Where Do Messages Go in a Bundle?

A bundle's message payload (as opposed to its blob payload) is contained in a plaintext file at the root bundle directory. The file is named messages.pgn. Messages are joined together via carriage return (\n).

Typically, messages.pgn will contain messages created by the bundle's author, plus messages created by the author's peers.

Here is an example of the contents of a messages.pgn file:

depth 0
kind chat_message
prev NONE

content:"Hello, world!"


depth 1
kind chat_message

content:"Good morning!"


depth 2
kind like



The example above is multiple pigeon messages joined together with a carriage return ("\n"). Notice that not all messages were created by the same author.

A peer can use the file above to update their local database. It is important to note that a client will always reject a message that it cannot verify. Below is an example of a refused update:

  • Peer A has verified peer B's messages up to depth 5.

  • Peer C sends Peer A a new bundle.

  • The bundle contains messages authored by peer B, starting at depth 8.

  • Since Peer A cannot verify message at depth 8 until it receives depth 7 and depth 6, the messages are rejected by peer A.

  • Clients always reject messages that cannot be verified. For example, if a messages.pgn file starts at depth 6 for a peer USER.ABC, and the local client has only verified USER.ABCs feed up to depth 3, the messages in the bundle will be rejected by the local client because it does not have enough information available to verify the authenticity of the message.

  • Messages must be ordered by depth for a particular author.

Where Do Files Go in a Bundle?

Files ("blobs") are transferred alongside the *.pgn message bundle. It is the responsibility of the receiver (not the sender) to calculate the multihash of an incoming file. The ensures that files are not misrepresented or tampered with.

Files added to a blob must follow these naming rules:

  • The filename extension is located in the same directory as the messages.pgn file.
  • The filename must follow 8.3 filename conventions.
  • The file extension must be .blb.
  • The filename cannot be longer than 8 chars.