Various updates to documentation

This commit is contained in:
Netscape Navigator 2020-05-01 09:21:57 -05:00
parent 692b309a7a
commit 2ed191befe
4 changed files with 993 additions and 35 deletions

View File

@ -1,24 +1,62 @@
# Concepts
# It Starts With a Message...
The most important prtocol concept is that of the "message".
In their most simple form, Pigeon protocol messages are just ASCII text documents. They are human readable and can even be created by hand in a text editor, though most clients will provide better means of authoring messages.
Below is an example of such a message:
```
author @MF312A76JV8S1XWCHV1XR6ANRDMPAT2G5K8PZTGKWV354PR82CD0.ed25519
kind weather_report
prev %ZV85NQS8B1BWQN7YAME1GB0G6XS2AVN610RQTME507DN5ASP2S6G.sha256
depth 3
lipmaa 2
temperature:"22.0C"
webcam_photo:&FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG.sha256
weather_reported_by:@0DC253VW8RP4KGTZP8K5G2TAPMDRNA6RX1VHCWX1S8VJ67A213FM.ed25519
signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08.sig.ed25519
```
Let's explore each line of a message.
### Line 1: `author`
### Line 2: `kind`
### Line 3: `prev`
### Line 4: `depth`
### Line 5: `lipmaa`
### Line 6: Empty carriage return (body start)
### Lines 7: Entry containing a string
### Lines 8: Entry referencing a blob
### Lines 9: Entry referencing a peer's identity
### Lines 10: Empty Carriage Return (footer start)
### Lines 11: Signature Line
### Lines 12: Empty Carriage Return (message end)
# Sharing Messages and Blobs via Bundles
# Glossary of Terms
**This list is out of date.** Numerous changes and problems were addressed in the implementation of a client. We will update this list in May of 2020.
* Base64: The protocol will use Base 64 Encoding with URL and Filename Safe Alphabet as specified in [RFC 4648](https://tools.ietf.org/html/rfc4648). The protocol always uses the standard "=" character for padding. Deviations from this will be explicitly noted.
* Identity: A base64 `ed25519` public key string starting with `@` and ending with `.ed25519`.
* Message Signature: An ED25519 signature starting with a `%` and end with `.sha256`. Messages (covered later) are referenced by a signature.
* String: A 1..62 byte list of ASCII characters, Starting and ending with `"`.
* Blob: Arbitrary binary data, with a current max size of 1.4 MB.
* Blob Hash: A base64'ed SHA256 of a Blob, starting with `&` and ending with `.sha256`.
* Pair: One `string` and on of the following: `Blob Hash|Signature|Identity|String`, joined with a `:` character between the two. See "key" and "value" below.
* Header: A reserved pair of information that is required by the protocol for internal reasons. Not to be confused by a `Pair`, which is user definable. The only Headers the protocol currently uses are `author`, `depth`, `kind`, `prev`. Headers do not have "string quotes" around the key, and the value is delimited by a space (` `) character. Example: `sequence 46`.
* Key: The value to the left of a `:` in a pair. Always a string.
* Value: The value to the right of a `:` in a pair. It is always one of the following: `Blob Hash, Signature, Identity, String`.
* Message: A document with an author, prev, depth, kind, an arbitrary set of attribute pairs and a footer.
* Footer: ??? TODO ????
* Signature: ??? TODO ???
* Kind: A string at the top of a message indicating the message's purpose. Example `"private_message"`, `"mention"`, `"share"`. Kinds may be namespaced by applications using the `.` character.
* Feed: A linked collection of messages.
* NONE: ????
* Bundle: A specially crafted text archive sent from one peer directly to another peer for the sake of synchronizing and gossiping feeds. Bundles are intricate and require their own document, found [here](bundles.md)
* Header
* Blob
* Crockford Base32
* NONE
* String
* Signature
* Value
* Message
* Blob Hash
* Key
* Pair
* Footer
* Bundle
* Kind
* Message Signature
* Feed
* Identity
# Running a CLI Client
@ -27,8 +65,4 @@ Pigeon currently has one CLI available. It is written in Ruby. Documentation can
# Up Next
This concludes the roadmap. To learn more, continue to the [idea bin](IDEAS.md).
# Up Next
This concludes the frequently asked questions section. To learn more, continue to the [roadmap](ROADMAP.md)
This concludes the developer documentation. Please email us with questions. To learn more, continue to the [idea bin](IDEAS.md).

4
FAQ.md
View File

@ -19,12 +19,12 @@
# When is Pigeon the Wrong Choice?
* When the application requires true deletion of data, ephemeral data or mutability of previously created data. Pigeon feeds are immutable, append-only and permanently replicated by peers.
* When the application requires realtime interactions or does not benefit from delay tolerance. Support for TCP or UDP sockets is unlikely to ever be added to core libraries.
* When the application requires realtime interactions or does not benefit from delay tolerance. Support for TCP or UDP sockets will never be added.
* Extremely "chatty" protocols. Pigeon was built with the assumption that data storage is cheap and data transfer is expensive and slow. Use cases with complex handshakes, pinging or timeouts may not be well suited to this protocol.
# Is This a Blockchain?
It's different than a block chain despite some similarity. A global blockchain is a singleton, and Pigeon has no singletons. Each Pigeon node maintains their own feed of messages rather than sharing a global feed. When you trust a peer, you agree to replicate their feed. You also replicate the feed of their peers. Unlike a blockchain, there are many feeds (rather than one) and they do not require consensus to coexist. Additionally, forking of feeds is not supported by Pigeon.
It's different than a block chain despite some similarity. A global blockchain is a singleton, and Pigeon has no singletons. Each Pigeon node maintains their own feed of messages rather than sharing a global feed. When you trust a peer, you agree to replicate their feed. You also replicate the feed of their peers. Unlike a blockchain, there are many feeds (rather than one) and they do not require consensus or coordination to coexist. Additionally, forking of feeds is not supported by Pigeon.
# Up Next

View File

@ -20,6 +20,15 @@ Email `contact` at `vaporsoft.xyz` to ask questions or get involved. Your feedba
* [Developer Docs and Specification](DEV_DOCS.md)
* [Ideas and Features](IDEAS.md)
## Help Wanted
If you wish to become involved with protocol development, there are a few areas we need help in currently:
Email us for more information.
* Documentation editors, proof readers and feedback.
* We need real-world applications to be built using the protocol! We are happy to assist you along the way.
* We need a BNF grammar for Pigeon messages.
# Overview
Pigeon can serve a number of use cases. Below are some examples:
@ -72,20 +81,20 @@ In summary, Pigeon protocol offers benefits above what a traditional sneakernet
## How Pigeon Differs From Secure Scuttlebutt (SSB)
As mentioned, Pigeon was inspired mostly by the work of of Secure Scuttlebutt. Pigeon takes a different approach in a few areas, however.
As mentioned, Pigeon was inspired mostly by the work of Secure Scuttlebutt. Pigeon takes a different approach in a few areas, however.
1. No reliance on networking in the core library. Although SSB is theoretically able to support Sneakernet-only operation, it is difficult in practice due to reliance on UDP, TCP, and in the case of pubs, DNS.
1. Pigeon uses a custom key/value serialization format instead of JSON. This has two benefits:
* Serialization and signing is much simpler. Indentation and whitespace are less likely to cause verification problems.
* Unlike JSON, pigeon messages do not allow nesting, which promotes simplified message schemas.
1. Pigeon uses Crockford flavored Base32 rather than URL safe Base64. This makes it easier to support old or low powered systems. easier to support FAT16 / embedded systems you might want to have an FAQ section with pretty much this
1. Pigeon uses Crockford flavored Base32 rather than URL safe Base64. This makes it easier to support old or low powered systems and is easier to support FAT16 / embedded systems.
1. Pigeon was designed for portability from the beginning. It has a small enough conceptual overhead that it will actually be possible to support platforms other than NodeJS. Complicated features (like network support) are ignored in favor of an easy-to-implement standard
1. It uses Lipmaa links, so you can verify a feed without downloading all 10,000 messages. This was inspired by the work of the Bamboo protocol.
1. It uses Lipmaa links, so you can verify a feed without downloading all messages. This was inspired by the work of the Bamboo protocol.
## How Does It Work?
Each node in a swarm of peers has a local "log". The log is an append only feed of messages written in an ASCII-based serialization format. Messages are signed with a secret key to validate a message's integrity and to prevent tampering by untrusted peers. Nodes in the swarm "follow" other logs from peers of interest. Nodes always replicate the logs of their peers and "gossip" information about peers across the swarm. Gossip information is packaged into "bundles" which contain backups of peer logs in an efficient binary format that can be easily transmitted via sneakernet, direct serial connection, or any high throughput medium, regardless of latency.
Each node in a swarm of peers has a local "log". The log is an append only feed of messages written in an ASCII-based serialization format. Messages are signed with a secret key to validate a message's integrity and to prevent tampering by malicious peers. Nodes in the swarm "follow" other logs from peers of interest. Nodes always replicate the logs of their peers and "gossip" information about peers across the swarm. Gossip information is packaged into "bundles" which contain backups of peer logs in an efficient format that can be easily transmitted via sneakernet, direct serial connection, or any high throughput medium, regardless of latency.
Log synchronization via Sneakernet is the main use case for Pigeon messages to be transmitted. SD Cards sent via postal mail offer an excellent medium for transmission of Pigeon messages, although any data transfer medium is theoretically possible.
@ -133,13 +142,14 @@ The [first working implementation of a Pigeon protocol client](https://tildegit.
## Constraints and Design Philosophy
* Support Offline-first by being offline-only. Never incorporate TCP or UDP features ever. Such concerns must be handled by higher-level protocols or by application developers. This is to ensure that the protocol is always a viable option for off-grid use cases.
* Offline-first means offline-only. Never incorporate TCP or UDP features ever. Such concerns must be handled by higher-level protocols or by application developers. This is to ensure that the protocol is always a viable option for off-grid use cases.
* Natural is better than simple. Convention over configuration. Do not make plugins for common use cases unless it would hurt portability.
* Prefer a monolithic internal structure. Avoid external dependencies except for limited use cases (Eg: crypto libs). Do not break things into smaller pieces until there are at least three real-world reasons to do so. Decoupling a library into a package for only 2 use cases is not acceptable.
* Maintain ecosystem diversity by having a protocol that can be easily and entirely ported to new languages and platforms.
* No singletons. No signing authorities, no servers of any kind, even locally, no differentiation between peers (eg: no "super peers").
* Configuration is always a design comprise. We will allow a limit of 10 configuration options for all eternity. These are simple key/value pairs. No nesting, no namespacing, no dots, no dashes, no nested config names, no arrays, none of that crap. Seriously, I'm watching you.
* Configuration is always a design compromise. We will allow a limit of 10 configuration options for all eternity. These are simple key/value pairs. No nesting, no namespacing, no dots, no dashes, no nested config names, no arrays, none of that crap. Seriously, I'm watching you.
* Assume CPU and RAM are not plentiful.
* Assume platform has no networking support. No servers. No hooks for startups, shutdowns, or reboots.
* Assume platform has no networking support. No servers. No hooks for startup, shutdown, or reboot.
* Assume block storage is plentiful when making resource allocation tradeoffs.
* Files are better than sessions, but be filesystem agnostic. Persistence mechanisms are implementation-specific.
* Provide tamper resistance. Privacy features will be added later.
@ -147,13 +157,14 @@ The [first working implementation of a Pigeon protocol client](https://tildegit.
* Have a formal specification (reference implementations are not OK).
* Minimize conceptual overhead (If it's not needed at least 80% of the time, don't add it).
* Use a serialization format that is deterministic and easy to parse on constrained devices.
* Natural is better than simple. Convention over configuration. Do not make plugins for common use cases unless it would hurt portability.
* Backwards compatibility. Numerous compromises have been made to support legacy systems, such as devices that lack network support and FAT16 file systems.
## Non-Goals
* Extreme configurability.
* Networking support at the protocol level (TCP, UDP, SSH, HTTP, Etc..)
* Networking support: The protocol will make no mention of TCP, UDP, SSH, HTTP, Etc..
* Configurability: Configuration is the root of all evil. Instead of trying to create something that serves every use case, focus on >80% use cases and allow edge cases to be handled by application developers.
* Anonymity: Though pseudonymity is possible via the use of multiple identities, the concept of anonymity or is not well suited to this protocol. Every message is signed by a known identity, though identities are easily created and can be backed by pseudonyms for privacy.
* Encryption. Like many internet protocols, encryption concerns are offloaded to the transport layer. Consider using an encrypted file system or encrypted compression application to store data.
# Up Next

913
glossary_terms.md Normal file
View File

@ -0,0 +1,913 @@
001 ability
001 able
001 about
001 above
001 acceptable
001 across
001 actual
001 actually
001 add
001 added
001 aebrpezjfzwba
001 agnostic
001 aj
001 allocation
001 along
001 alternative
001 amalgam
001 analog
001 ancient
001 anrdmpat
001 anyone
001 append
001 applied
001 apply
001 appreciated
001 approach
001 apps
001 april
001 ar
001 arrays
001 art
001 ascii
001 ask
001 asp
001 assist
001 asw
001 auditing
001 authoring
001 authorities
001 automatically
001 avn
001 avoid
001 ayx
001 ba
001 backup
001 backups
001 backwards
001 baggage
001 bare
001 bc
001 bc-
001 beginning
001 being
001 best
001 bew
001 beyond
001 big
001 blobs
001 block
001 blogging
001 bluetooth
001 bnf
001 board
001 body
001 both
001 boxer
001 brainstorming
001 break
001 browser
001 bug
001 bundle-consuming
001 but
001 bwqn
001 ca
001 cards
001 cause
001 cd
001 censorship
001 changed
001 changes
001 civilizations
001 code
001 common
001 compactness
001 compatibility
001 completed
001 complicated
001 compression
001 compromise
001 compromises
001 computer
001 computers
001 conditions
001 config
001 configurability
001 connection
001 consider
001 consortium
001 constrained
001 constraints
001 contain
001 containing
001 content
001 contents
001 continued
001 contrast
001 control
001 convention
001 cool
001 core
001 could
001 cpu
001 crap
001 create
001 crypto
001 cryptography
001 da
001 dae
001 dashes
001 data-integrity
001 database
001 date
001 db
001 dc
001 decentralized
001 decoupling
001 delay-tolerant
001 dependencies
001 designed
001 deterministic
001 diagram
001 difference
001 different
001 differentiation
001 difficult
001 direct
001 discovery
001 diversity
001 dn
001 dns
001 docs
001 documents
001 does
001 don't
001 dots
001 down
001 downloading
001 ds
001 due
001 early
001 easy
001 easy-to-implement
001 ecosystem
001 edge
001 editor
001 editors
001 efficient
001 efforts
001 embeddable
001 enable
001 encoding
001 end
001 enough
001 ensure
001 enthusiast
001 entirely
001 eternity
001 ever
001 evil
001 examples
001 excellent
001 exchange
001 exchanged
001 exists
001 explore
001 export
001 external
001 ez
001 f-
001 fag
001 famous
001 favor
001 feeds
001 filesystem
001 finalize
001 finalized
001 find
001 finish
001 finished-
001 fixes
001 fj
001 flavored
001 fm
001 follow
001 forged
001 form
001 formal
001 forth
001 framed
001 free
001 fun
001 functioning
001 fv
001 gained
001 game
001 games
001 gateway
001 gateways
001 gauntlet
001 gb
001 gitbook
001 glossary
001 goes
001 grammar
001 grant
001 grid
001 ground
001 guarantee
001 gui
001 guide
001 halting
001 happy
001 hash
001 having
001 header
001 hello
001 helped
001 hgy
001 hhvq
001 hierarchy
001 higher-level
001 hooks
001 hopefully
001 however
001 http
001 hurt
001 i-iii
001 i'm
001 i've
001 ignored
001 ii
001 iii
001 implementation-specific
001 import
001 important
001 improvements
001 incorporate
001 indentation
001 initial
001 innovative
001 integrity
001 interest
001 intermittent
001 internal
001 its
001 iv
001 jqg
001 jspjjqjrvbvgv
001 jssfbaz
001 jttfypkdbhtzj
001 jv
001 jvvp
001 kcw
001 kdfyax
001 kf
001 kfqcwsz
001 kgtzp
001 khmdzzzrwk
001 knowledge
001 known
001 kys
001 lack
001 language
001 larger
001 latency
001 later
001 layer
001 let's
001 level
001 lexer
001 likely
001 limit
001 link
001 links
001 listening
001 local
001 locally
001 lock
001 logger
001 logging
001 look
001 mail
001 main
001 maintenance
001 major
001 makes
001 making
001 mapping
001 maritime
001 maturity
001 mechanisms
001 mention
001 mentioned
001 mesh
001 messenger
001 metal
001 mf
001 microblogging
001 mike
001 minimalism
001 minimize
001 mjsstchyp
001 model
001 monolithic
001 much
001 mule
001 multiple
001 must
001 my
001 names
001 namespacing
001 natural
001 ndatge
001 needed
001 neighboring
001 nested
001 networks
001 never
001 newsgroup
001 nh
001 nice-to-haves
001 nntp
001 node
001 nodejs
001 non-goals
001 nor
001 npwz
001 nqs
001 number
001 off
001 off-grid
001 offer
001 offers
001 offline-first
001 offline-only
001 offloaded
001 ok
001 old
001 online
001 operation
001 opinion
001 opposed
001 option
001 options
001 outreach
001 outstanding
001 own
001 package
001 packaged
001 pages
001 pair
001 pairs
001 parse
001 parsed
001 partial
001 parties
001 pb
001 pcta
001 peer
001 peer's
001 peers-of-peers
001 performance
001 performantly
001 persistence
001 phases
001 philosophy
001 photo
001 pieces
001 pigeons
001 platform
001 plugins
001 point
001 point-of-interest
001 poor
001 portable
001 ported
001 postal
001 power
001 powered
001 pr
001 practice
001 prefer
001 prevent
001 prior
001 product
001 production
001 production-grade
001 proliferate
001 promote
001 promotes
001 proof
001 proof-of-concept
001 prototype
001 prtocol
001 pseudonymity
001 pseudonyms
001 published
001 pubs
001 pztgkwv
001 qdfnsaf
001 qy
001 racing
001 ram
001 read
001 readers
001 ready
001 reasons
001 reboot
001 recent
001 recipient
001 redundant
001 redundantly
001 reference
001 referenced
001 regain
001 regardless
001 remote
001 rename
001 replicate
001 reply
001 report
001 reported
001 require
001 research
001 resistance
001 resource
001 rh
001 role
001 root
001 rp
001 rqtme
001 rsf
001 running
001 rust
001 rx
001 safe
001 say
001 schemas
001 scheme
001 sd
001 secret
001 section
001 seen
001 sense
001 sensor
001 serial
001 seriously
001 serves
001 services
001 sessions
001 set
001 set-
001 should
001 showing
001 shown
001 shutdown
001 similar
001 simpler
001 simplified
001 simulated
001 since
001 singletons
001 site
001 sj
001 slow
001 small
001 smaller
001 sneakernet-only
001 software
001 solar
001 solicited
001 soliciting
001 some
001 something
001 ssh
001 stability
001 stabilize
001 standard
001 starts
001 startup
001 status-quo
001 store
001 store-and-forward
001 structure
001 suitable
001 summary
001 super
001 sync
001 synchronization
001 synchronizing
001 system
001 takes
001 tamper-resistant
001 tampering
001 tapmdrna
001 targets
001 tckp
001 temperature
001 temporarily
001 term
001 terms
001 tf
001 thanks
001 them
001 themselves
001 then
001 these
001 throughput
001 tied
001 tool
001 tpm
001 tradeoffs
001 transfer
001 transmission
001 transport
001 trying
001 turn-based
001 tyson
001 unless
001 until
001 update
001 uptime
001 url
001 used
001 utilize
001 validate
001 vaporsoft
001 verify
001 version
001 vhcwx
001 viable
001 vj
001 vmrg
001 vtj
001 vvgrz
001 wa
001 wanted
001 watching
001 way
001 ways
001 wea
001 webcam
001 were
001 what's
001 where
001 whitespace
001 who
001 windows
001 wip
001 within
001 writing
001 ws
001 wwxyjg
001 wyr
001 xr
001 xs
001 xwchv
001 xyz
001 yame
001 ydqc
001 years
001 ym
001 ymr
001 yqbeg
001 yzady
001 zv
002 addressed
002 after
002 akkgw
002 also
002 altered
002 anonymity
002 app
002 areas
002 ascii-based
002 asked
002 authored
002 available
002 back
002 backed
002 bamboo
002 become
002 benefits
002 between
002 bin
002 borrows
002 bundle
002 bundles
002 cannot
002 cli
002 conceptual
002 concerns
002 concludes
002 contact
002 crockford
002 current
002 custom
002 cx
002 delay
002 development
002 devices
002 differs
002 draft
002 dx
002 each
002 easier
002 efcj
002 eg
002 encrypted
002 encryption
002 even
002 every
002 except
002 existing
002 fat
002 feature
002 feedback
002 few
002 footer
002 found
002 frequently
002 friend
002 get
002 gmv
002 goal
002 gw
002 hand
002 handbook
002 handled
002 hc
002 heabfd
002 high
002 human
002 idea
002 identities
002 if
002 instead
002 intended
002 internet
002 iot
002 it's
002 json
002 just
002 kdkk
002 ktngsdbkmsa
002 languages
002 learn
002 legacy
002 less
002 libraries
002 libs
002 limited
002 long
002 low
002 made
002 maintain
002 malicious
002 means
002 message's
002 messaging
002 might
002 mostly
002 nesting
002 new
002 nodes
002 numerous
002 once
002 other
002 out
002 overhead
002 party
002 people
002 plentiful
002 points
002 possibly
002 privacy
002 production-scale
002 re-write
002 readable
002 referencing
002 reliance
002 replicated
002 resistant
002 roadmap
002 ruby
002 sent
002 series
002 servers
002 signed
002 signing
002 snde
002 spec
002 sqlite
002 string
002 suited
002 tamper
002 te
002 text
002 tgfds
002 things
002 third
002 three
002 tolerant
002 too
002 traditional
002 transmitted
002 two
002 unlike
002 verification
002 vw
002 wasm
002 we've
002 weather
002 well
002 what
002 wish
002 without
002 wkg
002 world
002 would
002 your
003 always
003 any
003 assume
003 base
003 below
003 better
003 blob
003 carriage
003 case
003 concept
003 connectivity
003 created
003 currently
003 design
003 developer
003 easily
003 email
003 empty
003 entry
003 feed
003 files
003 here
003 ideas
003 identity
003 implementations
003 information
003 inspired
003 involved
003 list
003 logs
003 make
003 may
003 medium
003 need
003 only
003 over
003 overview
003 platforms
003 please
003 portability
003 project
003 protocols
003 rather
003 return
003 see
003 serve
003 sig
003 simple
003 social
003 storage
003 swarm
003 tcp
003 their
003 theoretically
003 though
003 time
003 udp
003 value
003 was
003 work
004 although
004 build
004 clients
004 configuration
004 considered
004 continue
004 depth
004 developers
004 etc
004 features
004 first
004 focus
004 format
004 gossip
004 into
004 least
004 like
004 log
004 most
004 network
004 networking
004 next
004 none
004 one
004 prev
004 problems
004 provide
004 questions
004 real-world
004 serialization
004 sha
004 sharing
004 so
004 ssb
004 they
004 us
004 which
004 working
005 allow
005 been
005 built
005 do
005 embedded
005 file
005 help
005 how
005 i
005 implementation
005 key
005 library
005 many
005 more
005 peer-to-peer
005 specification
005 start
005 up
005 uses
006 at
006 author
006 data
006 documentation
006 example
006 has
006 have
006 kind
006 lines
006 lipmaa
006 on
006 scuttlebutt
006 secure
006 signature
006 using
006 when
006 written
006 you
007 application
007 complete
007 ed
007 possible
007 such
007 via
008 all
008 cases
008 line
008 than
008 there
009 sneakernet
009 systems
010 from
010 with
011 an
011 applications
011 as
011 peers
011 support
012 it
012 we
013 can
014 not
014 or
015 message
015 messages
015 use
016 no
017 client
018 phase
018 will
023 by
024 are
024 for
024 that
025 this
027 in
028 pigeon
028 protocol
029 be
041 and
048 is
050 of
055 to
072 the
078 a