Mostly finish message format document.

This commit is contained in:
Netscape Navigator 2020-05-05 07:42:04 -05:00
parent 03c85bc9dc
commit ca16cfd164
7 changed files with 116 additions and 968 deletions

31
dev_docs.md Normal file
View File

@ -0,0 +1,31 @@
# Messages: The Basic Building Block
The most important protocol concept is that of the "message".
In their most simple form, Pigeon protocol messages are just ASCII text documents. They are human readable and can even be created by hand in a text editor, though most clients will provide better means of authoring messages.
Below is an example of such a message:
```
author @MF312A76JV8S1XWCHV1XR6ANRDMPAT2G5K8PZTGKWV354PR82CD0.ed25519
kind weather_report
prev %ZV85NQS8B1BWQN7YAME1GB0G6XS2AVN610RQTME507DN5ASP2S6G.sha256
depth 3
lipmaa 2
temperature:"22.0C"
webcam_photo:&FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG.sha256
weather_reported_by:@0DC253VW8RP4KGTZP8K5G2TAPMDRNA6RX1VHCWX1S8VJ67A213FM.ed25519
signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08.sig.ed25519
```
The specifics of the message format are explained line-by-line [in the message format explanation document](message_format.md).
Please read this document before continuing.
# Bundles: The Transmission Medium
WORK IN PROGRESS.
# Up Next
This concludes the developer documentation. Please email us with questions. To learn more, continue to the [idea bin](IDEAS.md).

View File

View File

@ -1,913 +0,0 @@
001 ability
001 able
001 about
001 above
001 acceptable
001 across
001 actual
001 actually
001 add
001 added
001 aebrpezjfzwba
001 agnostic
001 aj
001 allocation
001 along
001 alternative
001 amalgam
001 analog
001 ancient
001 anrdmpat
001 anyone
001 append
001 applied
001 apply
001 appreciated
001 approach
001 apps
001 april
001 ar
001 arrays
001 art
001 ascii
001 ask
001 asp
001 assist
001 asw
001 auditing
001 authoring
001 authorities
001 automatically
001 avn
001 avoid
001 ayx
001 ba
001 backup
001 backups
001 backwards
001 baggage
001 bare
001 bc
001 bc-
001 beginning
001 being
001 best
001 bew
001 beyond
001 big
001 blobs
001 block
001 blogging
001 bluetooth
001 bnf
001 board
001 body
001 both
001 boxer
001 brainstorming
001 break
001 browser
001 bug
001 bundle-consuming
001 but
001 bwqn
001 ca
001 cards
001 cause
001 cd
001 censorship
001 changed
001 changes
001 civilizations
001 code
001 common
001 compactness
001 compatibility
001 completed
001 complicated
001 compression
001 compromise
001 compromises
001 computer
001 computers
001 conditions
001 config
001 configurability
001 connection
001 consider
001 consortium
001 constrained
001 constraints
001 contain
001 containing
001 content
001 contents
001 continued
001 contrast
001 control
001 convention
001 cool
001 core
001 could
001 cpu
001 crap
001 create
001 crypto
001 cryptography
001 da
001 dae
001 dashes
001 data-integrity
001 database
001 date
001 db
001 dc
001 decentralized
001 decoupling
001 delay-tolerant
001 dependencies
001 designed
001 deterministic
001 diagram
001 difference
001 different
001 differentiation
001 difficult
001 direct
001 discovery
001 diversity
001 dn
001 dns
001 docs
001 documents
001 does
001 don't
001 dots
001 down
001 downloading
001 ds
001 due
001 early
001 easy
001 easy-to-implement
001 ecosystem
001 edge
001 editor
001 editors
001 efficient
001 efforts
001 embeddable
001 enable
001 encoding
001 end
001 enough
001 ensure
001 enthusiast
001 entirely
001 eternity
001 ever
001 evil
001 examples
001 excellent
001 exchange
001 exchanged
001 exists
001 explore
001 export
001 external
001 ez
001 f-
001 fag
001 famous
001 favor
001 feeds
001 filesystem
001 finalize
001 finalized
001 find
001 finish
001 finished-
001 fixes
001 fj
001 flavored
001 fm
001 follow
001 forged
001 form
001 formal
001 forth
001 framed
001 free
001 fun
001 functioning
001 fv
001 gained
001 game
001 games
001 gateway
001 gateways
001 gauntlet
001 gb
001 gitbook
001 glossary
001 goes
001 grammar
001 grant
001 grid
001 ground
001 guarantee
001 gui
001 guide
001 halting
001 happy
001 hash
001 having
001 header
001 hello
001 helped
001 hgy
001 hhvq
001 hierarchy
001 higher-level
001 hooks
001 hopefully
001 however
001 http
001 hurt
001 i-iii
001 i'm
001 i've
001 ignored
001 ii
001 iii
001 implementation-specific
001 import
001 important
001 improvements
001 incorporate
001 indentation
001 initial
001 innovative
001 integrity
001 interest
001 intermittent
001 internal
001 its
001 iv
001 jqg
001 jspjjqjrvbvgv
001 jssfbaz
001 jttfypkdbhtzj
001 jv
001 jvvp
001 kcw
001 kdfyax
001 kf
001 kfqcwsz
001 kgtzp
001 khmdzzzrwk
001 knowledge
001 known
001 kys
001 lack
001 language
001 larger
001 latency
001 later
001 layer
001 let's
001 level
001 lexer
001 likely
001 limit
001 link
001 links
001 listening
001 local
001 locally
001 lock
001 logger
001 logging
001 look
001 mail
001 main
001 maintenance
001 major
001 makes
001 making
001 mapping
001 maritime
001 maturity
001 mechanisms
001 mention
001 mentioned
001 mesh
001 messenger
001 metal
001 mf
001 microblogging
001 mike
001 minimalism
001 minimize
001 mjsstchyp
001 model
001 monolithic
001 much
001 mule
001 multiple
001 must
001 my
001 names
001 namespacing
001 natural
001 ndatge
001 needed
001 neighboring
001 nested
001 networks
001 never
001 newsgroup
001 nh
001 nice-to-haves
001 nntp
001 node
001 nodejs
001 non-goals
001 nor
001 npwz
001 nqs
001 number
001 off
001 off-grid
001 offer
001 offers
001 offline-first
001 offline-only
001 offloaded
001 ok
001 old
001 online
001 operation
001 opinion
001 opposed
001 option
001 options
001 outreach
001 outstanding
001 own
001 package
001 packaged
001 pages
001 pair
001 pairs
001 parse
001 parsed
001 partial
001 parties
001 pb
001 pcta
001 peer
001 peer's
001 peers-of-peers
001 performance
001 performantly
001 persistence
001 phases
001 philosophy
001 photo
001 pieces
001 pigeons
001 platform
001 plugins
001 point
001 point-of-interest
001 poor
001 portable
001 ported
001 postal
001 power
001 powered
001 pr
001 practice
001 prefer
001 prevent
001 prior
001 product
001 production
001 production-grade
001 proliferate
001 promote
001 promotes
001 proof
001 proof-of-concept
001 prototype
001 prtocol
001 pseudonymity
001 pseudonyms
001 published
001 pubs
001 pztgkwv
001 qdfnsaf
001 qy
001 racing
001 ram
001 read
001 readers
001 ready
001 reasons
001 reboot
001 recent
001 recipient
001 redundant
001 redundantly
001 reference
001 referenced
001 regain
001 regardless
001 remote
001 rename
001 replicate
001 reply
001 report
001 reported
001 require
001 research
001 resistance
001 resource
001 rh
001 role
001 root
001 rp
001 rqtme
001 rsf
001 running
001 rust
001 rx
001 safe
001 say
001 schemas
001 scheme
001 sd
001 secret
001 section
001 seen
001 sense
001 sensor
001 serial
001 seriously
001 serves
001 services
001 sessions
001 set
001 set-
001 should
001 showing
001 shown
001 shutdown
001 similar
001 simpler
001 simplified
001 simulated
001 since
001 singletons
001 site
001 sj
001 slow
001 small
001 smaller
001 sneakernet-only
001 software
001 solar
001 solicited
001 soliciting
001 some
001 something
001 ssh
001 stability
001 stabilize
001 standard
001 starts
001 startup
001 status-quo
001 store
001 store-and-forward
001 structure
001 suitable
001 summary
001 super
001 sync
001 synchronization
001 synchronizing
001 system
001 takes
001 tamper-resistant
001 tampering
001 tapmdrna
001 targets
001 tckp
001 temperature
001 temporarily
001 term
001 terms
001 tf
001 thanks
001 them
001 themselves
001 then
001 these
001 throughput
001 tied
001 tool
001 tpm
001 tradeoffs
001 transfer
001 transmission
001 transport
001 trying
001 turn-based
001 tyson
001 unless
001 until
001 update
001 uptime
001 url
001 used
001 utilize
001 validate
001 vaporsoft
001 verify
001 version
001 vhcwx
001 viable
001 vj
001 vmrg
001 vtj
001 vvgrz
001 wa
001 wanted
001 watching
001 way
001 ways
001 wea
001 webcam
001 were
001 what's
001 where
001 whitespace
001 who
001 windows
001 wip
001 within
001 writing
001 ws
001 wwxyjg
001 wyr
001 xr
001 xs
001 xwchv
001 xyz
001 yame
001 ydqc
001 years
001 ym
001 ymr
001 yqbeg
001 yzady
001 zv
002 addressed
002 after
002 akkgw
002 also
002 altered
002 anonymity
002 app
002 areas
002 ascii-based
002 asked
002 authored
002 available
002 back
002 backed
002 bamboo
002 become
002 benefits
002 between
002 bin
002 borrows
002 bundle
002 bundles
002 cannot
002 cli
002 conceptual
002 concerns
002 concludes
002 contact
002 crockford
002 current
002 custom
002 cx
002 delay
002 development
002 devices
002 differs
002 draft
002 dx
002 each
002 easier
002 efcj
002 eg
002 encrypted
002 encryption
002 even
002 every
002 except
002 existing
002 fat
002 feature
002 feedback
002 few
002 footer
002 found
002 frequently
002 friend
002 get
002 gmv
002 goal
002 gw
002 hand
002 handbook
002 handled
002 hc
002 heabfd
002 high
002 human
002 idea
002 identities
002 if
002 instead
002 intended
002 internet
002 iot
002 it's
002 json
002 just
002 kdkk
002 ktngsdbkmsa
002 languages
002 learn
002 legacy
002 less
002 libraries
002 libs
002 limited
002 long
002 low
002 made
002 maintain
002 malicious
002 means
002 message's
002 messaging
002 might
002 mostly
002 nesting
002 new
002 nodes
002 numerous
002 once
002 other
002 out
002 overhead
002 party
002 people
002 plentiful
002 points
002 possibly
002 privacy
002 production-scale
002 re-write
002 readable
002 referencing
002 reliance
002 replicated
002 resistant
002 roadmap
002 ruby
002 sent
002 series
002 servers
002 signed
002 signing
002 snde
002 spec
002 sqlite
002 string
002 suited
002 tamper
002 te
002 text
002 tgfds
002 things
002 third
002 three
002 tolerant
002 too
002 traditional
002 transmitted
002 two
002 unlike
002 verification
002 vw
002 wasm
002 we've
002 weather
002 well
002 what
002 wish
002 without
002 wkg
002 world
002 would
002 your
003 always
003 any
003 assume
003 base
003 below
003 better
003 blob
003 carriage
003 case
003 concept
003 connectivity
003 created
003 currently
003 design
003 developer
003 easily
003 email
003 empty
003 entry
003 feed
003 files
003 here
003 ideas
003 identity
003 implementations
003 information
003 inspired
003 involved
003 list
003 logs
003 make
003 may
003 medium
003 need
003 only
003 over
003 overview
003 platforms
003 please
003 portability
003 project
003 protocols
003 rather
003 return
003 see
003 serve
003 sig
003 simple
003 social
003 storage
003 swarm
003 tcp
003 their
003 theoretically
003 though
003 time
003 udp
003 value
003 was
003 work
004 although
004 build
004 clients
004 configuration
004 considered
004 continue
004 depth
004 developers
004 etc
004 features
004 first
004 focus
004 format
004 gossip
004 into
004 least
004 like
004 log
004 most
004 network
004 networking
004 next
004 none
004 one
004 prev
004 problems
004 provide
004 questions
004 real-world
004 serialization
004 sha
004 sharing
004 so
004 ssb
004 they
004 us
004 which
004 working
005 allow
005 been
005 built
005 do
005 embedded
005 file
005 help
005 how
005 i
005 implementation
005 key
005 library
005 many
005 more
005 peer-to-peer
005 specification
005 start
005 up
005 uses
006 at
006 author
006 data
006 documentation
006 example
006 has
006 have
006 kind
006 lines
006 lipmaa
006 on
006 scuttlebutt
006 secure
006 signature
006 using
006 when
006 written
006 you
007 application
007 complete
007 ed
007 possible
007 such
007 via
008 all
008 cases
008 line
008 than
008 there
009 sneakernet
009 systems
010 from
010 with
011 an
011 applications
011 as
011 peers
011 support
012 it
012 we
013 can
014 not
014 or
015 message
015 messages
015 use
016 no
017 client
018 phase
018 will
023 by
024 are
024 for
024 that
025 this
027 in
028 pigeon
028 protocol
029 be
041 and
048 is
050 of
055 to
072 the
078 a

BIN
lipmaa.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 40 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 42 KiB

View File

@ -1,9 +1,8 @@
# It Starts With a Message...
# Exploring the Pigeon Message Format
The most important protocol concept is that of the "message".
In their most simple form, Pigeon protocol messages are just ASCII text documents. They are human readable and can even be created by hand in a text editor, though most clients will provide better means of authoring messages.
In the test that follows, we will explore a pigeon message line-by-line.
Below is an example of such a message:
The example message is shown in its entirety below:
```
author @MF312A76JV8S1XWCHV1XR6ANRDMPAT2G5K8PZTGKWV354PR82CD0.ed25519
@ -19,9 +18,7 @@ weather_reported_by:@0DC253VW8RP4KGTZP8K5G2TAPMDRNA6RX1VHCWX1S8VJ67A213FM.ed2551
signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08.sig.ed25519
```
Let's explore each line of a message.
### Line 1: `author`
### Line 1: `Author`
EXAMPLE:
@ -39,7 +36,7 @@ The steps to generate a valid identity are:
2. Add an `@` symbol to the beginning of the string from step 1.
3. Add a `.ed25519` string to the end of the string from step 2.
### Line 2: `kind`
### Line 2: `Kind`
EXAMPLE:
@ -59,7 +56,7 @@ It must meet the following criteria:
* underscores (`_`)
* Symbols used for multihashes, such as `@`, `&` and `%` (covered later).
### Line 3: `prev`
### Line 3: `Prev`
EXAMPLE:
@ -81,7 +78,7 @@ Message multihashes are calculated as follows:
2. The next 52 characters are a [Crockford base 32](https://www.crockford.com/base32.html) SHA512 hash of the previous message's content.
3. The message multihash ends in `.sha512`.
### Line 4: `depth`
### Line 4: `Depth`
EXAMPLE:
@ -93,21 +90,15 @@ Pigeon messages exist in a linear sequence which only moves forward and never "f
Every message has a `depth` field to indicate its "place in line".
Because every message has an ever-increasing integer that never duplicates, every message in a Pigeon feed will have a unique hash. This is true even if messages have identical body content.
### Line 5: `lipmaa`
### Line 5: `Lipmaa`
**THIS FIELD WAS WRITTEN INCORRECTLY. THIS WILL CHANGE SOON. YOU CAN SAFELY MOVE TO THE NEXT SECTION OF THE DOCS**
This concept was borrowed from the [Bamboo protocol](https://github.com/AljoschaMeyer/bamboo#links-and-entry-verification) and [Helger Lipmaa's thesis](https://kodu.ut.ee/~lipmaa/papers/thesis/thesis.pdf).
EXAMPLE:
The `lipmaa` field (often called a "Lipmaa Link") is a special kind of `prev` field that allows partial verification of feeds. This field makes it possible to verify a single message (or subset of messages) without downloading the entire chain of messages.
```
lipmaa 2
```
The `lipmaa` field (often called a "Lipmaa Link") is a special kind of `depth` field that allows partial verification of feeds. This field makes it possible to verify a single message (or subset of messages) without downloading the entire chain of messages.
![](lipmaa.svg)
![](lipmaa.png)
The `lipmaa` field is calculated as follows:
@ -140,41 +131,82 @@ def lipmaa(n)
end
```
### Line 6: Empty carriage return (body start)
### Lines 7: Entry containing a string
### Lines 8: Entry referencing a blob
### Lines 9: Entry referencing a peer's identity
### Lines 10: Empty Carriage Return (footer start)
### Line 6: Body Start (Empty Line)
Once all headers are added, a client must place an empty line (`\n`) after the header.
The empty line signifies the start of the message body.
Some notes about body entries:
* The body of a message starts and ends with an empty line (`\n`).
* Every body entry is a key value pair. Keys and values are separated by a `:` character (no spaces).
* A key must be 1-90 characters in length
* A key cannot contain whitespace or control characters
* A key may contain any of the following characters:
* alphanumeric characters (a-z, A-Z, 0-9)
* dashes (`-`)
* underscores (`_`)
* Symbols used for multihashes, such as `@`, `&` and `%` (covered later).
* A value may be a:
* A string (128 characters or less)
* A multihash referencing an identity (`@`), a message (`%`) or a blob (`&`).
### Lines 7: Entry Containing a String
EXAMPLE:
```
temperature:"22.0C"
```
Body entries are defined by user and contain key/value pairs of application-specific data.
When a key/value pair represents something other than an identity, blob or message ID, a string is used.
Strings can be used for any type of data that does not fit into the other three categories.
Strings must be less than or equal to 128 characters in length.
The example above is the most simple kind of body entry. It specifies an arbitrary string representing the current temperature.
### Lines 8: Entry Referencing a Blob
EXAMPLE:
```
webcam_photo:&FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG.sha256
```
Applications may attach files to messages in the form of blobs. Blobs are referenced using a blob multihash.
* Starts with a `&` character.
* Ends with `.sha256`
* Contains exactly 52 characters between the `&` and `.sha256` parts. This is a SHA256 hash of the blob's content, represented in Crockford Base 32 encoding.
A blob is referenced in a message's key or value. A client will include a blob's content in a "bundle" (explained later).
### Lines 9: Entry Referencing a Peer's Identity
EXAMPLE:
```
weather_reported_by:@0DC253VW8RP4KGTZP8K5G2TAPMDRNA6RX1VHCWX1S8VJ67A213FM.ed25519
```
A message may reference other identities (or its own identity) by using an identity sigil either in the key or value portion of the entry.
This is analogous to "social tagging" seen in many social networks.
### Lines 10: Empty Carriage Return (Footer Start)
The last part of a message is the footer. Like a message body, a message footer starts and ends with an empty line.
The footer is essential for ensuring the tamper resistant properties of a Pigeon message.
### Lines 11: Signature Line
### Lines 12: Empty Carriage Return (message end)
# Sharing Messages and Blobs via Bundles
# Glossary of Terms
**This list is out of date.** Numerous changes and problems were addressed in the implementation of a client. We will update this list in May of 2020.
EXAMPLE:
* Header
* Blob
* Crockford Base32
* NONE
* String
* Signature
* Value
* Message
* Blob Hash
* Key
* Pair
* Footer
* Bundle
* Kind
* Message Signature
* Feed
* Identity
```
signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08.sig.ed25519
```
# Running a CLI Client
Pigeon currently has one CLI available. It is written in Ruby. Documentation can be found [here](https://tildegit.org/PigeonProtocolConsortium/pigeon_ruby)
# Up Next
This concludes the developer documentation. Please email us with questions. To learn more, continue to the [idea bin](IDEAS.md).
A signature starts with the word `signature` followed by a space.
After that, the body (including the trailing `\n`) is signed using the author's ED25519 key.
The signature is encoded with Crockford base 32.
The signature ends with `.sig.ed25519`.
An empty carraige return is added after the signature line.