diff --git a/dev_docs.md b/dev_docs.md new file mode 100644 index 0000000..0a57ea4 --- /dev/null +++ b/dev_docs.md @@ -0,0 +1,31 @@ +# Messages: The Basic Building Block + +The most important protocol concept is that of the "message". +In their most simple form, Pigeon protocol messages are just ASCII text documents. They are human readable and can even be created by hand in a text editor, though most clients will provide better means of authoring messages. + +Below is an example of such a message: + +``` +author @MF312A76JV8S1XWCHV1XR6ANRDMPAT2G5K8PZTGKWV354PR82CD0.ed25519 +kind weather_report +prev %ZV85NQS8B1BWQN7YAME1GB0G6XS2AVN610RQTME507DN5ASP2S6G.sha256 +depth 3 +lipmaa 2 + +temperature:"22.0C" +webcam_photo:&FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG.sha256 +weather_reported_by:@0DC253VW8RP4KGTZP8K5G2TAPMDRNA6RX1VHCWX1S8VJ67A213FM.ed25519 + +signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08.sig.ed25519 +``` + +The specifics of the message format are explained line-by-line [in the message format explanation document](message_format.md). +Please read this document before continuing. + +# Bundles: The Transmission Medium + +WORK IN PROGRESS. + +# Up Next + +This concludes the developer documentation. Please email us with questions. To learn more, continue to the [idea bin](IDEAS.md). diff --git a/FAQ.md b/faq.md similarity index 100% rename from FAQ.md rename to faq.md diff --git a/glossary_terms.md b/glossary_terms.md deleted file mode 100644 index eeb9f7e..0000000 --- a/glossary_terms.md +++ /dev/null @@ -1,913 +0,0 @@ -001 ability -001 able -001 about -001 above -001 acceptable -001 across -001 actual -001 actually -001 add -001 added -001 aebrpezjfzwba -001 agnostic -001 aj -001 allocation -001 along -001 alternative -001 amalgam -001 analog -001 ancient -001 anrdmpat -001 anyone -001 append -001 applied -001 apply -001 appreciated -001 approach -001 apps -001 april -001 ar -001 arrays -001 art -001 ascii -001 ask -001 asp -001 assist -001 asw -001 auditing -001 authoring -001 authorities -001 automatically -001 avn -001 avoid -001 ayx -001 ba -001 backup -001 backups -001 backwards -001 baggage -001 bare -001 bc -001 bc- -001 beginning -001 being -001 best -001 bew -001 beyond -001 big -001 blobs -001 block -001 blogging -001 bluetooth -001 bnf -001 board -001 body -001 both -001 boxer -001 brainstorming -001 break -001 browser -001 bug -001 bundle-consuming -001 but -001 bwqn -001 ca -001 cards -001 cause -001 cd -001 censorship -001 changed -001 changes -001 civilizations -001 code -001 common -001 compactness -001 compatibility -001 completed -001 complicated -001 compression -001 compromise -001 compromises -001 computer -001 computers -001 conditions -001 config -001 configurability -001 connection -001 consider -001 consortium -001 constrained -001 constraints -001 contain -001 containing -001 content -001 contents -001 continued -001 contrast -001 control -001 convention -001 cool -001 core -001 could -001 cpu -001 crap -001 create -001 crypto -001 cryptography -001 da -001 dae -001 dashes -001 data-integrity -001 database -001 date -001 db -001 dc -001 decentralized -001 decoupling -001 delay-tolerant -001 dependencies -001 designed -001 deterministic -001 diagram -001 difference -001 different -001 differentiation -001 difficult -001 direct -001 discovery -001 diversity -001 dn -001 dns -001 docs -001 documents -001 does -001 don't -001 dots -001 down -001 downloading -001 ds -001 due -001 early -001 easy -001 easy-to-implement -001 ecosystem -001 edge -001 editor -001 editors -001 efficient -001 efforts -001 embeddable -001 enable -001 encoding -001 end -001 enough -001 ensure -001 enthusiast -001 entirely -001 eternity -001 ever -001 evil -001 examples -001 excellent -001 exchange -001 exchanged -001 exists -001 explore -001 export -001 external -001 ez -001 f- -001 fag -001 famous -001 favor -001 feeds -001 filesystem -001 finalize -001 finalized -001 find -001 finish -001 finished- -001 fixes -001 fj -001 flavored -001 fm -001 follow -001 forged -001 form -001 formal -001 forth -001 framed -001 free -001 fun -001 functioning -001 fv -001 gained -001 game -001 games -001 gateway -001 gateways -001 gauntlet -001 gb -001 gitbook -001 glossary -001 goes -001 grammar -001 grant -001 grid -001 ground -001 guarantee -001 gui -001 guide -001 halting -001 happy -001 hash -001 having -001 header -001 hello -001 helped -001 hgy -001 hhvq -001 hierarchy -001 higher-level -001 hooks -001 hopefully -001 however -001 http -001 hurt -001 i-iii -001 i'm -001 i've -001 ignored -001 ii -001 iii -001 implementation-specific -001 import -001 important -001 improvements -001 incorporate -001 indentation -001 initial -001 innovative -001 integrity -001 interest -001 intermittent -001 internal -001 its -001 iv -001 jqg -001 jspjjqjrvbvgv -001 jssfbaz -001 jttfypkdbhtzj -001 jv -001 jvvp -001 kcw -001 kdfyax -001 kf -001 kfqcwsz -001 kgtzp -001 khmdzzzrwk -001 knowledge -001 known -001 kys -001 lack -001 language -001 larger -001 latency -001 later -001 layer -001 let's -001 level -001 lexer -001 likely -001 limit -001 link -001 links -001 listening -001 local -001 locally -001 lock -001 logger -001 logging -001 look -001 mail -001 main -001 maintenance -001 major -001 makes -001 making -001 mapping -001 maritime -001 maturity -001 mechanisms -001 mention -001 mentioned -001 mesh -001 messenger -001 metal -001 mf -001 microblogging -001 mike -001 minimalism -001 minimize -001 mjsstchyp -001 model -001 monolithic -001 much -001 mule -001 multiple -001 must -001 my -001 names -001 namespacing -001 natural -001 ndatge -001 needed -001 neighboring -001 nested -001 networks -001 never -001 newsgroup -001 nh -001 nice-to-haves -001 nntp -001 node -001 nodejs -001 non-goals -001 nor -001 npwz -001 nqs -001 number -001 off -001 off-grid -001 offer -001 offers -001 offline-first -001 offline-only -001 offloaded -001 ok -001 old -001 online -001 operation -001 opinion -001 opposed -001 option -001 options -001 outreach -001 outstanding -001 own -001 package -001 packaged -001 pages -001 pair -001 pairs -001 parse -001 parsed -001 partial -001 parties -001 pb -001 pcta -001 peer -001 peer's -001 peers-of-peers -001 performance -001 performantly -001 persistence -001 phases -001 philosophy -001 photo -001 pieces -001 pigeons -001 platform -001 plugins -001 point -001 point-of-interest -001 poor -001 portable -001 ported -001 postal -001 power -001 powered -001 pr -001 practice -001 prefer -001 prevent -001 prior -001 product -001 production -001 production-grade -001 proliferate -001 promote -001 promotes -001 proof -001 proof-of-concept -001 prototype -001 prtocol -001 pseudonymity -001 pseudonyms -001 published -001 pubs -001 pztgkwv -001 qdfnsaf -001 qy -001 racing -001 ram -001 read -001 readers -001 ready -001 reasons -001 reboot -001 recent -001 recipient -001 redundant -001 redundantly -001 reference -001 referenced -001 regain -001 regardless -001 remote -001 rename -001 replicate -001 reply -001 report -001 reported -001 require -001 research -001 resistance -001 resource -001 rh -001 role -001 root -001 rp -001 rqtme -001 rsf -001 running -001 rust -001 rx -001 safe -001 say -001 schemas -001 scheme -001 sd -001 secret -001 section -001 seen -001 sense -001 sensor -001 serial -001 seriously -001 serves -001 services -001 sessions -001 set -001 set- -001 should -001 showing -001 shown -001 shutdown -001 similar -001 simpler -001 simplified -001 simulated -001 since -001 singletons -001 site -001 sj -001 slow -001 small -001 smaller -001 sneakernet-only -001 software -001 solar -001 solicited -001 soliciting -001 some -001 something -001 ssh -001 stability -001 stabilize -001 standard -001 starts -001 startup -001 status-quo -001 store -001 store-and-forward -001 structure -001 suitable -001 summary -001 super -001 sync -001 synchronization -001 synchronizing -001 system -001 takes -001 tamper-resistant -001 tampering -001 tapmdrna -001 targets -001 tckp -001 temperature -001 temporarily -001 term -001 terms -001 tf -001 thanks -001 them -001 themselves -001 then -001 these -001 throughput -001 tied -001 tool -001 tpm -001 tradeoffs -001 transfer -001 transmission -001 transport -001 trying -001 turn-based -001 tyson -001 unless -001 until -001 update -001 uptime -001 url -001 used -001 utilize -001 validate -001 vaporsoft -001 verify -001 version -001 vhcwx -001 viable -001 vj -001 vmrg -001 vtj -001 vvgrz -001 wa -001 wanted -001 watching -001 way -001 ways -001 wea -001 webcam -001 were -001 what's -001 where -001 whitespace -001 who -001 windows -001 wip -001 within -001 writing -001 ws -001 wwxyjg -001 wyr -001 xr -001 xs -001 xwchv -001 xyz -001 yame -001 ydqc -001 years -001 ym -001 ymr -001 yqbeg -001 yzady -001 zv -002 addressed -002 after -002 akkgw -002 also -002 altered -002 anonymity -002 app -002 areas -002 ascii-based -002 asked -002 authored -002 available -002 back -002 backed -002 bamboo -002 become -002 benefits -002 between -002 bin -002 borrows -002 bundle -002 bundles -002 cannot -002 cli -002 conceptual -002 concerns -002 concludes -002 contact -002 crockford -002 current -002 custom -002 cx -002 delay -002 development -002 devices -002 differs -002 draft -002 dx -002 each -002 easier -002 efcj -002 eg -002 encrypted -002 encryption -002 even -002 every -002 except -002 existing -002 fat -002 feature -002 feedback -002 few -002 footer -002 found -002 frequently -002 friend -002 get -002 gmv -002 goal -002 gw -002 hand -002 handbook -002 handled -002 hc -002 heabfd -002 high -002 human -002 idea -002 identities -002 if -002 instead -002 intended -002 internet -002 iot -002 it's -002 json -002 just -002 kdkk -002 ktngsdbkmsa -002 languages -002 learn -002 legacy -002 less -002 libraries -002 libs -002 limited -002 long -002 low -002 made -002 maintain -002 malicious -002 means -002 message's -002 messaging -002 might -002 mostly -002 nesting -002 new -002 nodes -002 numerous -002 once -002 other -002 out -002 overhead -002 party -002 people -002 plentiful -002 points -002 possibly -002 privacy -002 production-scale -002 re-write -002 readable -002 referencing -002 reliance -002 replicated -002 resistant -002 roadmap -002 ruby -002 sent -002 series -002 servers -002 signed -002 signing -002 snde -002 spec -002 sqlite -002 string -002 suited -002 tamper -002 te -002 text -002 tgfds -002 things -002 third -002 three -002 tolerant -002 too -002 traditional -002 transmitted -002 two -002 unlike -002 verification -002 vw -002 wasm -002 we've -002 weather -002 well -002 what -002 wish -002 without -002 wkg -002 world -002 would -002 your -003 always -003 any -003 assume -003 base -003 below -003 better -003 blob -003 carriage -003 case -003 concept -003 connectivity -003 created -003 currently -003 design -003 developer -003 easily -003 email -003 empty -003 entry -003 feed -003 files -003 here -003 ideas -003 identity -003 implementations -003 information -003 inspired -003 involved -003 list -003 logs -003 make -003 may -003 medium -003 need -003 only -003 over -003 overview -003 platforms -003 please -003 portability -003 project -003 protocols -003 rather -003 return -003 see -003 serve -003 sig -003 simple -003 social -003 storage -003 swarm -003 tcp -003 their -003 theoretically -003 though -003 time -003 udp -003 value -003 was -003 work -004 although -004 build -004 clients -004 configuration -004 considered -004 continue -004 depth -004 developers -004 etc -004 features -004 first -004 focus -004 format -004 gossip -004 into -004 least -004 like -004 log -004 most -004 network -004 networking -004 next -004 none -004 one -004 prev -004 problems -004 provide -004 questions -004 real-world -004 serialization -004 sha -004 sharing -004 so -004 ssb -004 they -004 us -004 which -004 working -005 allow -005 been -005 built -005 do -005 embedded -005 file -005 help -005 how -005 i -005 implementation -005 key -005 library -005 many -005 more -005 peer-to-peer -005 specification -005 start -005 up -005 uses -006 at -006 author -006 data -006 documentation -006 example -006 has -006 have -006 kind -006 lines -006 lipmaa -006 on -006 scuttlebutt -006 secure -006 signature -006 using -006 when -006 written -006 you -007 application -007 complete -007 ed -007 possible -007 such -007 via -008 all -008 cases -008 line -008 than -008 there -009 sneakernet -009 systems -010 from -010 with -011 an -011 applications -011 as -011 peers -011 support -012 it -012 we -013 can -014 not -014 or -015 message -015 messages -015 use -016 no -017 client -018 phase -018 will -023 by -024 are -024 for -024 that -025 this -027 in -028 pigeon -028 protocol -029 be -041 and -048 is -050 of -055 to -072 the -078 a \ No newline at end of file diff --git a/lipmaa.png b/lipmaa.png new file mode 100644 index 0000000..be6a655 Binary files /dev/null and b/lipmaa.png differ diff --git a/lipmaa.svg b/lipmaa.svg deleted file mode 100644 index 73c14da..0000000 --- a/lipmaa.svg +++ /dev/null @@ -1,2 +0,0 @@ - -
1
1
2
2
3
3
5
5
4
4
6
6
7
7
8
8
9
9
10
10
11
11
12
12
13
13
14
14
15
15
16
16
18
18
17
17
19
19
20
20
21
21
22
22
23
23
24
24
25
25
26
26
27
27
28
28
29
29
31
31
30
30
32
32
33
33
34
34
35
35
36
36
37
37
38
38
39
39
40
40 \ No newline at end of file diff --git a/DEV_DOCS.md b/message_format.md similarity index 55% rename from DEV_DOCS.md rename to message_format.md index 212deef..6ea0fbc 100644 --- a/DEV_DOCS.md +++ b/message_format.md @@ -1,9 +1,8 @@ -# It Starts With a Message... +# Exploring the Pigeon Message Format -The most important protocol concept is that of the "message". -In their most simple form, Pigeon protocol messages are just ASCII text documents. They are human readable and can even be created by hand in a text editor, though most clients will provide better means of authoring messages. +In the test that follows, we will explore a pigeon message line-by-line. -Below is an example of such a message: +The example message is shown in its entirety below: ``` author @MF312A76JV8S1XWCHV1XR6ANRDMPAT2G5K8PZTGKWV354PR82CD0.ed25519 @@ -19,9 +18,7 @@ weather_reported_by:@0DC253VW8RP4KGTZP8K5G2TAPMDRNA6RX1VHCWX1S8VJ67A213FM.ed2551 signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08.sig.ed25519 ``` -Let's explore each line of a message. - -### Line 1: `author` +### Line 1: `Author` EXAMPLE: @@ -39,7 +36,7 @@ The steps to generate a valid identity are: 2. Add an `@` symbol to the beginning of the string from step 1. 3. Add a `.ed25519` string to the end of the string from step 2. -### Line 2: `kind` +### Line 2: `Kind` EXAMPLE: @@ -59,7 +56,7 @@ It must meet the following criteria: * underscores (`_`) * Symbols used for multihashes, such as `@`, `&` and `%` (covered later). -### Line 3: `prev` +### Line 3: `Prev` EXAMPLE: @@ -81,7 +78,7 @@ Message multihashes are calculated as follows: 2. The next 52 characters are a [Crockford base 32](https://www.crockford.com/base32.html) SHA512 hash of the previous message's content. 3. The message multihash ends in `.sha512`. -### Line 4: `depth` +### Line 4: `Depth` EXAMPLE: @@ -93,21 +90,15 @@ Pigeon messages exist in a linear sequence which only moves forward and never "f Every message has a `depth` field to indicate its "place in line". Because every message has an ever-increasing integer that never duplicates, every message in a Pigeon feed will have a unique hash. This is true even if messages have identical body content. -### Line 5: `lipmaa` +### Line 5: `Lipmaa` **THIS FIELD WAS WRITTEN INCORRECTLY. THIS WILL CHANGE SOON. YOU CAN SAFELY MOVE TO THE NEXT SECTION OF THE DOCS** This concept was borrowed from the [Bamboo protocol](https://github.com/AljoschaMeyer/bamboo#links-and-entry-verification) and [Helger Lipmaa's thesis](https://kodu.ut.ee/~lipmaa/papers/thesis/thesis.pdf). -EXAMPLE: +The `lipmaa` field (often called a "Lipmaa Link") is a special kind of `prev` field that allows partial verification of feeds. This field makes it possible to verify a single message (or subset of messages) without downloading the entire chain of messages. -``` -lipmaa 2 -``` - -The `lipmaa` field (often called a "Lipmaa Link") is a special kind of `depth` field that allows partial verification of feeds. This field makes it possible to verify a single message (or subset of messages) without downloading the entire chain of messages. - -![](lipmaa.svg) +![](lipmaa.png) The `lipmaa` field is calculated as follows: @@ -140,41 +131,82 @@ def lipmaa(n) end ``` -### Line 6: Empty carriage return (body start) -### Lines 7: Entry containing a string -### Lines 8: Entry referencing a blob -### Lines 9: Entry referencing a peer's identity -### Lines 10: Empty Carriage Return (footer start) +### Line 6: Body Start (Empty Line) + +Once all headers are added, a client must place an empty line (`\n`) after the header. +The empty line signifies the start of the message body. + +Some notes about body entries: + + * The body of a message starts and ends with an empty line (`\n`). + * Every body entry is a key value pair. Keys and values are separated by a `:` character (no spaces). + * A key must be 1-90 characters in length + * A key cannot contain whitespace or control characters + * A key may contain any of the following characters: + * alphanumeric characters (a-z, A-Z, 0-9) + * dashes (`-`) + * underscores (`_`) + * Symbols used for multihashes, such as `@`, `&` and `%` (covered later). + * A value may be a: + * A string (128 characters or less) + * A multihash referencing an identity (`@`), a message (`%`) or a blob (`&`). + +### Lines 7: Entry Containing a String + +EXAMPLE: + +``` +temperature:"22.0C" +``` + +Body entries are defined by user and contain key/value pairs of application-specific data. +When a key/value pair represents something other than an identity, blob or message ID, a string is used. +Strings can be used for any type of data that does not fit into the other three categories. +Strings must be less than or equal to 128 characters in length. +The example above is the most simple kind of body entry. It specifies an arbitrary string representing the current temperature. + +### Lines 8: Entry Referencing a Blob + +EXAMPLE: + +``` +webcam_photo:&FV0FJ0YZADY7C5JTTFYPKDBHTZJ5JVVP5TCKP0605WWXYJG4VMRG.sha256 +``` + +Applications may attach files to messages in the form of blobs. Blobs are referenced using a blob multihash. + + * Starts with a `&` character. + * Ends with `.sha256` + * Contains exactly 52 characters between the `&` and `.sha256` parts. This is a SHA256 hash of the blob's content, represented in Crockford Base 32 encoding. + +A blob is referenced in a message's key or value. A client will include a blob's content in a "bundle" (explained later). + +### Lines 9: Entry Referencing a Peer's Identity + +EXAMPLE: + +``` +weather_reported_by:@0DC253VW8RP4KGTZP8K5G2TAPMDRNA6RX1VHCWX1S8VJ67A213FM.ed25519 +``` + +A message may reference other identities (or its own identity) by using an identity sigil either in the key or value portion of the entry. +This is analogous to "social tagging" seen in many social networks. + +### Lines 10: Empty Carriage Return (Footer Start) + +The last part of a message is the footer. Like a message body, a message footer starts and ends with an empty line. +The footer is essential for ensuring the tamper resistant properties of a Pigeon message. + ### Lines 11: Signature Line -### Lines 12: Empty Carriage Return (message end) -# Sharing Messages and Blobs via Bundles -# Glossary of Terms -**This list is out of date.** Numerous changes and problems were addressed in the implementation of a client. We will update this list in May of 2020. +EXAMPLE: - * Header - * Blob - * Crockford Base32 - * NONE - * String - * Signature - * Value - * Message - * Blob Hash - * Key - * Pair - * Footer - * Bundle - * Kind - * Message Signature - * Feed - * Identity +``` +signature JSPJJQJRVBVGV52K2058AR2KFQCWSZ8M8W6Q6PB93R2T3SJ031AYX1X74KCW06HHVQ9Y6NDATGE6NH3W59QY35M58YDQC5WEA1ASW08.sig.ed25519 +``` - -# Running a CLI Client - -Pigeon currently has one CLI available. It is written in Ruby. Documentation can be found [here](https://tildegit.org/PigeonProtocolConsortium/pigeon_ruby) - -# Up Next - -This concludes the developer documentation. Please email us with questions. To learn more, continue to the [idea bin](IDEAS.md). +A signature starts with the word `signature` followed by a space. +After that, the body (including the trailing `\n`) is signed using the author's ED25519 key. +The signature is encoded with Crockford base 32. +The signature ends with `.sig.ed25519`. +An empty carraige return is added after the signature line. diff --git a/ROADMAP.md b/roadmap.md similarity index 100% rename from ROADMAP.md rename to roadmap.md