# rss_item

guid: https://drewdevault.com/2020/06/21/BARE-message-encoding.html
title: Introducing the BARE message encoding
url: https://drewdevault.com/2020/06/21/BARE-message-encoding.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I like stateless tokens. We started with state<em>ful</em> tokens: where a generated
string acts as a unique identifier for a resource, and the resource itself is
looked up separately. For example, your sr.ht OAuth token is a stateful token:
we just generate a random number and hand it to you, something like
&ldquo;a97c4aeeec705f81539aa&rdquo;. To find the information associated with this token, we
query the database — our local <em>state</em> — to find it.</p>
<a href="#announcement">
  Click here to skip the context and read the actual announcement -&gt;
</a>
<p>But, increasingly, we&rsquo;ve been using stateless tokens, which are a bloody good
idea. The idea is that, instead of using random numbers, you encode the actual
state you need into the token. For example, your sr.ht login session cookie is a
JSON blob which is encrypted and base64 encoded. Rather than associating your
session with a record in the database, we just decrypt the cookie when your
browser sends it to us, and the session information is right there. This
improves performance and simplicity in a single stroke, which is a huge win in
my book.</p>
<p>There is one big problem, though: stateless tokens tend to be a lot larger than
their stateful counterparts. For a stateful token, we just need to generate
enough random numbers to be both unique and unpredictable, and then store the
rest of the data elsewhere. Not so for a stateless token, whose length is a
function of the amount of state which has been sequestered into it. Here&rsquo;s an
example: the cursor fields on the new GraphQL APIs are stateless. This is one of
them:</p>
<pre><code>gAAAAABe7-ysKcvmyavwKIT9k1uVLx_GXI6OunjFIHa3OJmK3eBC9NT6507PBr1WbuGtjlZSTYLYvicH2EvJXI1eAejR4kuNExpwoQsogkE9Ua6JhN10KKYzF9kJKW0hA_-737NurotB
</code></pre>
<p>A whopping 141 characters long! It&rsquo;s hardly as convenient to lug this monster
around. Most of the time it&rsquo;ll be programs doing the carrying, but it&rsquo;s still
annoying when you&rsquo;re messing with the API and debugging your programs. This
isn&rsquo;t an isolated example, either: these stateless tokens tend to be large
throughout sr.ht.</p>
<p>In general, JSON messages are pretty bulky. They represent everything as text,
which can be 2x as inefficient for certain kinds of data right off the bat.
They&rsquo;re also self-describing: the schema of the message is encoded into the
message itself; that is, the names of fields, hierarchy of objects, and data
types.</p>
<p>There are many alternatives that attempt to address this problem, and I
considered many of them. Here were a selected few of my conclusions:</p>
<ul>
<li><a href="https://developers.google.com/protocol-buffers/">protobuf</a>: too
complicated and too fragile, and I&rsquo;ve never been fond of the generated code
for protobufs in any language. Writing a third-party protobuf implementation
would be a gargantuan task, and there&rsquo;s no standard. RPC support is also
undesirable for this use-case.</li>
<li><a href="https://capnproto.org/">Cap&rsquo;n Proto</a>: fixed width, alignment, and so on
— good for performance, bad for message size. Too complex. RPC support
is also undesirable for this use-case. I also passionately hate C++ and I
cannot in good faith consider something which makes it their primary target.</li>
<li><a href="http://bsonspec.org/">BSON</a>: MonogoDB implementation details have leaked into
the specification, and it&rsquo;s extensible in the worst way. I appreciate that
JSON is a closed spec and no one is making vendor extensions for it —
and, similarly, a diverse extension ecosystem is not something I want to see
for this technology. Additionally, encoding schema into the message is wasting
space.</li>
<li><a href="https://msgpack.org/">MessagePack</a>: ruled out for similar reasons: too much
extensibility, and the schema is encoded into the message, wasting space.</li>
<li><a href="https://cbor.io/">CBOR</a>: ruled out for similar reasons: too much
extensibility, and the schema is encoded into the message. Has the advantage
of a specification, but the disadvantage of that spec being 54 pages long.</li>
</ul>
<p>There were others, but hopefully this should give you an idea of what I was
thinking about when evaluating my options.</p>
<p>There doesn&rsquo;t seem to be anything which meets my criteria just right:</p>
<ul>
<li>Optimized for small messages</li>
<li>Standardized</li>
<li>Easy to implement</li>
<li>Universal — little to no support for extensions</li>
<li>Simple — no extra junk that isn&rsquo;t contributing to the core mission</li>
</ul>
<p>The solution is evident.</p>
<p><a href="https://xkcd.com/927"><img src="https://imgs.xkcd.com/comics/standards.png" alt="xkcd comic 927, &ldquo;Standards&rdquo;"></a></p>
<p><a id="announcement"></a></p>
<h2 id="bare-binary-application-record-encoding">BARE: Binary Application Record Encoding</h2>
<p><a href="https://baremessages.org">BARE</a> meets all of the criteria:</p>
<ul>
<li><strong>Optimized for small messages</strong>: messages are binary, not self-describing,
and have no alignment or padding.</li>
<li><strong>Standardized &amp; simple</strong>: the specification is just over 1,000 words —
shorter than this blog post.</li>
<li><strong>Easy to implement</strong>: the first implementation (for Go) was done in a single
weekend (this weekend, in fact).</li>
<li><strong>Universal</strong>: there is room for user extensibility, but it&rsquo;s done in a manner
which does not require expanding the implementation nor making messages which
are incompatible with other implementations.</li>
</ul>
<p>Stateless tokens aren&rsquo;t the only messages that I&rsquo;ve wanted a simple binary
encoding for. On many occasions I&rsquo;ve evaluated and re-evaluated the same set of
existing solutions, and found none of them quite right. I hope that BARE will
help me solve many of these problems in the future, and I hope you find it
useful, too!</p>
<p>The cursor token I shared earlier in the article looks like this when encoded
with BARE:</p>
<pre><code>gAAAAABe7_K9PeskT6xtLDh_a3JGQa_DV5bkXzKm81gCYqNRV4FLJlVvG3puusCGAwQUrKFLO-4LJc39GBFPZomJhkyqrowsUw==
</code></pre>
<p>100 characters (41 fewer than JSON), which happens to be the minimum size of a
padded <a href="https://github.com/fernet/spec/">Fernet</a> message. If we compare only the
cleartext:</p>
<pre><code>JSON: eyJjb3VudCI6MjUsIm5leHQiOiIxMjM0NSIsInNlYXJjaCI6bnVsbH0=
BARE: EAUxMjM0NQA=
</code></pre>
<p>Much improved!</p>
<p>BARE also has an optional schema language for defining your message structure.
Here&rsquo;s a sample:</p>
<pre><code>type PublicKey data&lt;128&gt;
type Time string # ISO 8601

enum Department {
  ACCOUNTING
  ADMINISTRATION
  CUSTOMER_SERVICE
  DEVELOPMENT

  # Reserved for the CEO
  JSMITH = 99
}

type Customer {
  name: string
  email: string
  address: Address
  orders: []{
    orderId: i64
    quantity: i32
  }
  metadata: map[string]data
}

type Employee {
  name: string
  email: string
  address: Address
  department: Department
  hireDate: Time
  publicKey: optional
  metadata: map[string]data
}

type Person (Customer | Employee)

type Address {
  address: [4]string
  city: string
  state: string
  country: string
}
</code></pre><p>You can feed this into a code generator and get types which can encode &amp; decode
these messages. But, you can also describe your schema just using your
language&rsquo;s existing type system, like this:</p>
<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">type</span> <span class="nx">Coordinates</span> <span class="kd">struct</span> <span class="p">{</span>
    <span class="nx">X</span> <span class="kt">uint</span>  <span class="c1">// uint
</span><span class="c1"></span>    <span class="nx">Y</span> <span class="kt">uint</span>  <span class="c1">// uint
</span><span class="c1"></span>    <span class="nx">Z</span> <span class="kt">uint</span>  <span class="c1">// uint
</span><span class="c1"></span>    <span class="nx">Q</span> <span class="o">*</span><span class="kt">uint</span> <span class="c1">// optional&lt;uint&gt;
</span><span class="c1"></span><span class="p">}</span>

<span class="kd">func</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
    <span class="kd">var</span> <span class="nx">coords</span> <span class="nx">Coordinates</span>
    <span class="nx">payload</span> <span class="o">:=</span> <span class="p">[]</span><span class="kt">byte</span><span class="p">{</span><span class="mh">0x01</span><span class="p">,</span> <span class="mh">0x02</span><span class="p">,</span> <span class="mh">0x03</span><span class="p">,</span> <span class="mh">0x01</span><span class="p">,</span> <span class="mh">0x04</span><span class="p">}</span>
    <span class="nx">err</span> <span class="o">:=</span> <span class="nx">bare</span><span class="p">.</span><span class="nf">Unmarshal</span><span class="p">(</span><span class="nx">payload</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">coords</span><span class="p">)</span>
    <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
        <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
    <span class="p">}</span>
    <span class="nx">fmt</span><span class="p">.</span><span class="nf">Printf</span><span class="p">(</span><span class="s">&#34;coords: %d, %d, %d (%d)\n&#34;</span><span class="p">,</span> <span class="cm">/* coords: 1, 2, 3 (4) */</span>
        <span class="nx">coords</span><span class="p">.</span><span class="nx">X</span><span class="p">,</span> <span class="nx">coords</span><span class="p">.</span><span class="nx">Y</span><span class="p">,</span> <span class="nx">coords</span><span class="p">.</span><span class="nx">Z</span><span class="p">,</span> <span class="o">*</span><span class="nx">coords</span><span class="p">.</span><span class="nx">Q</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div><p>Bonus: you can get the schema language definition for this struct with
<code>schema.SchemaFor(coords)</code>.</p>
<h2 id="bare-is-under-development">BARE is under development</h2>
<p>There are some possible changes that could come to BARE before finalizing the
specification. Here are some questions I&rsquo;m thinking about:</p>
<ul>
<li>Should the schema language include support for arbitrary annotations to
inform code generators? I&rsquo;m inclined to think &ldquo;no&rdquo;, but if you use BARE and
find yourself wishing for this, tell me about it.</li>
<li>Should BARE have first-class support for bitfield enums?</li>
<li>Should maps be ordered?</li>
</ul>
<p><a href="mailto:~sircmpwn/public-inbox@lists.sr.ht">Feedback welcome</a>!</p>
<p><strong>Errata</strong></p>
<ul>
<li>This article was originally based on an older version of the draft
specification, and was updated accordingly.</li>
</ul>


guid: https://drewdevault.com/2020/06/26/Vendor-purpose-OS.html
title: General-purpose OS, special-purpose OS, and now: vendor-purpose OS
url: https://drewdevault.com/2020/06/26/Vendor-purpose-OS.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>There have, historically, been two kinds of operating systems: general-purpose,
and special-purpose. These roles are defined by the function they serve for the
user. Examples of general-purpose operating systems include Unix (Linux, BSD,
etc), Solaris, Haiku, Plan 9, and so on. These are well-suited to general
computing tasks, and are optimized to solve the most problems possible, perhaps
at the expense of those in some niche domains. Special-purpose operating systems
serve those niche domains, and are less suitable for general computing. Examples
of these include FreeRTOS, Rockbox, Genode, and so on.</p>
<p>These terms distinguish operating systems by the problems they solve for the
user. However, a disturbing trend is emerging in which the user is not the party
whose problems are being solved, and perhaps this calls for a new term. I
propose &ldquo;vendor-purpose operating system&rdquo;.</p>
<p>I would use this term to describe Windows, macOS, Android, and iOS, and perhaps
some others besides. Arguably, the first two used to be general purpose
operating systems, and the latter two were once special-purpose operating
systems.  Increasingly, these operating systems are making design decisions
which benefit the vendor <em>at the expense</em> of the user. For example: Windows has
ads and excessive spyware, prevents you from making a local login without a
Microsoft account, and aggressively pushes you to switch to Edge from other web
browsers, as well as many other examples besides.</p>
<p>Apple is more subtle from the end-user&rsquo;s perspective. They eschew standards to
build walled gardens, opting for Metal rather than Vulkan, for example. They use
cryptographic signatures to enforce a racket against developers who just want to
ship their programs. They bully vendors in the app store into adding things like
microtransactions to increase their revenue. They&rsquo;ve also long been making
similar moves in their hardware design, adding anti-features which are
explicitly designed to increase their profit — adding false costs which
are ultimately passed onto the consumer.</p>
<p>All of these decisions are making the OS worse for users in order to provide
more value to the vendor. The operating system is becoming <em>less</em> suited to its
general-purpose tasks, as the vendor-purpose anti-features deliberately get in
the way. They also become less suited at special-purpose tasks for the same
reasons. These changes <em>are</em> making improvements for one purpose: the vendor&rsquo;s
purpose. Therefore, I am going to start refering to these operating systems as
&ldquo;vendor purpose&rdquo;, generally alongside a curse and a raising of the middle
finger.</p>


guid: https://drewdevault.com/2020/07/14/March-2nd-1943.html
title: March 2nd, 1943
url: https://drewdevault.com/2020/07/14/March-2nd-1943.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>It&rsquo;s March 2nd, 1943. The user asks your software to schedule a meeting with
Acmecorp at &ldquo;9 AM on the first Monday of next month&rdquo;.</p>
<pre>
<code>
[6:17:45] homura ~ $ cal -3 2 March 1943
    February 1943          March 1943            April 1943
Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa  Su Mo Tu We Th Fr Sa
    1  2  3  4  5  6      1 <span style="background: black; color: white"> 2</span> 3  4  5  6               1  2  3
 7  8  9 10 11 12 13   7  8  9 10 11 12 13   4 <span style="background: #666; color: white"> 5</span>  6  7  8  9 10
14 15 16 17 18 19 20  14 15 16 17 18 19 20  11 12 13 14 15 16 17
21 22 23 24 25 26 27  21 22 23 24 25 26 27  18 19 20 21 22 23 24
28                    28 29 30 31           25 26 27 28 29 30
</code>
</pre>
<p>Right now, California is on Pacific Standard Time (PST) and Arizona is on
Mountain Standard Time (MST). On March 8th, California will transition to
Pacific Daylight Time (PDT), one hour ahead. Arizona does not observe DST, so
they&rsquo;ll stay behind.</p>
<p>At least until April 1st — when the governor will sign an emergency order
moving the state to MDT, effective immediately.</p>
<p>Back on March 2nd, you send an email to each participant telling them about the
meeting. One of them has their locale set to en_GB, so some of the participants
need to be sent &ldquo;04/05/43&rdquo; and some &ldquo;05/04/43&rdquo;.</p>
<p>A moment later, the user asks you to tell it the number of hours betweeen now
and the meeting they just scheduled. The subject of the meeting is purchasing
fuel for a machine that the user is now filling with enough fuel to last until
then.</p>
<p>On the day of the meeting, the user drives to the Navajo reservation to conduct
some unrelated business, and has to attend the meeting by phone. The reservation
has been on daylight savings time since March 8th, by the way, they never stayed
behind with the rest of Arizona. The user expects the software to warn them 1
hour prior to the meeting start. The border of the reservation is defined by a
river, which is slowly moving East.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p>
<p><a href="https://mm.icann.org/pipermail/tz-announce/">The changelog for the IANA zoneinfo database</a>
is great, by the way, you should read it.
<a href="https://mm.icann.org/mailman/listinfo/tz-announce">Or subscribe</a> to get it
periodically<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> material delivered to your inbox!</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Okay, that last bit isn&rsquo;t true. But imagine if it was! <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>But with what period? 😉 <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/07/15/Status-update-July-2020.html
title: Status update, July 2020
url: https://drewdevault.com/2020/07/15/Status-update-July-2020.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Hello again! Another month of FOSS development behind us, and we&rsquo;re back again
to share the results. I took a week off at the end of June, so my progress this
month is somewhat less than usual. Regardless, I have some updates for you,
mainly in the domain of SourceHut work.</p>
<p>But before we get to that, let&rsquo;s go over this month&rsquo;s small victories. One was
the invention of the <a href="https://baremessages.org">BARE message format</a>, which I
wrote <a href="https://drewdevault.com/2020/06/21/BARE-message-encoding.html">a blog post about</a> if you want to learn more. Since that
article, five new implementations have appeared from various authors: Rust,
Python, JavaScript, D, and Zig.</p>
<p>I also wrote a couple of not-blogposts for this site (drewdevault.com),
including a page <a href="https://drewdevault.com/dynlib">dispelling misconceptions about static linking</a>,
and a page (that I hope you&rsquo;ll contribute to!) with <a href="https://drewdevault.com/editing">videos of people editing
text</a>. Just dropping a link here in case you missed them; they didn&rsquo;t
appear in RSS and aren&rsquo;t blog posts. To help find random stuff like that on this
site, I&rsquo;ve also established a <a href="https://drewdevault.com/misc">misc page</a>.</p>
<p>Okay, on to SourceHut. Perhaps the most exciting development is the addition of
<a href="https://sourcehut.org/blog/2020-07-14-setting-up-ci-for-mailing-lists/">continuous integration to the mailing lists</a>. I&rsquo;ve been working
towards this for some time now, and it&rsquo;s the first of many features which are
now possible thanks to the addition of the project hub. I intend to complete
some follow-up work improving the CI feature further still in the coming weeks.
I&rsquo;m also planning an upgrade for the hardware that runs hg.sr.ht during the same
timeframe.</p>
<p>That&rsquo;s all the news I have for now, somewhat less than usual. Some time off was
much-needed, though. Thanks for your continued support, and I hope you continue
to enjoy using my software!</p>
<details>
<summary>...</summary>
<pre>
$ cat main.$ext
use io;
use strings;
use sys;
<p>export fn main void =
{
for (let i = 0; sys::envp[i] != null; i += 1) {
let s = strings::from_c(sys::envp[i]);
io::println(s);
};
};
$ $redacted run main.$ext
error: main.$ext:8:41: incorrect type (&amp;char) for parameter 1 (&amp;char)
let s = strings::from_c(sys::envp[i]);
^&mdash; here
$ vim main.$ext
$ cat main.$ext
use io;
use strings;
use sys;</p>
<p>export fn main void =
{
for (let i = 0; sys::envp[i] != null; i += 1) {
let s = strings::from_c(sys::envp[i]);
io::println(s);
free(s);
};
};
$ $redacted run main.$ext
DISPLAY=:0
EDITOR=vim</p>
<h1 id="heading">&hellip;</h1>
<p></pre></p>
</details>


guid: https://drewdevault.com/2020/07/27/Anti-AGPL-propaganda.html
title: The falsehoods of anti-AGPL propaganda
url: https://drewdevault.com/2020/07/27/Anti-AGPL-propaganda.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Google is well-known for <a href="https://opensource.google/docs/using/agpl-policy/">forbidding the use of</a> software
using the <a href="https://www.gnu.org/licenses/agpl-3.0.en.html">GNU Affero General Public License</a>, commonly known as &ldquo;AGPL&rdquo;.
Google is also well-known for being the subject of cargo-culting by fad
startups. Unfortunately, this means that they are susceptible to what is
ultimately anti-AGPL propaganda from Google, with little to no basis in fact.</p>
<p><em>Obligatory: I&rsquo;m not a lawyer; this is for informational purposes only.</em></p>
<p>In truth, the terms of the AGPL are pretty easy to comply with. The basic
obligations of the AGPL which set it apart from other licenses are as follows:</p>
<ul>
<li>Any derivative works of AGPL-licensed software must also use the AGPL.</li>
<li>Any users of such software are entitled to the source code under the terms of
the AGPL, including users accessing it over the network such as with their web
browser or via an API or internet protocol.</li>
</ul>
<p>If you&rsquo;re using AGPL-licensed software like a database engine or <a href="https://sr.ht/~sircmpwn/sourcehut/">my own
AGPL-licensed works</a>, and you haven&rsquo;t made any changes to the source
code, all you have to do is provide a link to the upstream source code
somewhere, and if users ask for it, direct them there. If you <em>have</em> modified
the software, you simply have to publish your modifications. The easiest way to
do this is to send it as a patch upstream, but you could use something as simple
as providing a tarball to your users.</p>
<p>The nuances are detailed and cover many edge cases to prevent abuse. But in
general, just publish your modifications under the same AGPL terms and you&rsquo;ll
be good to go. The license is usually present in the source code as a <code>COPYING</code>
or <code>LICENSE</code> file, so if you just tar up your modified source code and drop a
link on your website, that&rsquo;s good enough. If you want to go the extra mile and
express your gratitude to the original software developers, consider submitting
your changes for upstream inclusion. Generally, the feedback you&rsquo;ll receive will
help to make your changes better for your use-case, too; and submitting your
work upstream will prevent your copy from diverging from upstream.</p>
<p>That&rsquo;s pretty easy, right? I&rsquo;m positive that your business has to deal with much
more onerous contracts than the AGPL. Then why does Google make a fuss about it?</p>
<p><a href="https://opensource.google/docs/using/agpl-policy/">The Google page about the AGPL</a> details inaccurate (but
common<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>) misconceptions about the obligations of the AGPL that don&rsquo;t follow
from the text. Google states that if, for example, Google Maps used PostGIS as
its data store, and PostGIS used the AGPL, Google would be required to release
the Google Maps code. This is not true. They would be required to release <em>their
PostGIS patches</em> in this situation. AGPL does not extend the GPL in that it
makes the Internet count as a form of linking which creates a derivative work,
as Google implies, but rather that it makes anyone who uses the software via
the Internet entitled to its source code. It does not update the &ldquo;what counts
as a &lsquo;derivative work&rsquo;&rdquo; algorithm, so to speak — it updates the &ldquo;what
counts as &lsquo;distributing&rsquo; the software&rdquo; algorithm.</p>
<p>The reason they spread these misconceptions is straightforward: they want to
discourage people from using the AGPL, because they cannot productize such
software effectively. Google wants to be able to incorporate FOSS software into
their products and sell it to users without the obligation to release their
derivative works. Google is an Internet company, and they offer Internet
services. The original GPL doesn&rsquo;t threaten their scheme because their software
is accessed over the Internet, not distributed to end-users directly.</p>
<p>By discouraging the use of AGPL in the broader community, Google hopes to create
a larger set of free- and open-source software that they can take for their own
needs without any obligations to upstream. Ask yourself: why is documentation of
internal-facing decisions like what software licenses to use being published in
a public place? The answer is straightforward: to influence the public. This is
propaganda.</p>
<p>There&rsquo;s a bizarre idea that software companies which eschew the AGPL in favor of
something like MIT are doing so specifically because they want companies &ldquo;like
Google<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>&rdquo; to pay for their software, and they know that they have no chance if
they use AGPL. In truth, Google was never going to buy your software. If you
don&rsquo;t use the AGPL, they&rsquo;re just going to take your software and give nothing
back. If you do use the AGPL, they&rsquo;re just going to develop a solution in-house.
There&rsquo;s no outcome where Google pays you.</p>
<p>Don&rsquo;t be afraid to use the AGPL, and don&rsquo;t be afraid to use software which uses
the AGPL. The obligations are not especially onerous or difficult, despite what
Google would have you believe. The license isn&rsquo;t that long — read it and
see for yourself.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Likely common <em>because of this page</em>. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>By the way, there are no more than 10 companies world-wide which are &ldquo;like Google&rdquo; by any measure. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/08/01/pkg-go-dev-sucks.html
title: pkg.go.dev is more concerned with Google's interests than good engineering
url: https://drewdevault.com/2020/08/01/pkg-go-dev-sucks.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>pkg.go.dev sucks. It&rsquo;s certainly <em>prettier</em> than godoc.org, but under the
covers, it&rsquo;s a failure of engineering characteristic of the Google approach.</p>
<p>Go is a <em>pretty good</em> programming language. I have long held that this is not
attributable to Google&rsquo;s stewardship, but rather to a small number of language
designers and a clear line of influences which is drawn entirely from outside of
Google — mostly from Bell Labs. pkg.go.dev provides renewed support for my
argument: it has all the hallmarks of Google crapware and none of the
deliberate, good engineering work that went into Go&rsquo;s design.</p>
<p>It was apparent from the start that this is what it would be. pkg.go.dev was
launched as a closed-source product,
<a href="https://blog.golang.org/pkg.go.dev-2020">justified</a> by pointing out that
godoc.org is too complex to run on an intranet, and pkg.go.dev has the same
problem. There are many problems to take apart in this explanation: the
assumption that the only reason an open source platform is desirable is for
running it on your intranet; the unstated assumption that such complexity
is necessary or agreeable in the first place; and the
<a href="https://github.com/golang/go/issues/25443">systemic</a> <a href="https://github.com/golang/go/issues/30029">erosion</a>
of the existing (and simple!) tools which <em>could</em> have been used for this
purpose prior to this change. The attitude towards open source was only changed
following pkg.go.dev&rsquo;s harsh reception by the community.</p>
<p>But this attitude <em>did</em> change, and it is open-source now<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> <sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>, so let&rsquo;s give
them credit for that. The good intentions are spoilt by the fact that pkg.go.dev
fetches the list of modules from <a href="https://proxy.golang.org/">proxy.golang.org</a>:
a closed-source proxy through which all of your go module fetches are being
routed and tracked (oh, you didn&rsquo;t know? They never told you, after all).
Anyway, enough of the gross disregard for the values of open source and user
privacy; I <em>do</em> have some technical problems to talk about.</p>
<p>One concern comes from a blatant failure to comprehend the fundamentally
decentralized nature of git hosting. Thankfully, git.sr.ht is supported now<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>
— but only <em>the</em> git.sr.ht, i.e. the hosted instance, not the software.
pkg.go.dev hard-codes a list of centralized git hosting services, and completely
disregards the idea of git hosting as <em>software</em> rather than as a <em>platform</em>.
Any GitLab instance other than gitlab.com (such as
<a href="https://gitlab.freedesktop.org">gitlab.freedesktop.org</a> or
<a href="https://salsa.debian.org/public">salsa.debian.org</a>); any
<a href="https://gogs.io/">Gogs</a> or <a href="https://gitea.io/en-us/">Gitea</a> like
<a href="https://codeberg.org">Codeberg</a>; cgit instances like
<a href="https://git.kernel.org/">git.kernel.org</a>; none of these are going to work
unless every host is added and the list is kept up-to-date manually. Your
intranet instance of cgit? Not a chance.</p>
<p>They were also given an opportunity here to fix a long-standing problem with Go
package discovery, namely that it requires every downstream git repository host
has to (1) provide a web interface and (2) include <em>Go-specific</em> meta tags in
the HTML. The hubris to impose your <em>programming language</em>&rsquo;s needs onto a
language-agnostic version control system! I asked: they have no interest in the
better-engineered — but more worksome — approach of pursing a
language agnostic design.</p>
<p>The worldview of the developers is whack, the new site introduces dozens of
regressions, and all it really improves upon is the visual style — which
could trivially have been done to godoc.org. The goal is shipping a shiny new
product — not engineering a good solution. This is typical of Google&rsquo;s
engineering ethos in general. pkg.go.dev sucks, and is added the large (and
growing) body of evidence that Google is bad for Go.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Setting aside the fact that the production pkg.go.dev site is amended with closed-source patches. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>The GitHub comment explaining the change of heart included a link to a Google Groups discussion which requires you to log in with a Google account in order to <em>read</em>.<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> If you go the long way around and do some guesswork searching the archives yourself, you <a href="https://groups.google.com/d/msg/golang-dev/mfiPCtJ1BGU/ibeimu3WEgAJ">can find it</a> without logging in. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Commenting on Go patches also requires a Google account, by the way. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>But not hg.sr.ht! <a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/08/10/How-to-contribute-to-FOSS.html
title: I want to contribute to your project, how do I start?
url: https://drewdevault.com/2020/08/10/How-to-contribute-to-FOSS.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I get this question a lot! The answer is usually&hellip; don&rsquo;t. If you already know
what you want to do, then the question doesn&rsquo;t need to be asked.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> But, if you
don&rsquo;t already know what you want to do, then your time might be better spent
elsewhere!</p>
<p>The best contributors are always intrinsically motivated. Some contributors show
up every now and then who appreciate the value the project gives to them and
want to give something back. Their gratitude is definitely appreciated<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>, but
these kinds of contributions tend to require more effort from the maintainers,
and don&rsquo;t generally lead to recurring contributions. Projects you already like
are less likely to need help when compared to incomplete projects that you don&rsquo;t
already depend on — so this model leaves newer projects with fewer
contributors and encourages established projects to grow in complexity.</p>
<p>Instead, you should focus on scratching your own itches. Is there a bug which is
getting on your nerves? A conspicuously absent feature? Work on those!</p>
<p>If there&rsquo;s nothing specific that you want to work on, then you may be better off
finding something to do in a different project. Don&rsquo;t be afraid to work on any
free- and open-source codebase that you encounter — nearly all of them
will accept your patches. If something is bothering you about another project,
then go fix it! Someone has a cool idea and needs help realizing it? Get
involved! If we spread the contributions around, the FOSS ecosystem will
flourish and the benefits will come back around to our project, too.</p>
<p>So, if you want to contribute to open-source — as a whole — here are
my tips:</p>
<ul>
<li>Find problems which you are intrinsically motivated to work on.</li>
<li>Focus on developing skills to get up to speed on new codebases fast.</li>
<li>Don&rsquo;t be afraid to work on <em>any</em> project — new languages, tools,
libraries; learn enough of them and it&rsquo;ll only get easier to learn more.</li>
<li>When you file bug reports with a FOSS project, get into the habit of following
up with a patch which addresses the problem.</li>
<li>Get used to introducing yourself to maintainers and talking through the code;
it always pays to ask.</li>
</ul>
<p>If you want to work on a specific project, and you have a specific goal in mind:
perfect! If you don&rsquo;t have a specific goal in mind, try to come up with some.
And if you&rsquo;re still drawing a blank, consider another project.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Or perhaps the better question is &ldquo;where should I start with this goal?&rdquo; <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>For real, we don&rsquo;t hear &ldquo;thanks&rdquo; very often and expressions of gratitude are often our only reward for our work. We do appreciate it :) <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/08/13/Web-browsers-need-to-stop.html
title: Web browsers need to stop
url: https://drewdevault.com/2020/08/13/Web-browsers-need-to-stop.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Enough is enough.</p>
<p>The web and web browsers have become Lovecraftian horrors of an unprecedented
scale. They&rsquo;ve long since left &ldquo;scope creep&rdquo; territory and entered &ldquo;oh my god
please just stop&rdquo; territory, and are trucking on through to hitherto unexplored
degrees of <em>obscene</em> scope. And we <em>don&rsquo;t want</em> what they&rsquo;re selling. Google
pitches garbage like AMP<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> and pushing dubious half-assed specs like Web
Components. Mozilla just fired everyone relevant<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> to focus on crap no one
asked for like Pocket, and fad nonsense like a paid VPN service and <del>virtual
reality tech</del>.<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> <em>[2020-08-14: It has been pointed out that the VR team was
also fired.]</em></p>
<p>Microsoft gave up entirely. Mozilla just hammered the last few nails into their
casket.<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup> <del>Safari is a joke</del><sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup>. Google is all that&rsquo;s left, and they&rsquo;re
<em>not</em> a good steward of the open web. The browsers are drowning under their own
scope.  The web is dead.</p>
<p>I call for an immediate and indefinite suspension of the addition of new
developer-facing APIs to web browsers. Browser vendors need to start thinking
about <em>reducing</em> scope and <em>cutting</em> features. WebUSB, WebBluetooth, WebXR,
<del>WebDRM</del> <del>WebMPAA</del> <del>WebBootlicking</del> <del>replacing User-Agent with
Vendor-Agent cause let&rsquo;s be honest with ourselves at this point</del> &ldquo;Encrypted
Media Extensions&rdquo; — this crap all needs to go. At some point you need to
stop adding scope and start focusing on performance, efficiency, reliability,
and security<sup id="fnref:6"><a href="#fn:6" class="footnote-ref" role="doc-noteref">6</a></sup> at the scope you already have.</p>
<p>Enough is enough.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p><em>No one</em> wants AMP. Google knows it, you know it, I know it. If you&rsquo;re a Google engineer who is still working on AMP, you are a disgrace to your field. <a href="https://drewdevault.com/2020/05/05/We-are-complicit-in-our-employers-deeds.html">Take responsibility for the code you write</a>. This project needs to be dead and buried and the earth above salted, and it needs to happen yesterday. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p><del>No layoffs or pay cuts at the management level, of course!</del> It&rsquo;s not like they&rsquo;re <em>responsible</em> for these problems, it&rsquo;s not like anyone&rsquo;s fucking <em>responsible</em> for any of this, it&rsquo;s not like the very idea of <em>personal responsibility</em> has been forgotten by both executives and engineers, no sir! <em>[2020-08-14: It has been pointed out that some VPs were laid off. I also wish to clarify that the personal responsibility I find absent at the engineering level is more of a commentary on Google than Mozilla.]</em> <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Oh good, the <em>web</em> is exactly what VR needs! It&rsquo;s definitely <em>not</em> a huge time-sink requiring the highly skilled low-level engineering talent which Mozilla just finished <em>laying off</em>, or years of effort and millions of dollars just to realize that the new state of the art is still just an expensive and underwhelming product whose few end-user applications make half of their users motion sick. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>Next time they should aim for their executive&rsquo;s heads, maybe they&rsquo;ll jostle them around enough to get the two wires in each of their heads to make contact so that they&rsquo;re briefly capable of making basic decisions and not just collecting multi-million-dollar paychecks. <a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>2020-08-14: I haven&rsquo;t used Safari in over 10 years, so maybe it&rsquo;s not so bad. However, so long as it&rsquo;s single-platform and closed source, it&rsquo;s still a net negative on the ecosystem. <a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>The web <em>might</em> be one for four on these right now. <a href="#fnref:6" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/08/16/Status-update.html
title: Status update, August 2020
url: https://drewdevault.com/2020/08/16/Status-update.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Greetings! Today is another rainy day here in Philadelphia, which rather sours
my plans of walking over to the nearby cafe to order some breakfast to-go. But I
am tired, and if I&rsquo;m going to make it to the end of this blog post in one piece,
I&rsquo;m gonna need a coffee. brb.</p>
<p>Hey, that was actually pretty refreshing. It&rsquo;s just drizzling, and the rain is
nice and cool. Alright, here goes! What&rsquo;s new? I&rsquo;ll leave the Wayland news for
<a href="https://emersion.fr/blog">Simon Ser&rsquo;s blog</a> this month - he&rsquo;s been working on
some exciting stuff. The <a href="https://baremessages.org/">BARE encoding</a> announced
last month has received some great feedback and refinements, and there are now
six projects providing BARE support for their author&rsquo;s favorite programming
language<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. There have also been some improvements to the Go implementation
which should help with some SourceHut plans later on.</p>
<p>On the subject of SourceHut, I&rsquo;ve focused mainly on infrastructure improvements
this month. There is a new server installed for hg.sr.ht, which will also be
useful as a testbed for additional ops work planned for future expansion.
Additionally, the PostgreSQL backup system has been overhauled and made more
resilient, both to data loss and to outages. A lot of other robustness
improvements have been made fleet-wide in monitoring. I&rsquo;ll be working on more
user-facing features again next month, but in the meanwhile, contributors like
наб have sent many patches in which I&rsquo;ll cover in detail in the coming &ldquo;What&rsquo;s
cooking&rdquo; post for <a href="https://sourcehut.org/blog">sourcehut.org</a>.</p>
<p>Otherwise, I&rsquo;ve been taking it easy this month. I definitely haven&rsquo;t been
spending a lot of my time on a secret project, no sir. Thanks again for your
support! I&rsquo;ll see you next month.</p>
<details>
<summary>?</summary>
<pre>
use io;
use io_uring = linux::io_uring;
use linux;
use strings;
<p>export fn main void = {
let uring = match (io_uring::init(256u32, 0u32)) {
err: linux::error =&gt; {
io::println(&ldquo;io_uring::init error:&quot;);
io::println(linux::errstr(err));
return;
},
u: io_uring::io_uring =&gt; u,
};</p>
<pre><code>let buf: [8192]u8 = [0u8...];
let text: nullable *str = null;
let wait = 0u;
let offs = 0z;
let read: *io_uring::sqe = null: *io_uring::sqe,
    write: *io_uring::sqe = null: *io_uring::sqe;
let eof = false;

while (!eof) {
    read = io_uring::must_get_sqe(&amp;uring);
    io_uring::prep_read(read, linux::STDIN_FILENO,
        &amp;buf, len(buf): u32, offs);
    io_uring::sqe_set_user_data(read, &amp;read);
    wait += 1u;

    let ev = match (io_uring::submit_and_wait(&amp;uring, wait)) {
        err: linux::error =&gt; {
            io::println(&quot;io_uring::submit error:&quot;);
            io::println(linux::errstr(err));
            return;
        },
        ev: uint =&gt; ev,
    };

    wait -= ev;

    for (let i = 0; i &lt; ev; i += 1) {
        let cqe = match (io_uring::get_cqe(&amp;uring, 0u, 0u)) {
            err: linux::error =&gt; {
                io::println(&quot;io_uring::get_cqe error:&quot;);
                io::println(linux::errstr(err));
                return;
            },
            c: *io_uring::cqe =&gt; c,
        };

        if (io_uring::cqe_get_user_data(cqe) == &amp;read) {
            if (text != null) {
                free(text);
            };

            if (cqe.res == 0) {
                eof = true;
                break;
            };

            text = strings::must_decode_utf8(buf[0..cqe.res]);
            io_uring::cqe_seen(&amp;uring, cqe);

            write = io_uring::must_get_sqe(&amp;uring);
            io_uring::prep_write(write, linux::STDOUT_FILENO,
                text: *char, len(text): u32, 0);
            io_uring::sqe_set_user_data(write, &amp;write);
            wait += 1u;
            offs += cqe.res;
        } else if (io_uring::cqe_get_user_data(cqe) == &amp;write) {
            assert(cqe.res &gt; 0);
            io_uring::cqe_seen(&amp;uring, cqe);
        } else {
            assert(false, &quot;Unknown CQE user data&quot;);
        };
    };
};

io_uring::close(&amp;uring);
</code></pre>
<p>};
</pre></p>
<details>
<summary>hmm?</summary>
<p>I might note that I wrote this program to test my io_uring wrapper; it's not
representative of how normal programs will do I/O in the future.</p>
</details>
</details>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Or in some cases, the language the author is begrudgingly stuck with. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/08/17/Engineers-solve-problems.html
title: Software engineers solve problems
url: https://drewdevault.com/2020/08/17/Engineers-solve-problems.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Software engineers solve problems. A problem you may have encountered is, for
example, &ldquo;this function has a bug&rdquo;, and you&rsquo;re probably already more or less
comfortable solving these problems. Here are some other problems you might
encounter on the way:</p>
<ol>
<li>Actually, the bug ultimately comes from a third-party program</li>
<li>Hm, it uses a programming language I don&rsquo;t know</li>
<li>Oh, the bug is in that programming language&rsquo;s compiler</li>
<li>This subsystem of the compiler would have to be overhauled</li>
<li>And the problem is overlooked by the language specification</li>
</ol>
<p>I&rsquo;ve met many engineers who, when standing at the base of this mountain,
conclude that the summit is too far away and clearly not their responsibility,
and subsequently give up. But remember: as an engineer, your job is to apply
creativity to solving problems. Are these not themselves problems to which the
engineering process may be applied?</p>
<p>You can introduce yourself to the maintainers of the third-party program and
start working on a solution. You can study the programming language you don&rsquo;t
know, at least as much as is necessary to understand and correct the bug. You
can read the compiler&rsquo;s source code, and identify the subsystem which needs
overhauling, then introduce yourself to <em>those</em> maintainers and work on the
needed overhaul. The specification is probably managed by a working group, reach
out to them and have an erratta issued or a clarification added to the upcoming
revision.</p>
<p>The scope of fixing this bug is broader than you thought, but if you apply a
deliberate engineering process to each problem that you encounter, eventually
you will complete the solution. This process of recursively solving problems to
get at the one you want to solve is called &ldquo;<a href="http://catb.org/jargon/html/Y/yak-shaving.html">yak
shaving</a>&rdquo;, and it&rsquo;s a necessary
part of your workflow.</p>


guid: https://drewdevault.com/2020/08/24/Alice-in-Wonderland.html
title: Alice in Wonderland and the theft of the public domain
url: https://drewdevault.com/2020/08/24/Alice-in-Wonderland.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Disney&rsquo;s <em>Alice in Wonderland</em> is one of my favorite movies and an undisputed
classic. After its release in 1951, <em>Alice</em> holds a fond place in billions of
children&rsquo;s hearts, over almost four generations. And it has been stolen from
those generations, as part of the theft of one of these generations' greatest
treasures: the public domain.</p>
<p><img src="https://l.sr.ht/pMHw.jpg" alt=""></p>
<p>I often use this film as an example when arguing about copyright. Almost
everyone I speak to was born well after the film&rsquo;s release (in fact, this is
true of almost everyone <em>alive today</em>), but they remember it fondly regardless.
Many people I&rsquo;ve spoken to would agree that it even played a formative role in
their childhoods; it&rsquo;s a film dear to many hearts. My mom is very fond of the
Cheshire Cat in particular, and owns quite a bit of relevant merchandise.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p>
<p>Like many films from their &ldquo;Golden Age&rdquo;, Disney&rsquo;s <em>Alice</em> is itself a derivative
work, based on Lewis Carroll&rsquo;s 1865 book. However, Disney&rsquo;s film won&rsquo;t enter the
public domain until 2046, and until then, no one can create derivative works of
their own without receiving permission from and paying a tithe to Disney. And if
modern-day copyright law, <a href="https://en.wikipedia.org/wiki/Copyright_Term_Extension_Act">bought and paid for by Disney</a>, had
been in force at the time Alice in Wonderland was made, they would have released
their film 17 years <em>before</em> Carroll&rsquo;s novel entered the public domain.</p>
<p>Carroll, who died in 1898, was 53 years dead when the film was released. Everyone
who is listed in the credits for Disney&rsquo;s Alice in Wonderland is also dead, with
the exception of Kathryn Beaumont, who played the role of none other than Alice
herself.<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> <sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> She was 12 years old at the time. And still today, the
copyright remains in force, though no creators remain to enjoy its privileges.
It shall remain so for another 26 years, when I can finally celebrate my
Alice-in-Wonderland-themed 53rd birthday party, having been robbed of the
privilege at age 11.<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup></p>
<p>Copyright was established in the United States to incentivize artists,
musicians, authors, writers, and other creatives to create novel art, allowing
them to enjoy the exclusive rights to it for a short period<sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup>, then ultimately
<em>enriching</em> the public domain. The obscene copyright terms we&rsquo;re faced with
today have robbed the American public of its national heritage. Any work made
today will not enter the public domain during the lifetimes of any of its
contemporaries, let alone soon enough for those contemporaries to <em>do</em> anything
with it.</p>
<p>A system designed to incentivize creation has become a system which incentivises
the opposite: rent seeking. A rent which is sought from the American public, in
exchange for which we&rsquo;re no longer getting our end of the deal.</p>
<p>Well, the deal is off.</p>
<link rel="stylesheet" href="https://drewdevault.com/video-js.css">
<script>
window.HELP_IMPROVE_VIDEOJS = false;
</script>
<script src="https://drewdevault.com/video.js"></script>
<video class="video-js vjs-16-9" data-setup="{}" controls>
  <source src="https://drewdevault.com/alice.webm" type="video/webm">
  <p>Your browser does not support HTML5 video, or webm. Either way you're not
  going to watch this video.</p>
</video>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>She is not sure how much of that merchandise is officially licensed. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>The last person credited for Alice in Wonderland to have died was <a href="https://en.wikipedia.org/wiki/Don_Lusk">Don Lusk</a>, who died in 2018 at the age of 105. He lived through World War I, fought in World War II, then went on to animate 17 films for Disney. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Another honorable mention goes to <a href="https://en.wikipedia.org/wiki/Ben_Sharpsteen">Ben Sharpsteen</a>, the production director on <em>Alice</em>, who enjoyed the status of oldest staff member on the production, having been born in 1895. He was alive in Lewis Carroll&rsquo;s lifetime! <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>Okay, I&rsquo;ll fess up: I never had any plans of an Alice-themed birthday party when I was 11, or at any other age. But you can bet I&rsquo;m planning one for my 53rd now! <a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>14 years, or 28 years if renewed <a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/08/27/Microsoft-plays-their-hand.html
title: Embrace, extend, and finally extinguish - Microsoft plays their hand
url: https://drewdevault.com/2020/08/27/Microsoft-plays-their-hand.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>GitHub took a note out of the Microsoft &ldquo;<abbr title="Embrace, Extend,
Extinguish">EEE</abbr>&rdquo; playbook when designing their git services. They
<strong>embraced</strong> git, and then rather than building an interface on top of email
— the collaboration mechanism that git was designed to use, and which is
still used for Linux kernel development<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> — they built their &ldquo;pull
requests&rdquo; mechanism.</p>
<p>They took terminology which already had meaning — &ldquo;fork&rdquo;, meaning the
creation a separate governing body and development upstream for a codebase, a
rather large task; and &ldquo;pull request&rdquo;, a git workflow which prepares an email
asking a receipient to pull a large branch of changes from a non-centralized
source — and replaced these decentralized, open systems with a completely
incompatible system designed to keep you on GitHub and to teach you to
collaborate using GitHub&rsquo;s proprietary tools. They <strong>extended</strong> git in a
proprietary way.</p>
<p>Microsoft knows a good deal when they see one, and picked up GitHub for a cool
$7,500,000,000, after they had already completed the two steps in Microsoft&rsquo;s
<a href="https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish">anti-open-source playbook</a>. They joined the Linux Foundation in late 2016,
after Azure failed to win people back to Windows Server, admitting defeat while
simultaneously carving out a space from which they could project their interests
over the kernel.</p>
<p>Today, I discovered this article, &ldquo;<a href="https://www.theregister.com/2020/08/25/linux_kernel_email/">Relying on plain-text email is a
&lsquo;barrier to entry&rsquo; for kernel development, says Linux Foundation board
member</a>&rdquo;, a title which conveniently chooses to refer to Sarah Novotny by
her role as a Linux Foundation board member, rather than by her full title,
&ldquo;Sarah Novotny, Microsoft employee, transitive owner of GitHub, and patroness
saint of conflicts of interests.&rdquo; Finally, they&rsquo;re playing the <strong>extinguish</strong>
card. Naturally, a representative of Microsoft, a company which has long waged
war against open source, and GitHub, a company which explicitly built an
incompatible proprietary system to extend git, would have an interest in
dismantling the distributed, open system that git was designed for.</p>
<p>I represent <a href="https://sourcehut.org">sourcehut</a>, a GitHub competitor which does
what GitHub wouldn&rsquo;t — interoperate with open, distributed protocols, and
in the form of 100% free and open-source software. I agree that the UX of
email-driven development could be better! But instead of investing $7.5B into
throwing the baby out with the bathwater, we&rsquo;ve <a href="https://git-send-email.io/">built interactive
tutorials</a>, <a href="https://lists.sr.ht/~emersion/mrsh-dev/patches/4728">designed better mailing lists</a>, <a href="https://sr.ht/_fUk.webm">built web interfaces for
patch submission</a>, <a href="https://sourcehut.org/blog/2020-07-14-setting-up-ci-for-mailing-lists/">implemented CI for emails</a> and <a href="https://github.com/git/git/commits?author=ddevault">sent improvements to
git upstream</a>. I wrote <a href="https://aerc-mail.org/">an entire mail client which makes it easier to use
these tools</a>. We&rsquo;re planning on web-based review interface, too. The result
is a UX which provides a similar experience to GitHub, but without disrupting
the established open ecosystem.</p>
<p><em>This</em> is how you improve the ecosystem, Microsoft. Take notes. Stick with the
embrace, move your extending <em>upstream</em>, and forget about extinguish.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>And hundreds of other projects, including git itself. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/09/02/Linux-development-is-profoundly-distributed.html
title: Linux development is distributed - profoundly so
url: https://drewdevault.com/2020/09/02/Linux-development-is-profoundly-distributed.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>The standard introduction to git starts with an explanation of what it means to
use a &ldquo;distributed&rdquo; version control system. It&rsquo;s pointed out that every
developer has a complete local copy of the repository and can work independently
and offline, often contrasting this design with systems like SVN and CVS.  The
explanation usually stops here. If you want to learn more, consider git&rsquo;s roots:
it is the version control system purpose-built for Linux, the largest and most
active open source project in the world. To learn more about the true nature of
distributed development, we should observe Linux.</p>
<p>Pull up your local copy of the Linux source code (you have one of those,
right?<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>) and open the MAINTAINERS file. Scroll down to line 150 or so and
let&rsquo;s start reading some of these entries.</p>
<p>Each of these represents a different individual or group which has some interest
in the Linux kernel, often a particular driver. Most of them have an &ldquo;F&rdquo; entry,
which indicates which files they&rsquo;re responsible for in the source code. Most
have an &ldquo;L&rdquo; entry, which has a mailing list you can post questions, bug reports,
and patches to, as well as an individual maintainer (&ldquo;M&rdquo;) or maintainers who are
known to have expertise and autonomy over this part of the kernel. Many of them
— but, hmm, not all — also have a tree (&ldquo;T&rdquo;), which is a dedicated
git repo with their copy of Linux, for staging changes to the kernel. This is
common with larger drivers or with &ldquo;meta&rdquo; organizations, which oversee
development of entire subsystems.</p>
<p>However, this presents a simplified view. Look carefully at the &ldquo;DRM&rdquo; drivers
(<a href="https://en.wikipedia.org/wiki/Direct_Rendering_Manager">Direct Rendering Manager</a>); a group of drivers and maintainers who are
collectively responsible for graphics on Linux. There are many drivers and many
maintainers, but a careful eye will notice that there are many similarities as
well. A lot of them use the same mailing list, <a href="mailto:dri-devel@lists.freedesktop.org">dri-devel@lists.freedesktop.org</a>,
and many of them use the same git repository:
<code>git://anongit.freedesktop.org/drm/drm-misc</code>. It&rsquo;s not mentioned in this file,
but many of them also shared the FreeDesktop bugzilla until recently, then moved
to the FreeDesktop GitLab; and many of them share the <code>#dri-devel</code> IRC channel
on Freenode. And again I&rsquo;m simplifying — there are also many related IRC
channels and git repos, and some larger drivers like AMDGPU have dedicated
mailing lists and trees.</p>
<p>There&rsquo;s more complexity to this system still. For example, not all of these
subsystems are using git. The Intel TXT subsystem uses Mercurial. The Device
Mapper team (one of the largest and most important Linux subsystems) uses
<a href="https://savannah.nongnu.org/projects/quilt">Quilt</a>. And like Linux DRM is a meta-project for many DRM-related subsystems
&amp; drivers, there are higher-level meta projects still, such as driver-core,
which manages code and subsystems common to <em>all</em> I/O drivers. There are also
cross-cutting concerns, such as the interaction between linux-usb and various
network driver teams.</p>
<p>Patches to any particular driver could first end up on a domain-specific mailing
list, with a particular maintainer being responsible for reviewing and
integrating the patch, with their own policies and workflows and tooling. Then
it might flow upwards towards another subsystem with its own similar features,
and then up again towards meta-meta trees like linux-staging, and eventually to
Linus' tree<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>. Along the way it might receive feedback from other projects if it
has cross-cutting concerns, tracing out an ever growing and shrinking bubble of
inclusion among the trees, ultimately ending up in every tree. And that&rsquo;s
<em>still</em> a simplification — for example, an important bug fix may sidestep
all of this entirely and get applied on top of a downstream distribution kernel,
ending up on end-user machines before it&rsquo;s made much progress upstream at all.</p>
<p>This complex <em>graph</em> of Linux development has code flowing smoothly between
hundreds of repositories, emails exchanging between hundreds of mailing lists,
passing through the hands of dozens of maintainers, several bug trackers,
various CI systems, all day, every day, ten-thousand fold. This is truly
illustrative of <strong>distributed</strong> software development, well above and beyond the
typical explanation given to a new git user. The profound potential of the
distributed git system can be plainly seen in the project for which it was
principally designed. It&rsquo;s also plain to see how difficult it would be to adapt
this system to something like GitHub pull requests, despite how easy many who
are perplexed by the email-driven workflow wish it to be<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>. As a matter of
fact, several Linux teams are already using GitHub and GitLab and even pull or
merge requests on their respective platforms.  However, scaling this system up
to the entire kernel would be a great challenge indeed.</p>
<p>By the way — that MAINTAINERS file? Scroll to the bottom. My copy is
<em>19,000 lines long</em>.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Okay, just in case: <code>git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git</code> <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>That&rsquo;s not the only destination; for example, some patches will end up in the LTS kernels as well. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>If you are among the perplexed, <a href="https://git-send-email.io">my interactive git send-email tutorial</a> takes about 10 minutes and is often recommended to new developers by Greg KH himself. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/09/15/Status-update-September-2020.html
title: Status update, September 2020
url: https://drewdevault.com/2020/09/15/Status-update-September-2020.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>A mercifully cool September is upon us, and after years of searching, I finally
was able to secure Club Mate in the US. Let&rsquo;s decant a bottle and recant the
story of this month&rsquo;s progress in free software development.</p>
<p>First of all, I&rsquo;ve been able to put a pin on operations work on SourceHut for
the time being, and focus again on its software development. The GraphQL APIs
are a major focus area here, and I&rsquo;ve made a lot of progress towards OAuth 2.0
support and writable GraphQL APIs. Additionally, I&rsquo;ve laid out a number of
prioritized tickets for the beta — with the &ldquo;beta&rdquo; label on todo.sr.ht
— and have been picking items off of the list one at a time, mainly
focusing on meta.sr.ht improvements at first. I&rsquo;ll go into more detail in the
What&rsquo;s Cooking post for SourceHut later today, stay tuned.</p>
<p>There has been some advancements in the little projects: a second Python
implementation of BARE has appeared, another in Common Lisp, and one in PHP;
bringing the total implementations to nine. We have a pretty decent spread of
support among programming languages!</p>
<p>Not much more news to share today. Been focusing in on SourceHut and a secret
project, so check out the What&rsquo;s Cooking post for more details. Thanks for your
support!</p>
<details>
<summary>...</summary>
<img
  src="https://cmpwn.com/system/media_attachments/files/000/970/411/original/f5d4c6f553a5a0ce.png?1599509919"
  alt="A screenshot of a programming language with a DRAFT watermark. The page shown describes the syntax and semantics of enum types."
/>
</details>


guid: https://drewdevault.com/2020/09/20/The-potential-of-federation.html
title: The unrealized potential of federation
url: https://drewdevault.com/2020/09/20/The-potential-of-federation.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>There are some major problems on the internet which may seem intractable. How do
we prevent centralization of our communication tools under the authority of a
few, whose motivations may not align with our interests? How do we build
internet-scale infrastructure without a megacorp-scale budget? Can we make our
systems reliable and fault-tolerant — in the face of technical <em>and</em>
social problems?</p>
<p><strong>Federation</strong> is an idea which takes a swing at all of these problems.</p>
<p><em>Note: apparently some cryptocurrency enthusiasts are parading this article
around to peddle their garbage. Cryptocurrency is the digitally woke techbro&rsquo;s
ponzi scheme, and is a massive waste of electricity and developer effort. Anyone
who tells you anything positive about anything which is even remotely connected
to cryptocurrency almost certainly has ulterior motives and you should steer
clear. So hopefully that settles that. And cryptocurrency is a P2P system,
anyway, NOT a federation!</em></p>
<p>The key trait of a software system which is <em>federated</em> is that the servers are
controlled by independent, sovereign entities, and that they exist together
under a common web of communication protocols and social agreements. This
occupies a sort of middle ground between the centralized architecture and the
peer-to-peer (or &ldquo;decentralized&rdquo;) architecture. Federation enjoys the advantages
of both, and few of the drawbacks.</p>
<p>In a federated software system, groups of users are built around small,
neighborly instances of servers. These are usually small servers, sporting only
modest resource requirements to support their correspondingly modest userbase.
Crucially, these small servers speak to <em>one another</em> using standard protocols,
allowing users of one instance to communicate seamlessly with users of other
instances. You can build a culture and shared sense of identity on your
instance, but also reach out and easily connect with other instances.</p>
<p>The governance of a federated system then becomes distributed among many
operators. Every instance has the following privileges:</p>
<ol>
<li>To set the rules which govern users of their instance</li>
<li>To set the rules which govern who they federate with</li>
</ol>
<p>And, because there are hundreds or even thousands of instances, the users get
the privilege of choosing an instance whose rules they like, and which federates
with other instances they wish to talk to. This system also makes it hard for
marketing and spam to get a foothold — it optimizes for a self-governing
system of human beings talking to human beings, and not for corporations to push
their products.</p>
<p>The costs of scaling up a federation is distributed manageably among these
operators. Small instances, with their modest server requirements, are often
cheap enough that a sysadmin can comfortably pay for the expenses out of pocket.
If not, it&rsquo;s usually quite easy to solicit donations from the users to keep
things running. New operators appear all the time, and the federation scales up
a little bit more.</p>
<p>Unlike P2P systems, the federated model allows volunteer sysadmins to use their
skills to expand access to the service to non-technical users, without placing
the burden on those non-technical users to set up, understand, maintain, or
secure servers or esoteric software. The servers are also always online and
provide strong identities and authenticity guarantees — eliminating an
entire class of P2P problems.</p>
<p>A popular up-and-coming protocol for federation is ActivityPub, but it&rsquo;s not the
only way to build a federated system. You&rsquo;re certainly familiar with another
federation which is not based on ActivityPub: email. IRC and Matrix also provide
federated protocols in the instant messaging domain. Personally, I don&rsquo;t like
ActivityPub, but AP is not necessary to reap the benefits of federation. Many
different kinds of communication systems can be designed with federation in
mind, and adjust their approach to accommodate their specific needs, evident in
each of these examples.</p>
<p>In short, federation distributes governance and cost, and can allow us to tackle
challenges that we couldn&rsquo;t overcome without it. The free software community
needs to rally behind federation, because no one else will. For all of the
reasons which make it worth doing, it is not rewarding for corporations.  They
would much rather build walled gardens and centralize, centralize, centralize
— it&rsquo;s more profitable!  Democratic software which puts control into the
hands of the users is something we&rsquo;re going to have to take for ourselves. Viva
la federación!</p>


guid: https://drewdevault.com/2020/09/21/Gemini-TOFU.html
title: TOFU recommendations for Gemini
url: https://drewdevault.com/2020/09/21/Gemini-TOFU.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I will have more to say about <a href="https://gemini.circumlunar.space/">Gemini</a> in the future, but for now, I wanted to
write up some details about one thing in particular: the trust-on-first-use
algorithm I implemented for my client, <a href="https://sr.ht/~sircmpwn/gmni">gmni</a>. I think you should implement
this algorithm, too!</p>
<p>First of all, it&rsquo;s important to note that the Gemini specification explicitly
mentions TOFU and the role of self-signed certificates: they are the norm in
Geminiland, and if your client does not support them then you&rsquo;re going to be
unable to browse many sites. However, the exact details are left up to the
implementation. Here&rsquo;s what mine does:</p>
<p>First, on startup, it finds the known_hosts file. For my client, this is
<code>~/.local/share/gmni/known_hosts</code> (the exact path is adjusted as necessary per
the XDG basedirs specification). Each line of this file represents a known host,
and each host has four fields separated by spaces, in this order:</p>
<ul>
<li>Hostname (e.g. gemini.circumlunar.space)</li>
<li>Fingerprint algorithm (e.g. SHA-512)</li>
<li>Fingerprint, in hexadecimal, with &lsquo;:&rsquo; between each octet (e.g. 55:01:D8&hellip;)</li>
<li>Unix timestamp of the certificate&rsquo;s notAfter date</li>
</ul>
<p>If a known_hosts entry is encountered with a hashing algorithm you don&rsquo;t
understand, it is disregarded.</p>
<p>Then, when processing a request and deciding whether or not to trust its
certificate, take the following steps:</p>
<ol>
<li>Verify that the certificate makes sense. Check the notBefore and notAfter
dates against the current time, and check that the hostname is correct
(including wildcards). Apply any other scrutiny you want, like enforcing a
good hash algorithm or an upper limit on the expiration date. If these checks
do not pass, the trust state is INVALID, GOTO 5.</li>
<li>Compute the certificate&rsquo;s fingerprint. Use the entire certificate (in OpenSSL
terms, <code>X509_digest</code> will do this), not just the public key.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></li>
<li>Look up the known_hosts record for this hostname. If one is found, but the
record is expired, disregard it. If one is found, and the fingerprint does
not match, the trust state is UNTRUSTED, GOTO 5. Otherwise, the trust state
is TRUSTED. GOTO 7.</li>
<li>The trust state is UNKNOWN. GOTO 5.</li>
<li>Display information about the certficate and its trust state to the user, and
prompt them to choose an action, from the following options:
<ul>
<li>If INVALID, the user&rsquo;s choices are ABORT or TRUST_TEMPORARY.</li>
<li>If UNKNOWN, the user&rsquo;s choices are ABORT, TRUST_TEMPORARY, or TRUST_ALWAYS.</li>
<li>If UNTRUSTED, abort the request and display a diagnostic message. The user
must manually edit the known_hosts file to correct the issue.</li>
</ul>
</li>
<li>Complete the requested action:
<ul>
<li>If ABORT, terminate the request.</li>
<li>If TRUST_TEMPORARY, update the session&rsquo;s list of known hosts.</li>
<li>If TRUST_ALWAYS, append a record to the known_hosts file and update the
session&rsquo;s list of known hosts.</li>
</ul>
</li>
<li>Allow the request to proceed.</li>
</ol>
<p>If the trust state is UNKNOWN, instead of requring user input to proceed, the
implementation MAY proceed with the request IF the UI displays that a new
certificate was trusted and provides a means to review the certificate and
revoke that trust.</p>
<p>Note that being signed by a certificate authority in the system trust store is
not considered meaningful to this algorithm. Such a cert is TOFU&rsquo;d all the same.</p>
<p>That&rsquo;s it! If you have feedback on this approach, please <a href="mailto:sir@cmpwn.com">send me an
email</a>.</p>
<p>My implementation doesn&rsquo;t <em>entirely</em> match this behavior, but it&rsquo;s close and
I&rsquo;ll finish it up before 1.0. If you want to read the code, <a href="https://git.sr.ht/~sircmpwn/gmni/tree/master/src/tofu.c">here it is</a>.</p>
<p>Bonus recommendation for servers: you <strong>should</strong> use a self-signed certificate,
and you <strong>should not</strong> use a certificate signed by one of the mainstream
certificate authorities. We don&rsquo;t need to carry along the legacy CA cabal into
our brave new Gemini future.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Rationale: this fingerprint matches the output of <code>openssl x509 -sha512 -fingerprint</code>. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/09/25/A-story-of-two-libcs.html
title: A tale of two libcs
url: https://drewdevault.com/2020/09/25/A-story-of-two-libcs.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I received a bug report from Debian today, who had fed some garbage into
<a href="https://git.sr.ht/~sircmpwn/scdoc">scdoc</a>, and it gave them a SIGSEGV back.
Diving into this problem gave me a good opportunity to draw a comparison between
musl libc and glibc. Let&rsquo;s start with the stack trace:</p>
<pre><code>==26267==ERROR: AddressSanitizer: SEGV on unknown address 0x7f9925764184
(pc 0x0000004c5d4d bp 0x000000000002 sp 0x7ffe7f8574d0 T0)
==26267==The signal is caused by a READ memory access.
    0 0x4c5d4d in parse_text /scdoc/src/main.c:223:61
    1 0x4c476c in parse_document /scdoc/src/main.c
    2 0x4c3544 in main /scdoc/src/main.c:763:2
    3 0x7f99252ab0b2 in __libc_start_main
/build/glibc-YYA7BZ/glibc-2.31/csu/../csu/libc-start.c:308:16
    4 0x41b3fd in _start (/scdoc/scdoc+0x41b3fd)
</code></pre><p>And if we pull up that line of code, we find&hellip;</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">isalnum</span><span class="p">(</span><span class="n">last</span><span class="p">)</span> <span class="o">||</span> <span class="p">((</span><span class="n">p</span><span class="o">-&gt;</span><span class="n">flags</span> <span class="o">&amp;</span> <span class="n">FORMAT_UNDERLINE</span><span class="p">)</span> <span class="o">&amp;&amp;</span> <span class="o">!</span><span class="n">isalnum</span><span class="p">(</span><span class="n">next</span><span class="p">)))</span> <span class="p">{</span>
</code></pre></div><p>Hint: p is a valid pointer. &ldquo;last&rdquo; and &ldquo;next&rdquo; are both uint32_t. The segfault
happens in the second call to isalnum. And, the key: it can only be reproduced
on glibc, not on musl libc. If you did a double-take, you&rsquo;re not alone. There&rsquo;s
nothing here which could have caused a segfault.</p>
<p>Since it was narrowed down to glibc, I pulled up the source code and went
digging for the isalnum implementation, expecting some stupid bullshit. But
before I get into their stupid bullshit, of which I can assure you there is <em>a
lot</em>, let&rsquo;s briefly review the happy version. This is what the musl libc
<code>isalnum</code> implementation looks like:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">int</span> <span class="nf">isalnum</span><span class="p">(</span><span class="kt">int</span> <span class="n">c</span><span class="p">)</span>
<span class="p">{</span>
	<span class="k">return</span> <span class="n">isalpha</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="o">||</span> <span class="n">isdigit</span><span class="p">(</span><span class="n">c</span><span class="p">);</span>
<span class="p">}</span>

<span class="kt">int</span> <span class="nf">isalpha</span><span class="p">(</span><span class="kt">int</span> <span class="n">c</span><span class="p">)</span>
<span class="p">{</span>
	<span class="k">return</span> <span class="p">((</span><span class="kt">unsigned</span><span class="p">)</span><span class="n">c</span><span class="o">|</span><span class="mi">32</span><span class="p">)</span><span class="o">-</span><span class="sc">&#39;a&#39;</span> <span class="o">&lt;</span> <span class="mi">26</span><span class="p">;</span>
<span class="p">}</span>

<span class="kt">int</span> <span class="nf">isdigit</span><span class="p">(</span><span class="kt">int</span> <span class="n">c</span><span class="p">)</span>
<span class="p">{</span>
	<span class="k">return</span> <span class="p">(</span><span class="kt">unsigned</span><span class="p">)</span><span class="n">c</span><span class="o">-</span><span class="sc">&#39;0&#39;</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div><p>As expected, for any value of <code>c</code>, isalnum will never segfault. Because why the
fuck would isalnum segfault? Okay, now, let&rsquo;s compare this to the
<a href="https://sourceware.org/git/?p=glibc.git;a=blob;f=ctype/ctype.h;h=351495aa4feaf23993fe65afc0760615268d044e;hb=HEAD">glibc implementation</a>. When opening this header, you&rsquo;re greeted with the
typical GNU bullshit, but let&rsquo;s trudge through and grep for isalnum.</p>
<p>The first result is this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">enum</span>
<span class="p">{</span>
  <span class="n">_ISupper</span> <span class="o">=</span> <span class="n">_ISbit</span> <span class="p">(</span><span class="mi">0</span><span class="p">),</span>        <span class="cm">/* UPPERCASE.  */</span>
  <span class="n">_ISlower</span> <span class="o">=</span> <span class="n">_ISbit</span> <span class="p">(</span><span class="mi">1</span><span class="p">),</span>        <span class="cm">/* lowercase.  */</span>
  <span class="c1">// ...
</span><span class="c1"></span>  <span class="n">_ISalnum</span> <span class="o">=</span> <span class="n">_ISbit</span> <span class="p">(</span><span class="mi">11</span><span class="p">)</span>        <span class="cm">/* Alphanumeric.  */</span>
<span class="p">};</span>
</code></pre></div><p>This looks like an implementation detail, let&rsquo;s move on.</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">__exctype</span> <span class="p">(</span><span class="n">isalnum</span><span class="p">);</span>
</code></pre></div><p>But what&rsquo;s <code>__exctype</code>? Back up the file a few lines&hellip;</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="cp">#define __exctype(name) extern int name (int) __THROW
</span></code></pre></div><p>Okay, apparently that&rsquo;s just the prototype. Not sure why they felt the need to
write a macro for that. Next search result&hellip;</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="cp">#if !defined __NO_CTYPE
</span><span class="cp"># ifdef __isctype_f
</span><span class="cp"></span><span class="n">__isctype_f</span> <span class="p">(</span><span class="n">alnum</span><span class="p">)</span>
<span class="c1">// ...
</span></code></pre></div><p>Okay, this looks useful. What is <code>__isctype_f</code>? Back up the file now&hellip;</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="cp">#ifndef __cplusplus
</span><span class="cp"># define __isctype(c, type) \
</span><span class="cp">  ((*__ctype_b_loc ())[(int) (c)] &amp; (unsigned short int) type)
</span><span class="cp">#elif defined __USE_EXTERN_INLINES
</span><span class="cp"># define __isctype_f(type) \
</span><span class="cp">  __extern_inline int                                                         \
</span><span class="cp">  is##type (int __c) __THROW                                                  \
</span><span class="cp">  {                                                                           \
</span><span class="cp">    return (*__ctype_b_loc ())[(int) (__c)] &amp; (unsigned short int) _IS##type; \
</span><span class="cp">  }
</span><span class="cp">#endif
</span></code></pre></div><p>Oh&hellip;. oh dear. It&rsquo;s okay, we&rsquo;ll work through this together. Let&rsquo;s see,
<code>__isctype_f</code> is some kind of inline function&hellip; wait, this is the else branch
of <code>#ifndef __cplusplus</code>. Dead end. Where the fuck is isalnum <em>actually</em>
defined? Grep again&hellip; okay&hellip; here we are?</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="cp">#if !defined __NO_CTYPE
</span><span class="cp"># ifdef __isctype_f
</span><span class="cp"></span><span class="n">__isctype_f</span> <span class="p">(</span><span class="n">alnum</span><span class="p">)</span>
<span class="c1">// ...
</span><span class="c1"></span><span class="cp"># elif defined __isctype
</span><span class="cp"># define isalnum(c)     __isctype((c), _ISalnum) </span><span class="c1">// &lt;- this is it
</span></code></pre></div><p>Hey, there&rsquo;s that implementation detail from earlier! Remember this?</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">enum</span>
<span class="p">{</span>
  <span class="n">_ISupper</span> <span class="o">=</span> <span class="n">_ISbit</span> <span class="p">(</span><span class="mi">0</span><span class="p">),</span>        <span class="cm">/* UPPERCASE.  */</span>
  <span class="n">_ISlower</span> <span class="o">=</span> <span class="n">_ISbit</span> <span class="p">(</span><span class="mi">1</span><span class="p">),</span>        <span class="cm">/* lowercase.  */</span>
  <span class="c1">// ...
</span><span class="c1"></span>  <span class="n">_ISalnum</span> <span class="o">=</span> <span class="n">_ISbit</span> <span class="p">(</span><span class="mi">11</span><span class="p">)</span>        <span class="cm">/* Alphanumeric.  */</span>
<span class="p">};</span>
</code></pre></div><p>Let&rsquo;s suss out that macro real quick:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="cp"># include &lt;bits/endian.h&gt;
</span><span class="cp"># if __BYTE_ORDER == __BIG_ENDIAN
</span><span class="cp">#  define _ISbit(bit)   (1 &lt;&lt; (bit))
</span><span class="cp"># else </span><span class="cm">/* __BYTE_ORDER == __LITTLE_ENDIAN */</span><span class="cp">
</span><span class="cp">#  define _ISbit(bit)   ((bit) &lt; 8 ? ((1 &lt;&lt; (bit)) &lt;&lt; 8) : ((1 &lt;&lt; (bit)) &gt;&gt; 8))
</span><span class="cp"># endif
</span></code></pre></div><p>Oh, for fuck&rsquo;s sake. Whatever, let&rsquo;s move on and just assume this is a magic
number. The other macro is <code>__isctype</code>, which is similar to the <code>__isctype_f</code> we
were just looking at a moment ago. Let&rsquo;s go look at that <code>ifndef __cplusplus</code>
branch again:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="cp">#ifndef __cplusplus
</span><span class="cp"># define __isctype(c, type) \
</span><span class="cp">  ((*__ctype_b_loc ())[(int) (c)] &amp; (unsigned short int) type)
</span><span class="cp">#elif defined __USE_EXTERN_INLINES
</span><span class="cp"></span><span class="c1">// ...
</span><span class="c1"></span><span class="cp">#endif
</span></code></pre></div><p>&hellip;</p>
<p>Well, at least we have a pointer dereference now, that could explain the
segfault. What&rsquo;s <code>__ctype_b_loc</code>?</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="cm">/* These are defined in ctype-info.c.
</span><span class="cm">   The declarations here must match those in localeinfo.h.
</span><span class="cm">
</span><span class="cm">   In the thread-specific locale model (see `uselocale&#39; in &lt;locale.h&gt;)
</span><span class="cm">   we cannot use global variables for these as was done in the past.
</span><span class="cm">   Instead, the following accessor functions return the address of
</span><span class="cm">   each variable, which is local to the current thread if multithreaded.
</span><span class="cm">
</span><span class="cm">   These point into arrays of 384, so they can be indexed by any `unsigned
</span><span class="cm">   char&#39; value [0,255]; by EOF (-1); or by any `signed char&#39; value
</span><span class="cm">   [-128,-1).  ISO C requires that the ctype functions work for `unsigned
</span><span class="cm">   char&#39; values and for EOF; we also support negative `signed char&#39; values
</span><span class="cm">   for broken old programs.  The case conversion arrays are of `int&#39;s
</span><span class="cm">   rather than `unsigned char&#39;s because tolower (EOF) must be EOF, which
</span><span class="cm">   doesn&#39;t fit into an `unsigned char&#39;.  But today more important is that
</span><span class="cm">   the arrays are also used for multi-byte character sets.  */</span>
<span class="k">extern</span> <span class="k">const</span> <span class="kt">unsigned</span> <span class="kt">short</span> <span class="kt">int</span> <span class="o">**</span><span class="nf">__ctype_b_loc</span> <span class="p">(</span><span class="kt">void</span><span class="p">)</span>
     <span class="n">__THROW</span> <span class="n">__attribute__</span> <span class="p">((</span><span class="n">__const__</span><span class="p">));</span>
<span class="k">extern</span> <span class="k">const</span> <span class="n">__int32_t</span> <span class="o">**</span><span class="nf">__ctype_tolower_loc</span> <span class="p">(</span><span class="kt">void</span><span class="p">)</span>
     <span class="n">__THROW</span> <span class="n">__attribute__</span> <span class="p">((</span><span class="n">__const__</span><span class="p">));</span>
<span class="k">extern</span> <span class="k">const</span> <span class="n">__int32_t</span> <span class="o">**</span><span class="nf">__ctype_toupper_loc</span> <span class="p">(</span><span class="kt">void</span><span class="p">)</span>
     <span class="n">__THROW</span> <span class="n">__attribute__</span> <span class="p">((</span><span class="n">__const__</span><span class="p">));</span>
</code></pre></div><p>That is just so, super cool of you, glibc. I just <em>love</em> dealing with locales.
Anyway, my segfaulted process is sitting in gdb, and equipped with all of this
information I wrote the following monstrosity:</p>
<pre><code>(gdb) print ((unsigned int **(*)(void))__ctype_b_loc)()[next]
Cannot access memory at address 0x11dfa68
</code></pre><p>Segfault found. Reading that comment again, we see &ldquo;ISO C requires that the
ctype functions work for &lsquo;unsigned char&rsquo; values and for EOF&rdquo;. If we
cross-reference that with the specification:</p>
<blockquote>
<p>In all cases [of functions defined by ctype.h,] the argument is an int, the
value of which shall be representable as an unsigned char or shall equal the
value of the macro EOF.</p>
</blockquote>
<p>So the fix is obvious at this point. Okay, fine, my bad. My code is wrong. I
apparently cannot just hand a UCS-32 codepoint to isalnum and expect it to tell
me if it&rsquo;s between 0x30-0x39, 0x41-0x5A, or 0x61-0x7A.</p>
<p>But, I&rsquo;m going to go out on a limb here: maybe isalnum should never cause a
program to segfault no matter what input you give it. Maybe because the spec
says you <em>can</em> does not mean you <em>should</em>. Maybe, just maybe, the behavior of
this function should not depend on five macros, whether or not you&rsquo;re using a
C++ compiler, the endianness of your machine, a look-up table, thread-local
storage, and two pointer dereferences.</p>
<p>Here&rsquo;s the musl version as a quick reminder:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">int</span> <span class="nf">isalnum</span><span class="p">(</span><span class="kt">int</span> <span class="n">c</span><span class="p">)</span>
<span class="p">{</span>
	<span class="k">return</span> <span class="n">isalpha</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="o">||</span> <span class="n">isdigit</span><span class="p">(</span><span class="n">c</span><span class="p">);</span>
<span class="p">}</span>

<span class="kt">int</span> <span class="nf">isalpha</span><span class="p">(</span><span class="kt">int</span> <span class="n">c</span><span class="p">)</span>
<span class="p">{</span>
	<span class="k">return</span> <span class="p">((</span><span class="kt">unsigned</span><span class="p">)</span><span class="n">c</span><span class="o">|</span><span class="mi">32</span><span class="p">)</span><span class="o">-</span><span class="sc">&#39;a&#39;</span> <span class="o">&lt;</span> <span class="mi">26</span><span class="p">;</span>
<span class="p">}</span>

<span class="kt">int</span> <span class="nf">isdigit</span><span class="p">(</span><span class="kt">int</span> <span class="n">c</span><span class="p">)</span>
<span class="p">{</span>
	<span class="k">return</span> <span class="p">(</span><span class="kt">unsigned</span><span class="p">)</span><span class="n">c</span><span class="o">-</span><span class="sc">&#39;0&#39;</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div><p>Bye!</p>


guid: https://drewdevault.com/2020/10/01/Spamtoberfest.html
title: Spamtoberfest
url: https://drewdevault.com/2020/10/01/Spamtoberfest.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>As I&rsquo;ve <a href="https://drewdevault.com/2020/08/10/How-to-contribute-to-FOSS.html">written before</a>, the best contributors to a FOSS project are
intrinsically motivated to solve problems in your software. This sort of
contribution is often fixing an important problem and places a smaller burden on
maintainers to spend their time working with the contributor. I&rsquo;ve previously
contrasted this with the &ldquo;I want to help out!&rdquo; contributions, where a person
just has a vague desire to help out. Those contributions are, generally, less
valuable and place a greater burden on the maintainer. Now, DigitalOcean has
lowered the bar even further with Hacktoberfest.</p>
<p><em>Disclaimer: I am the founder of a FOSS project hosting company similar to GitHub.</em></p>
<p>As I write this, a Digital Ocean-sponsored and GitHub-enabled Distributed Denial
of Service (DDoS) attack is ongoing, wasting the time of thousands of free
software maintainers with an onslaught of meaningless spam. Bots are spamming
<a href="https://github.com/search?q=amazing+project+is:pr&amp;type=Issues">tens of thousands</a> of pull requests like this:</p>
<p><a href="https://github.com/hundredrabbits/100r.co/pull/39/files"><img src="https://l.sr.ht/71VU.png" alt="Screenshot of a spam pull request on GitHub which adds garbage to the README.md file"></a></p>
<p>The official response from both Digital Ocean and GitHub appears to be passing
the buck.  Digital Ocean addresses spam in their FAQ, putting the burden of
dealing with it entirely on the maintainers:</p>
<blockquote>
<p>Spammy pull requests can be given a label that contains the word &ldquo;invalid&rdquo; or
&ldquo;spam&rdquo; to discount them. Maintainers are faced with the majority of spam that
occurs during Hacktoberfest, and we dislike spam just as much as you. If
you&rsquo;re a maintainer, please label any spammy pull requests submitted to the
repositories you maintain as &ldquo;invalid&rdquo; or &ldquo;spam&rdquo;, and close them. Pull
requests with this label won&rsquo;t count toward Hacktoberfest.</p>
</blockquote>
<p>via <a href="https://hacktoberfest.digitalocean.com/details">Hacktoberfest FAQ</a></p>
<p>Here&rsquo;s GitHub&rsquo;s response:</p>
<blockquote>
<p>The content and activity you are reporting appears to be related to
Hacktoberfest. Please keep in mind that GitHub Staff is not enforcing
Hacktoberfest rules; we will, however, enforce our own Acceptable Use
Policies. According to the Hacktoberfest FAQ&hellip; [same quote as given above]</p>
</blockquote>
<p>via <a href="https://twitter.com/kyleknighted/status/1311685461828612097">@kyleknighted@twitter.com</a></p>
<p>So, according to these two companies, whose responsibility is it to deal with
the spam that <em>they&rsquo;ve</em> created? The maintainers, of course! All for a T-Shirt.</p>
<p>Let&rsquo;s be honest. Hacktoberfest has never generated anything of value for open
source. It&rsquo;s a marketing stunt which sends a deluge of low-effort contributions
to maintainers, leaving them to clean up the spam. I&rsquo;ve never been impressed
with Hacktoberfest contributions, even the ones which aren&rsquo;t obviously written
by a bot:</p>
<p><a href="https://github.com/whatwg/html/pull/5975/files"><img src="https://l.sr.ht/F-sU.png" alt="Screenshot of a pull request which needlessly comment a CSS file"></a></p>
<p>Hacktoberfest is, and has always been, about one thing: marketing for Digital
Ocean.</p>
<iframe
src="https://oc.todon.fr/@val/104960502585461740/embed"
class="mastodon-embed"
style="max-width: 100%; border: 0; margin: 0 auto; display: block;"
width="400"
height="530"
allowfullscreen="allowfullscreen"></iframe>
<p>This is what we get with corporate-sponsored &ldquo;social coding&rdquo;, brought to you by
Digital Ocean and GitHub and McDonalds, home of the Big Mac™. When you
build the Facebook of coding, you get the Facebook of coding. We don&rsquo;t need to
give away T-Shirts to incentivize drive-by drivel from randoms who will never
get any closer to open source than a +1/-1 README.md change.</p>
<p>What would <em>actually</em> benefit FOSS is to enable the strong mentorship necessary
raise a new generation of <strong>software engineers</strong> under the tutelage of
maintainers who can rely on a strong support system to do their work. Programs
like Google Summer of Code do this better. Programs where a marketing department
spends $5,000 on T-Shirts to flood maintainers with garbage and clothe people in
ads are doing the opposite: <em>hurting</em> open source.</p>
<p><a href="https://l.sr.ht/KoFK.png"><img src="https://l.sr.ht/KoFK.png" alt="Screenshot of a friend&rsquo;s notifications, 9 out of 11 of which are spam"></a></p>
<p>Check out <a href="https://twitter.com/shitoberfest">@shitoberfest on Twitter</a> for more
Hacktoberfest garbage.</p>
<p><strong>Update 2020-10-03</strong>: Digital Ocean
<a href="https://hacktoberfest.digitalocean.com/hacktoberfest-update">has updated their rules</a>,
among other things asking maintainers to opt-in, to reduce spam.</p>


guid: https://drewdevault.com/2020/10/09/Four-principles-of-software-engineering.html
title: Four principles of software engineering
url: https://drewdevault.com/2020/10/09/Four-principles-of-software-engineering.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Software should be <strong>robust</strong>. It should be designed to accommodate all known
edge cases. In practice, this means predicting and handling all known error
cases, enumerating and addressing all classes of user inputs, reasoning about
and planning for the performance characteristics of your program, and so on.</p>
<p>Software should be <strong>reliable</strong>. It should be expected to work for an extended
length of time under design conditions without failures. Ideally, it should work
outside of design conditions up to some threshold.</p>
<p>Software should also be <strong>stable</strong>. It should not change in incompatible or
unexpected ways; if it works today it should also work tomorrow. If it has to
change, a plan shall be written. Stakeholders (including users!) should be given
advance notice and should be involved in the planning stage.</p>
<p>Finally, software should be <strong>simple</strong>. Only as many moving parts should be
included as necessary to meet the other three goals. All software has bugs, but
complicated software (1) has more bugs and (2) is more difficult to diagnose
and fix. Note that designing a simple solution is usually more difficult than
designing a complex solution.</p>
<p><em>This (short) article is based on <a href="https://cmpwn.com/@sir/104931806273081351">a Mastodon post</a> I wrote a few weeks ago.</em></p>


guid: https://drewdevault.com/2020/10/15/Status-update-October-2020.html
title: Status update, October 2020
url: https://drewdevault.com/2020/10/15/Status-update-October-2020.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I&rsquo;m writing this month&rsquo;s status update from a brand-new desktop workstation
(well, I re-used the GPU), my first new workstation in about 10 years. I hope
this new one lasts for another decade! I aimed for something smaller and
lightweight this time — it&rsquo;s a Mini-ITX build. I&rsquo;ve only been running this
for a few days, so let me tell you about the last few accomplishments which are
accountable to my venerable workstation&rsquo;s final days of life.</p>
<p>First, there&rsquo;s been a ton of important work completed for SourceHut&rsquo;s API 2.0
plans. All of the main blockers for the first version of meta.sr.ht&rsquo;s writable
GraphQL API are resolved, and after implementing a few more resolvers it should
be in a shippable state. This included riggings for database transactions,
simplification of the mini-&ldquo;ORM&rdquo; I built, and support for asyncronous work like
delivering webhooks. The latter called for a new library, <a href="https://sr.ht/~sircmpwn/dowork/">dowork</a>,
which you&rsquo;re free to reuse to bring asyncronous work processing to your Go
programs.</p>
<p>I also built a new general-purpose daemon for SourceHut called
<a href="https://sr.ht/~sircmpwn/chartsrv/">chartsrv</a>, which can be used to generate graphs from
<a href="https://prometheus.io/">Prometheus</a> data. The following is a real-time graph of the load
average on the builds.sr.ht workers:</p>
<p><img src="https://metrics.sr.ht/chart.svg?title=Build%20worker%20load%20average&amp;query=avg_over_time%28node_load15%7Binstance%3D~%22cirno%5B0-9%5D%2B.sr.ht%3A80%22%7D%5B1h%5D%29&amp;max=64&amp;since=336h&amp;stacked&amp;step=10000&amp;height=3&amp;width=10" alt="A chart which hopefully shows a reasonable load average across all workers"></p>
<p>I&rsquo;ve been getting more into <a href="https://gemini.circumlunar.space/">Gemini</a> this month, and have completed
three (or four?) whole projects for it:</p>
<ul>
<li><a href="https://sr.ht/~sircmpwn/gmni/">gmni</a> and gmnlm: a client implementation and line-mode browser</li>
<li><a href="https://sr.ht/~sircmpwn/gmnisrv/">gmnisrv</a>: a server implementation</li>
<li><a href="https://sr.ht/~sircmpwn/kineto/">kineto</a>: an HTTP-&gt;Gemini portal</li>
</ul>
<p>The (arguably) fourth project is the completion of a Gemini version of this
blog, which is available at <code>gemini://drewdevault.com</code>, or via the kineto portal
at <a href="https://portal.drewdevault.com">portal.drewdevault.com</a>. I&rsquo;ll be posting
some content exclusively on Gemini (and I already have!), so get yourself a
client if you want to tune in.</p>
<p>I have also invested some effort into <a href="https://git.sr.ht/~sircmpwn/himitsu">himitsu</a>, a project I shelved
for so long that you probably don&rsquo;t remember it. Worry not, I have rewritten the
README.md to give you a better introduction to it. Here&rsquo;s a screenshot for your
viewing pleasure:</p>
<p><img src="https://l.sr.ht/hr4G.png" alt="A GUI dialog asking a user to consent to allow an application to access their IMAP credentials"></p>
<p>Bonus update: two new <a href="https://baremessages.org">BARE</a> implementations have
appeared: OCaml and Java.</p>
<p>That&rsquo;s all for now! I&rsquo;ll see you for the next update soon. Thanks for your
support!</p>
<details>
  <summary>...</summary>
  <img src="https://l.sr.ht/y15d.png" alt="A screenshot of a page of a programming language specification detailing the syntax of tagged unions" />
</details>


guid: https://drewdevault.com/2020/10/22/Firefox-the-embarassment-of-FOSS.html
title: Firefox: The Jewel^WEmbarassment of Open Source
url: https://drewdevault.com/2020/10/22/Firefox-the-embarassment-of-FOSS.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Circa 2006, the consensus on Firefox was concisely stated by this classic xkcd:</p>
<p><a href="https://xkcd.com/198/"><img src="https://imgs.xkcd.com/comics/perspective.png" alt="A stick-figure comic. The title reads &ldquo;Sometimes, when I first wake up, I am caught in the horrible grip of perspective.&rdquo; The character, waking up, says &ldquo;It may be a jewel of open source, but Firefox is just a browser. It shows webpages. What the hell is wrong with us?&rdquo; The caption reads &ldquo;Fortunately, this subsides quickly.&rdquo;"></a></p>
<p>This feeling didn&rsquo;t last. In 2016, I wrote
<a href="https://drewdevault.com/2016/05/11/In-Memoriam-Mozilla.html">In Memoriam - Mozilla</a>, and in 2017,
<a href="https://drewdevault.com/2017/12/16/Firefox-is-on-a-slippery-slope.html">Firefox is on a slippery slope</a>.
Well, I was right, and Firefox (and Mozilla) have only become worse since. The
fuck-up culture is so ingrained in Mozilla in 2020 that it&rsquo;s hard to see it ever
getting better again.</p>
<p>In the time since my last article on the subject, Mozilla has:</p>
<ul>
<li>Laid off 25% of its employees, mostly engineers, many of whom work on Firefox<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></li>
<li>Raised executive pay 400% as their market share declined 85%<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup></li>
<li>Sent a record of all browsing traffic to CloudFlare by default<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup></li>
<li>Added advertisements to the new tab page on Firefox<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup></li>
<li>Used their brand to enter the saturated VPN grift market<sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup></li>
<li>Built a walled garden for add-ons, then let the walls crash in<sup id="fnref:6"><a href="#fn:6" class="footnote-ref" role="doc-noteref">6</a></sup></li>
<li>Started, and killed, a dozen projects which were not Firefox<sup id="fnref:7"><a href="#fn:7" class="footnote-ref" role="doc-noteref">7</a></sup></li>
</ul>
<p>The most interesting things they&rsquo;ve been involved in in the past few years are
Rust and Servo, and they fired most or all of their engineers involved in both.
And, yesterday, <a href="https://blog.mozilla.org/blog/2020/10/20/mozilla-reaction-to-u-s-v-google/">Mozilla published a statement</a> siding with
Google on anti-trust, failing to disclose the fact that Google pays to keep their
lights on.</p>
<p>Is this the jewel of open source? No, not anymore. Firefox is the embarrassment
of open source, and it&rsquo;s the only thing standing between Google and an
all-encompassing monopoly over the web. Mozilla has divested from Firefox and
started funnelling what money is left out of their engineering payroll and into
their executive pockets. The web is dead, and its fetid corpse persists only
as the layer of goop that Google scrapes between its servers and your screen.
Anyone who still believes that Mozilla will save the web is a fool.</p>
<p>As I have <a href="https://drewdevault.com/2020/03/18/Reckless-limitless-scope.html">stated before</a>, the scope of web browsers has been
increasing at a reckless pace for <em>years</em>, to the point where it&rsquo;s literally
impossible to build a new web browser. We have no recourse left to preserve the
web. This is why I&rsquo;m throwing my weight behind <a href="https://gemini.circumlunar.space/">Gemini</a>, a new protocol
which is <em>much simpler</em> than the web, and which you can implement yourself in a
weekend.</p>
<p>Forget about the web, it&rsquo;s a lost cause. Let&rsquo;s move on.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p><a href="https://arstechnica.com/information-technology/2020/08/firefox-maker-mozilla-lays-off-250-workers-says-covid-19-lowered-revenue/">Mozilla cuts 250 jobs, says Firefox development will be affected</a> <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p><a href="http://calpaterson.com/mozilla.html">Firefox usage is down 85% despite Mozilla&rsquo;s top exec pay going up 400%</a> <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p><a href="https://blog.mozilla.org/blog/2020/02/25/firefox-continues-push-to-bring-dns-over-https-by-default-for-us-users/">Firefox continues push to bring DNS over HTTPS by default for US users</a> <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p><a href="https://blog.mozilla.org/futurereleases/2018/04/30/a-privacy-conscious-approach-to-sponsored-content/">A Privacy-Conscious Approach to Sponsored Content</a> <a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p><a href="https://vpn.mozilla.org/">Mozilla VPN</a> <a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p><a href="https://hacks.mozilla.org/2019/05/technical-details-on-the-recent-firefox-add-on-outage/">Technical Details on the Recent Firefox Add-on Outage</a> <a href="#fnref:6" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p><a href="https://killedbymozilla.com/">Killed by Mozilla</a> <a href="#fnref:7" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/10/23/Im-handing-wlroots-and-sway-to-Simon.html
title: I'm handing over maintenance of wlroots and sway to Simon Ser
url: https://drewdevault.com/2020/10/23/Im-handing-wlroots-and-sway-to-Simon.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Over the past several months, I&rsquo;ve been gradually weaning down my role in both
projects, and as a contributor to Wayland in general. I feel that I&rsquo;ve already
accomplished everything I set out to do with Wayland — and more! I have
been happily using sway as my daily driver for well over a year with no
complaints or conspicuously absent features. For me, there&rsquo;s little reason to
stay involved. This will likely come as no surprise to many who&rsquo;ve kept their
ear to the ground in these communities.</p>
<p>Simon has been an important co-maintainer on wlroots and sway for several years,
and also serves as a maintainer for Wayland itself, and Weston. I trust him with
these projects, and he&rsquo;s been doing a stellar job so far — no real change
in his work is necessary for this hand-off. Simon works for SourceHut full-time
and his compensation covers his role in the Wayland community, so you can trust
that the health of the project is unaffected, too.</p>
<p>There&rsquo;s still plenty of great things to come from these projects without me.
Many improvements are underway and more are planned for the future. Don&rsquo;t worry:
sway and wlroots have already demonstrated that they work quite well without my
active involvement.</p>
<p>Good luck, Simon, and thanks for all of your hard work! I&rsquo;m proud of you!</p>


guid: https://drewdevault.com/2020/11/01/What-is-Gemini-anyway.html
title: What is this Gemini thing anyway, and why am I excited about it?
url: https://drewdevault.com/2020/11/01/What-is-Gemini-anyway.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I&rsquo;ve been writing about some specific topics in the realm of Gemini on my blog
over the past two months or so, but I still haven&rsquo;t written a broader
introduction to Gemini, what I&rsquo;m doing with it, and why you should be excited
about it, too. Let&rsquo;s do that today!</p>
<p><a href="https://gemini.circumlunar.space/">Gemini</a> is a network protocol for exchanging
hypertext documents — &ldquo;hypertext&rdquo; in the general sense of the word, not
with respect to the hypertext markup language (HTML) that web browsers
understand. It&rsquo;s a simple network protocol which allows clients to request
hypertext documents (in its own document format, gemtext). It is, in some
respects, an evolution of <a href="https://en.wikipedia.org/wiki/Gopher_(protocol)">Gopher</a>, but more modernized and streamlined.</p>
<p>Gemini is very simple. The protocol uses TLS to establish an encrypted
connection (using self-signed certificates and
<abbr title="trust on first use">TOFU</abbr> rather than certificate
authorities), and performs a very simple exchange: the client sends the URL it
wants to retrieve, terminated with CRLF.  The server responds with an
informative line, consisting of a numeric status code and some additional
information (such as the document&rsquo;s mimetype), then writes the document and
closes the connection. Authentication, if desired, is done with client
certificates. User input, if desired, is done with a response code which conveys
a prompt string and a request for user input, followed by a second request with
the user&rsquo;s response filled into the URL&rsquo;s query string. And that&rsquo;s pretty much
it!</p>
<pre><code>$ openssl s_client -quiet -crlf   \
    -servername drewdevault.com   \
    -connect drewdevault.com:1965 \
  | awk '{ print &quot;response: &quot; $0 }'
gemini://drewdevault.com
response: 20 text/gemini
response: ```ASCII art of a rocket next to &quot;Drew DeVault&quot; in a stylized font
response:   /\
response:   ||    ________                         ________       ____   ____            .__   __
response:   ||    \______ \_______   ______  _  __ \______ \   ___\   \ /   /____   __ __|  |_/  |_
response:  /||\    |    |  \_  __ \_/ __ \ \/ \/ /  |    |  \_/ __ \   Y   /\__  \ |  |  \  |\   __\
response: /:||:\   |    `   \  | \/\  ___/\     /   |    `   \  ___/\     /  / __ \|  |  /  |_|  |
response: |:||:|  /_______  /__|    \___  &gt;\/\_/   /_______  /\___  &gt;\___/  (____  /____/|____/__|
response: |/||\|        \/            \/                 \/     \/             \/
response:   **
response:   **
response: ```
[...]
</code></pre><p>So why am I excited about it?</p>
<p>My disdain for web browsers is well documented<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. Web browsers are
<em>extraordinarily</em> complex, and any attempt to build a new one would be a
Sisyphean task. Successfully completing that implementation, if even possible,
would necessarily produce a Lovecraftian mess: unmaintainable, full of security
vulnerabilities, with gigabytes in RAM use and hours in compile times. And given
that all of the contemporary web browsers that implement a sufficiently useful
subset of web standards are ass and getting assier, what should we do?</p>
<p>The problem is unsolvable. We cannot have the &ldquo;web&rdquo; without all of these
problems. But what we can have is something different, like Gemini. Gemini does
not solve all of the web&rsquo;s problems, but it addresses a subset of its use-cases
better than the web does, and that excites me. I want to discard the parts of
the web that Gemini does better, and explore other solutions for anything that&rsquo;s
left of the web which is worth keeping (hint: much of it is not).</p>
<p>There are some aspects of Gemini which I approve of immensely:</p>
<ul>
<li>It&rsquo;s dead simple. A client or server implementation can be written from
scratch by a single person in the space of an afternoon or two. A new web
browser could take hundreds of engineers millions of hours to complete.</li>
<li>It&rsquo;s not extensible. Gemini is designed to be <em>difficult</em> to extend without
breaking backwards compatibility, and almost all proposals for expansion on
the mailing list are ultimately shot down. This is a good thing: extensibility
is generally a bad idea. Extensions ultimately lead to more complexity and
Gemini might suffer the same fate as the web if not for its disdain for
extensions.</li>
<li>It&rsquo;s opinionated about document formatting. There are no inline links (every
link goes on its own line), no formatting, and no inline images. Gemini
strictly separates the responsibility of content and presentation. Providing
the content is the exclusive role of the server, and providing the
presentation is the exclusive role of the client. There are no stylesheets and
authors have very little say in <em>how</em> their content is presented. It&rsquo;s still
possible for authors to express themselves within these constraints —
as with any other constraints — but it allows clients to be simpler and
act more as <em>user</em> agents than <em>vendor</em> agents.</li>
</ul>
<p>Some people argue that what we should have is &ldquo;the web, but less of it&rdquo;, i.e. a
&ldquo;sane&rdquo; subset of web standards. I don&rsquo;t agree (for one, I don&rsquo;t think there <em>is</em>
a &ldquo;sane&rdquo; subset of those standards), but I&rsquo;ll save that for another blog post.
Gemini is a new medium, and it&rsquo;s different from the web. Anyone who checking it
out should be prepared for that and open to working within its constraints.
Limitations breed creativity!</p>
<p>For my part, I have been working on a number of Gemini projects. For one, this
blog is now available <a href="gemini://drewdevault.com">on Gemini</a>, and I have started
writing some Gemini-exclusive content for it. I&rsquo;ve also written some software
you&rsquo;re welcome to use:</p>
<p><a href="https://sr.ht/~sircmpwn/gmni"><strong>libgmni</strong></a>,
<a href="https://sr.ht/~sircmpwn/gmni"><strong>gmni</strong></a>,
and <a href="https://sr.ht/~sircmpwn/gmni"><strong>gmnlm</strong></a> are my suite of Gemini client
software, all written in C11 and only depending on a POSIX-like system and
OpenSSL. libgmni is a general-purpose Gemini client library with <a href="https://git.sr.ht/~sircmpwn/gmni/tree/master/include/gmni">a simple
interface</a>. gmni is a
cURL-like <a href="https://drewdevault.com/gmni.1.html">command line tool</a> for
performing Gemini requests. Finally, gmnlm is a line-mode browser with a rich
feature-set. Together these tools weigh just under 4,000 lines of code, of which
about 1,600 are the URL parser from cURL vendored in.</p>
<p><a href="https://portal.drewdevault.com/gmnisrv.gmi"><strong>gmnisrv</strong></a> is a high-performance
Gemini server, also written in C11 for POSIX systems with OpenSSL. It supports
zero-configuration TLS, CGI scripting, auto-indexing, regex routing and URL
rewrites, and I have a couple more things planned for 1.0. It clocks in at about
6,700 lines, of which the same 1,600 are vendored from cURL, and an additional
2,800 lines are vendored from Fabrice Bellard&rsquo;s
<a href="https://bellard.org/quickjs/">quickjs</a> regex implementation.</p>
<p><a href="https://portal.drewdevault.com/kineto.gmi"><strong>kineto</strong></a> is an HTTP-to-Gemini
gateway, implemented as a single Go file (under 500 lines) with the assistance
of ~adnano&rsquo;s <a href="https://sr.ht/~adnano/go-gemini/">go-gemini</a> library. My Gemini
blog <a href="https://portal.drewdevault.com">is available through this portal</a> if you
would like to browse it.</p>
<p>So dive in and explore! Install gmnisrv on your server and set up a Gemini space
for yourself. Read the feeds from
<a href="gemini://gemini.circumlunar.space/capcom/">CAPCOM</a>. Write some software of your
own!</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p><a href="https://drewdevault.com/2020/08/13/Web-browsers-need-to-stop.html">Exhibit A</a>, <a href="https://drewdevault.com/2020/03/18/Reckless-limitless-scope.html">Exhibit B</a>, <a href="https://cmpwn.com/@sir/104894723861368333">Exhibit C</a> <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/11/06/Utility-vs-usability.html
title: Utility vs usability
url: https://drewdevault.com/2020/11/06/Utility-vs-usability.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>In many fields, professional-grade tooling requires a high degree of knowledge
and training to use properly, usually more than is available to the amateur. The
typical mechanic&rsquo;s tool chest makes my (rather well-stocked, in my opinion) tool
bag look quite silly. A racecar driver is using a vehicle which is much more
complex than, say, the soccer mom&rsquo;s mini-van. Professional-grade tools are,
necessarily, more complex and require skill to use.</p>
<p>There are two attributes to consider when classifying these tools: <em>utility</em> and
<em>usability</em>. These are not the same thing. Some tools have both high utility and
high usability, such as a pencil. Some are highly usable, but of low utility,
such as a child&rsquo;s tricycle. Tools of both low-utility and low-usability are
uncommon, but I&rsquo;m sure you can think of a few examples from your own experiences
:)</p>
<p>When designing tools, it is important to consider both of these attributes, and
it helps to keep the intended audience in mind. I think that many programmers
today are overly concerned with usability, and insufficiently concerned with
utility. Some programmers (although this sort prefers &ldquo;developer&rdquo;) go so far as
to fetishize usability <em>at the expense</em> of utility.</p>
<p>In some cases, sacrificing utility in favor of usability is an acceptable
trade-off. In the earlier example&rsquo;s case, it&rsquo;s unlikely that anyone would argue
that the soccer mom should be loading the tots into an F1 racecar. However, it&rsquo;s
equally absurd to suppose that the F1 driver should bring a mini-van to the race
track. In the realm of programming, this metaphor speaks most strongly to me in
the design of programming tools.</p>
<p>I argue that most programmers are professionals who are going to invest several
years into learning the craft. This is the audience for whom I design my tools.
What trouble is it to spend an extra hour learning a somewhat less intuitive
<a href="https://git-send-email.io">code review tool</a> when the programming language
whose code you&rsquo;re reviewing required months to learn and years to master?</p>
<p><a href="https://xkcd.com/1205/"><img src="https://imgs.xkcd.com/comics/is_it_worth_the_time_2x.png" alt="An XKCD comic which depicts a table that compares &ldquo;time to improve task&rdquo; versus &ldquo;time saved by improvement&rdquo; and how to figure if the improvement is worthwhile"></a></p>
<p>I write tools to maximize the productivity of professional programmers. Ideally,
we can achieve both usability and utility, and often we do just that.
But, sometimes, these tools require a steeper learning curve. If they are <em>more
useful</em> in spite of that, they will usually save heaps of time in the long run.</p>
<p>Instead of focusing on dumbing down our tools, maximizing usability at the
expense of utility, we should focus on making powerful tools <em>and</em> fostering a
culture of <em>mentorship</em>. Senior engineers should be helping their juniors learn
and grow to embrace and build a new generation of more and more productive
tooling, considering usability all the while but never at the expense of
utility.</p>
<p>I&rsquo;ll address mentorship in more detail in future posts. For now, I&rsquo;ll just state
that mentorship is the praxis of my tooling philosophy. We can build better,
more powerful, and more productive tools, even if they require a steeper
learning curve, so long as we&rsquo;re prepared to teach people how to use them, and
they&rsquo;re prepared to learn.</p>


guid: https://drewdevault.com/2020/11/15/Status-update-November-2020.html
title: Status update, November 2020
url: https://drewdevault.com/2020/11/15/Status-update-November-2020.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Greetings, humanoids! Our fleshy vessels have aged by 2.678×10⁶ seconds, and you
know what that means: time for another status update! Pour a cup of your
favorite beverage stimulant and gather &lsquo;round for some news.</p>
<p>First off, today is the second anniversary of SourceHut&rsquo;s alpha being opened to
the public, and as such, I&rsquo;ve prepared a special <a href="https://sourcehut.org/blog/2020-11-15-sourcehut-2-year-alpha/">blog post</a> for
you to read. I&rsquo;ll leave the sr.ht details out of this post and just send you off
to read about it there.</p>
<p>What else is new? Well, a few things. For one, I&rsquo;ve been working more on Gemini.
I added CGI support to <a href="https://sr.ht/~sircmpwn/gmnisrv">gmnisrv</a> and wrote a
few <a href="https://git.sr.ht/~sircmpwn/cgi-scripts">CGI scripts</a> to do neato Gemini
things with. I&rsquo;ve also added regexp routing and URL rewriting support. We can
probably ship gmnisrv 1.0 as soon as the last few bugs are flushed out, and a
couple of minor features are added, and we might switch to another SSL
implementation as well. Thanks to the many contributors who&rsquo;ve helped out:
William Casarin, Tom Lebreux, Kenny Levinsen, Eyal Sawady, René Wagner,
dbandstra, and mbays.</p>
<p>In <a href="https://baremessages.org">BARE</a> news: Elm, Erlang, Java, and Ruby
implementations have appeared, and I have submitted a
<a href="https://datatracker.ietf.org/doc/draft-devault-bare/">draft RFC</a> to the IETF
for standardization.</p>
<p>Finally, I wrote a new Wayland server for you. Its only dependencies are a POSIX
system and a C11 compiler — and it works with Nvidia GPUs, or even systems
without OpenGL support at all. Here&rsquo;s the code:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="cp">#include</span> <span class="cpf">&lt;poll.h&gt;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;signal.h&gt;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;stdint.h&gt;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;stdio.h&gt;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;stdlib.h&gt;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;string.h&gt;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;sys/ioctl.h&gt;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;sys/mman.h&gt;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;sys/socket.h&gt;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;sys/un.h&gt;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;time.h&gt;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;unistd.h&gt;</span><span class="cp">
</span><span class="cp"></span>
	<span class="k">typedef</span> <span class="n">int16_t</span> <span class="n">i16</span><span class="p">;</span> <span class="k">typedef</span> <span class="n">int32_t</span> <span class="n">i32</span><span class="p">;</span> <span class="k">typedef</span> <span class="n">uint16_t</span> <span class="n">u16</span><span class="p">;</span>
   <span class="k">typedef</span> <span class="n">uint32_t</span> <span class="n">u32</span><span class="p">;</span> <span class="k">typedef</span> <span class="n">uint8_t</span> <span class="n">u8</span><span class="p">;</span> <span class="k">typedef</span> <span class="kt">int</span> <span class="n">I</span><span class="p">;</span> <span class="k">typedef</span> <span class="n">size_t</span> <span class="n">S</span><span class="p">;</span>
	  <span class="k">typedef</span> <span class="k">struct</span><span class="p">{</span><span class="kt">int</span> <span class="n">b</span><span class="p">;</span><span class="n">u32</span> <span class="n">a</span><span class="p">,</span><span class="n">c</span><span class="p">,</span><span class="n">d</span><span class="p">;</span><span class="kt">char</span><span class="o">*</span><span class="n">e</span><span class="p">;</span><span class="n">i32</span> <span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">,</span><span class="n">w</span><span class="p">,</span><span class="n">h</span><span class="p">,</span><span class="n">s</span><span class="p">,</span><span class="n">fmt</span><span class="p">;}</span><span class="n">O</span><span class="p">;</span>
	       <span class="k">typedef</span> <span class="k">struct</span><span class="p">{</span><span class="kt">char</span><span class="o">*</span><span class="n">a</span><span class="p">;</span><span class="n">u32</span> <span class="n">b</span><span class="p">;</span><span class="n">I</span><span class="p">(</span><span class="o">*</span><span class="n">c</span><span class="p">)(</span><span class="n">I</span><span class="p">,</span><span class="n">I</span><span class="p">,</span><span class="n">u32</span><span class="p">,</span><span class="n">u16</span><span class="p">);}</span><span class="n">G</span><span class="p">;</span>

	  <span class="k">struct</span> <span class="n">pollfd</span> <span class="n">fds</span><span class="p">[</span><span class="mi">33</span><span class="p">];</span><span class="kt">char</span> <span class="n">a</span><span class="p">[</span><span class="mh">0xFFFF</span><span class="p">];</span><span class="n">I</span> <span class="n">b</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span><span class="n">c</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span><span class="n">d</span><span class="o">=-</span><span class="mi">1</span><span class="p">;</span><span class="n">u32</span> <span class="n">e</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span>
			      <span class="n">O</span> <span class="n">f</span><span class="p">[</span><span class="mi">128</span><span class="p">][</span><span class="mi">128</span><span class="p">];</span><span class="n">G</span> <span class="n">g</span><span class="p">[];</span>

<span class="cp">#define AR I z,I y,u32 x,u16 c
</span><span class="cp">#define SN(n) (((n)%4)==0?(n):(n)+(4-((n)%4)))
</span><span class="cp"></span>
<span class="kt">void</span> <span class="nf">u8w</span><span class="p">(</span><span class="n">FILE</span><span class="o">*</span><span class="n">f</span><span class="p">,</span><span class="n">u32</span> <span class="n">ch</span><span class="p">){</span><span class="kt">char</span> <span class="n">b</span><span class="p">[</span><span class="mi">4</span><span class="p">];</span><span class="k">for</span> <span class="p">(</span><span class="n">I</span> <span class="n">i</span><span class="o">=</span><span class="mi">2</span><span class="p">;</span><span class="n">i</span><span class="o">&gt;</span><span class="mi">0</span><span class="p">;</span><span class="o">--</span><span class="n">i</span><span class="p">){</span><span class="n">b</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">=</span><span class="p">(</span><span class="n">ch</span><span class="o">&amp;</span><span class="mh">0x3f</span><span class="p">)</span><span class="o">|</span><span class="mh">0x80</span><span class="p">;</span><span class="n">ch</span><span class="o">&gt;&gt;=</span><span class="mi">6</span>
<span class="p">;}</span><span class="n">b</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">=</span><span class="n">ch</span><span class="o">|</span><span class="mh">0xE0</span><span class="p">;</span><span class="n">fwrite</span><span class="p">(</span><span class="n">b</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="n">f</span><span class="p">);}</span><span class="kt">void</span> <span class="nf">xrgb</span><span class="p">(</span><span class="n">u32</span><span class="o">*</span><span class="n">data</span><span class="p">,</span><span class="n">i32</span> <span class="n">w</span><span class="p">,</span><span class="n">i32</span> <span class="n">h</span><span class="p">,</span><span class="n">i32</span> <span class="n">s</span><span class="p">){</span><span class="k">struct</span>
<span class="n">winsize</span> <span class="n">sz</span><span class="p">;</span><span class="n">ioctl</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="n">TIOCGWINSZ</span><span class="p">,</span><span class="o">&amp;</span><span class="n">sz</span><span class="p">);</span><span class="o">--</span><span class="n">sz</span><span class="p">.</span><span class="n">ws_row</span><span class="p">;</span><span class="n">printf</span><span class="p">(</span><span class="s">&#34;</span><span class="se">\x1b</span><span class="s">[H</span><span class="se">\x1b</span><span class="s">[2J</span><span class="se">\x1b</span><span class="s">[3J&#34;</span><span class="p">);</span>
<span class="k">for</span><span class="p">(</span><span class="n">I</span> <span class="n">y</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="n">y</span><span class="o">&lt;</span><span class="n">sz</span><span class="p">.</span><span class="n">ws_row</span><span class="p">;</span><span class="o">++</span><span class="n">y</span><span class="p">){</span><span class="k">for</span><span class="p">(</span><span class="n">I</span> <span class="n">x</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="n">x</span><span class="o">&lt;</span><span class="n">sz</span><span class="p">.</span><span class="n">ws_col</span><span class="p">;</span><span class="o">++</span><span class="n">x</span><span class="p">){</span><span class="n">I</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="n">u32</span> <span class="n">c</span> <span class="o">=</span> <span class="mh">0x2800</span><span class="p">;</span><span class="k">const</span>
<span class="n">I</span> <span class="n">f</span><span class="p">[]</span><span class="o">=</span><span class="p">{</span><span class="mi">0</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">7</span><span class="p">};</span><span class="k">for</span><span class="p">(</span><span class="n">I</span> <span class="n">my</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="n">my</span><span class="o">&lt;</span><span class="mi">4</span><span class="p">;</span><span class="o">++</span><span class="n">my</span><span class="p">)</span><span class="k">for</span><span class="p">(</span><span class="n">I</span> <span class="n">mx</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="n">mx</span><span class="o">&lt;</span><span class="mi">2</span><span class="p">;</span><span class="o">++</span><span class="n">mx</span><span class="p">){</span><span class="n">u32</span> <span class="n">p</span><span class="o">=</span><span class="n">data</span><span class="p">[((</span>
<span class="n">y</span><span class="o">*</span><span class="mi">4</span><span class="o">+</span><span class="n">my</span><span class="p">)</span><span class="o">*</span><span class="n">h</span><span class="p">)</span><span class="o">/</span><span class="p">(</span><span class="n">sz</span><span class="p">.</span><span class="n">ws_row</span><span class="o">*</span><span class="mi">4</span><span class="p">)</span><span class="o">*</span><span class="p">(</span><span class="n">s</span><span class="o">/</span><span class="mi">4</span><span class="p">)</span><span class="o">+</span><span class="p">((</span><span class="n">x</span><span class="o">*</span><span class="mi">2</span><span class="o">+</span><span class="n">mx</span><span class="p">)</span><span class="o">*</span><span class="n">w</span><span class="p">)</span><span class="o">/</span><span class="p">(</span><span class="n">sz</span><span class="p">.</span><span class="n">ws_col</span><span class="o">*</span><span class="mi">2</span><span class="p">)];</span><span class="n">u8</span> <span class="n">avg</span><span class="o">=</span><span class="p">((</span><span class="n">p</span><span class="o">&amp;</span><span class="mh">0xFF</span><span class="p">)</span><span class="o">+</span><span class="p">((</span><span class="n">p</span>
<span class="o">&gt;&gt;</span><span class="mi">8</span><span class="p">)</span><span class="o">&amp;</span><span class="mh">0xFF</span><span class="p">)</span><span class="o">+</span><span class="p">((</span><span class="n">p</span><span class="o">&gt;&gt;</span><span class="mi">16</span><span class="p">)</span><span class="o">&amp;</span><span class="mh">0xFF</span><span class="p">))</span><span class="o">/</span><span class="mi">3</span><span class="p">;</span><span class="k">if</span><span class="p">(</span><span class="n">avg</span><span class="o">&gt;</span><span class="mh">0x80</span><span class="p">)</span><span class="n">c</span><span class="o">|=</span><span class="mi">1</span><span class="o">&lt;&lt;</span><span class="n">f</span><span class="p">[</span><span class="n">i</span><span class="o">++</span><span class="p">];}</span><span class="n">u8w</span><span class="p">(</span><span class="n">stdout</span><span class="p">,</span><span class="n">c</span><span class="p">);}</span><span class="n">putchar</span><span class="p">(</span>
<span class="sc">&#39;\n&#39;</span><span class="p">);}</span><span class="n">fflush</span><span class="p">(</span><span class="n">stdout</span><span class="p">);}</span><span class="n">O</span><span class="o">*</span><span class="nf">ao</span><span class="p">(</span><span class="n">I</span> <span class="n">z</span><span class="p">,</span><span class="n">u32</span> <span class="n">y</span><span class="p">,</span><span class="n">I</span> <span class="n">x</span><span class="p">){</span><span class="k">for</span><span class="p">(</span><span class="n">S</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="n">i</span><span class="o">&lt;</span><span class="mi">128</span><span class="p">;</span><span class="o">++</span><span class="n">i</span><span class="p">)</span><span class="k">if</span><span class="p">(</span><span class="n">f</span><span class="p">[</span><span class="n">z</span><span class="p">][</span><span class="n">i</span><span class="p">].</span><span class="n">a</span><span class="o">==</span><span class="mi">0</span><span class="p">){</span>
<span class="n">f</span><span class="p">[</span><span class="n">z</span><span class="p">][</span><span class="n">i</span><span class="p">].</span><span class="n">a</span><span class="o">=</span><span class="n">y</span><span class="p">;</span><span class="n">f</span><span class="p">[</span><span class="n">z</span><span class="p">][</span><span class="n">i</span><span class="p">].</span><span class="n">b</span><span class="o">=</span><span class="n">x</span><span class="p">;</span><span class="k">return</span> <span class="o">&amp;</span><span class="n">f</span><span class="p">[</span><span class="n">z</span><span class="p">][</span><span class="n">i</span><span class="p">];}</span><span class="k">return</span> <span class="mi">0</span><span class="p">;}</span><span class="n">O</span><span class="o">*</span><span class="nf">go</span><span class="p">(</span><span class="n">I</span> <span class="n">z</span><span class="p">,</span><span class="n">u32</span> <span class="n">y</span><span class="p">){</span><span class="k">for</span><span class="p">(</span><span class="n">S</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="n">i</span><span class="o">&lt;</span>
<span class="mi">128</span><span class="p">;</span><span class="o">++</span><span class="n">i</span><span class="p">)</span><span class="k">if</span><span class="p">(</span><span class="n">f</span><span class="p">[</span><span class="n">z</span><span class="p">][</span><span class="n">i</span><span class="p">].</span><span class="n">a</span><span class="o">==</span><span class="n">y</span><span class="p">)</span><span class="k">return</span> <span class="o">&amp;</span><span class="n">f</span><span class="p">[</span><span class="n">z</span><span class="p">][</span><span class="n">i</span><span class="p">];</span><span class="k">return</span> <span class="mi">0</span><span class="p">;}</span><span class="kt">void</span> <span class="nf">wh</span><span class="p">(</span><span class="n">I</span> <span class="n">z</span><span class="p">,</span><span class="n">i32</span> <span class="n">y</span><span class="p">,</span><span class="n">i16</span> <span class="n">x</span><span class="p">,</span> <span class="n">i16</span> <span class="n">w</span>
<span class="p">){</span><span class="n">write</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="o">&amp;</span><span class="n">y</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">i32</span> <span class="n">u</span><span class="o">=</span><span class="p">((</span><span class="n">w</span><span class="o">+</span><span class="mi">8</span><span class="p">)</span><span class="o">&lt;&lt;</span><span class="mi">16</span><span class="p">)</span><span class="o">|</span><span class="n">x</span><span class="p">;</span><span class="n">write</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="o">&amp;</span><span class="n">u</span><span class="p">,</span><span class="mi">4</span><span class="p">);}</span><span class="kt">void</span> <span class="nf">ws</span><span class="p">(</span><span class="n">I</span> <span class="n">z</span><span class="p">,</span><span class="kt">char</span><span class="o">*</span><span class="n">y</span><span class="p">){</span><span class="n">i32</span> <span class="n">l</span><span class="o">=</span>
<span class="n">strlen</span><span class="p">(</span><span class="n">y</span><span class="p">)</span><span class="o">+</span><span class="mi">1</span><span class="p">;</span><span class="n">write</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="o">&amp;</span><span class="n">l</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">l</span><span class="o">=</span><span class="n">SN</span><span class="p">(</span><span class="n">l</span><span class="p">);</span><span class="n">write</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">y</span><span class="p">,</span><span class="n">l</span><span class="p">);}</span><span class="n">I</span> <span class="nf">rs</span><span class="p">(</span><span class="n">I</span> <span class="n">z</span><span class="p">){</span><span class="n">u32</span> <span class="n">l</span><span class="p">;</span><span class="n">read</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="o">&amp;</span><span class="n">l</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">l</span><span class="o">=</span>
<span class="n">SN</span><span class="p">(</span><span class="n">l</span><span class="p">);</span><span class="n">read</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">a</span><span class="p">,</span><span class="n">l</span><span class="p">);</span><span class="k">return</span> <span class="n">l</span><span class="o">+</span><span class="mi">4</span><span class="p">;}</span><span class="n">I</span> <span class="nf">ga</span><span class="p">(</span><span class="n">I</span> <span class="n">z</span><span class="p">,</span><span class="n">I</span> <span class="n">y</span><span class="p">,</span><span class="n">u32</span> <span class="n">x</span><span class="p">,</span><span class="n">u16</span> <span class="n">w</span><span class="p">){</span><span class="n">u32</span> <span class="n">b</span><span class="p">,</span><span class="n">u</span><span class="p">,</span><span class="n">t</span><span class="p">,</span><span class="n">s</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">u</span><span class="p">,</span>
<span class="mi">4</span><span class="p">);</span><span class="n">I</span> <span class="n">sz</span><span class="o">=</span><span class="n">rs</span><span class="p">(</span><span class="n">y</span><span class="p">)</span><span class="o">+</span><span class="mi">12</span><span class="p">;</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">b</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">t</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="o">--</span><span class="n">u</span><span class="p">;</span><span class="n">ao</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">t</span><span class="p">,</span><span class="n">u</span><span class="p">);</span><span class="k">switch</span><span class="p">(</span><span class="n">u</span><span class="p">){</span><span class="k">case</span> <span class="mi">1</span><span class="o">:++</span><span class="n">s</span><span class="p">;</span><span class="n">wh</span>
<span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">t</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">write</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">s</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="o">--</span><span class="n">s</span><span class="p">;</span><span class="k">break</span><span class="p">;</span><span class="k">case</span> <span class="mi">6</span><span class="o">:</span><span class="n">s</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="k">break</span><span class="p">;</span><span class="k">default</span><span class="o">:</span><span class="k">return</span> <span class="n">sz</span><span class="p">;}</span><span class="n">wh</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">t</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">4</span>
<span class="p">);</span><span class="n">write</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">s</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="k">return</span> <span class="n">sz</span><span class="p">;}</span><span class="n">I</span> <span class="nf">gb</span><span class="p">(</span><span class="n">AR</span><span class="p">){</span><span class="n">u32</span> <span class="n">w</span><span class="p">,</span><span class="n">u</span><span class="p">;</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">I</span> <span class="n">t</span><span class="o">=</span><span class="n">d</span><span class="p">;</span><span class="n">d</span><span class="o">=-</span><span class="mi">1</span><span class="p">;</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">u</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span>
<span class="n">O</span> <span class="o">*</span><span class="n">o</span><span class="o">=</span><span class="n">ao</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">w</span><span class="p">,</span><span class="mi">15</span><span class="p">);</span><span class="n">o</span><span class="o">-&gt;</span><span class="n">e</span><span class="o">=</span><span class="n">mmap</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="n">u</span><span class="p">,</span><span class="n">PROT_READ</span><span class="p">,</span><span class="n">MAP_PRIVATE</span><span class="p">,</span><span class="n">t</span><span class="p">,</span><span class="mi">0</span><span class="p">);</span><span class="k">return</span> <span class="mi">8</span><span class="p">;}</span><span class="n">I</span> <span class="nf">gc</span><span class="p">(</span><span class="n">AR</span><span class="p">){</span><span class="n">u32</span>
<span class="n">w</span><span class="p">;</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">ao</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">w</span><span class="p">,</span><span class="mi">8</span><span class="o">+</span><span class="n">c</span><span class="p">);</span><span class="k">return</span> <span class="mi">4</span><span class="p">;}</span><span class="n">I</span> <span class="nf">gd</span><span class="p">(</span><span class="n">AR</span><span class="p">){</span><span class="n">O</span><span class="o">*</span><span class="n">o</span><span class="p">,</span><span class="o">*</span><span class="n">r</span><span class="p">,</span><span class="o">*</span><span class="n">t</span><span class="p">;</span><span class="n">i32</span> <span class="n">u</span><span class="p">,</span><span class="n">w</span><span class="p">;</span><span class="k">switch</span><span class="p">(</span><span class="n">c</span><span class="p">){</span><span class="k">case</span> <span class="mi">1</span>
<span class="o">:</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">u</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">go</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">x</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">d</span><span class="o">=</span><span class="n">u</span><span class="p">;</span><span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="n">u</span><span class="o">==</span><span class="mi">0</span><span class="p">){</span><span class="n">O</span> <span class="o">*</span><span class="n">b</span><span class="o">=</span><span class="n">go</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">u</span><span class="p">);</span><span class="n">xrgb</span>
<span class="p">((</span><span class="n">u32</span><span class="o">*</span><span class="p">)</span><span class="n">b</span><span class="o">-&gt;</span><span class="n">e</span><span class="p">,</span><span class="n">b</span><span class="o">-&gt;</span><span class="n">w</span><span class="p">,</span><span class="n">b</span><span class="o">-&gt;</span><span class="n">h</span><span class="p">,</span><span class="n">b</span><span class="o">-&gt;</span><span class="n">s</span><span class="p">);}</span><span class="n">wh</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">u</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">);</span><span class="k">return</span> <span class="mi">12</span><span class="p">;</span><span class="k">case</span> <span class="mi">2</span><span class="o">:</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">u</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="o">&amp;</span>
<span class="n">u</span><span class="p">,</span> <span class="mi">4</span><span class="p">);</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">u</span><span class="p">,</span> <span class="mi">4</span><span class="p">);</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">u</span><span class="p">,</span> <span class="mi">4</span><span class="p">);</span><span class="k">return</span> <span class="mi">16</span><span class="p">;</span><span class="k">case</span> <span class="mi">3</span><span class="o">:</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="k">struct</span> <span class="n">timespec</span>
<span class="n">ts</span><span class="o">=</span><span class="p">{.</span><span class="n">tv_sec</span><span class="o">=</span><span class="mi">0</span><span class="p">,.</span><span class="n">tv_nsec</span><span class="o">=</span><span class="mf">1.6e6</span><span class="p">};</span><span class="n">nanosleep</span><span class="p">(</span><span class="o">&amp;</span><span class="n">ts</span><span class="p">,</span><span class="mi">0</span><span class="p">);</span><span class="n">wh</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">w</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">write</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="k">return</span>
<span class="mi">4</span><span class="p">;</span><span class="k">case</span> <span class="mi">6</span><span class="o">:</span><span class="n">o</span><span class="o">=</span><span class="n">go</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">x</span><span class="p">);</span><span class="n">r</span><span class="o">=</span><span class="n">go</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">o</span><span class="o">-&gt;</span><span class="n">c</span><span class="p">);</span><span class="k">if</span><span class="p">(</span><span class="n">r</span><span class="o">&amp;&amp;</span><span class="n">r</span><span class="o">-&gt;</span><span class="n">b</span><span class="o">==</span><span class="mi">12</span><span class="p">){</span><span class="n">t</span><span class="o">=</span><span class="n">go</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">r</span><span class="o">-&gt;</span><span class="n">c</span><span class="p">);</span><span class="k">if</span><span class="p">(</span><span class="n">t</span><span class="o">-&gt;</span><span class="n">b</span><span class="o">==</span><span class="mi">13</span><span class="p">){</span><span class="n">u32</span>
<span class="n">s</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">wh</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">t</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">12</span><span class="p">);</span><span class="n">write</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">s</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">write</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">s</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">write</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">s</span><span class="p">,</span><span class="mi">4</span><span class="p">);}</span><span class="n">wh</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">r</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span>
<span class="n">write</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">e</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="o">++</span><span class="n">e</span><span class="p">;}</span><span class="k">break</span><span class="p">;}</span><span class="k">return</span> <span class="mi">0</span><span class="p">;}</span><span class="n">I</span> <span class="nf">ge</span><span class="p">(</span><span class="n">AR</span><span class="p">){</span><span class="n">u32</span> <span class="n">w</span><span class="p">,</span><span class="n">u</span><span class="p">;</span><span class="k">if</span><span class="p">(</span><span class="n">c</span><span class="o">==</span><span class="mi">2</span><span class="p">){</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">read</span><span class="p">(</span>
<span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">u</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">O</span><span class="o">*</span><span class="n">o</span><span class="o">=</span><span class="n">ao</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">w</span><span class="p">,</span><span class="mi">12</span><span class="p">);</span><span class="n">o</span><span class="o">-&gt;</span><span class="n">d</span><span class="o">=</span><span class="n">u</span><span class="p">;</span><span class="n">go</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">u</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">c</span><span class="o">=</span><span class="n">w</span><span class="p">;</span><span class="k">return</span> <span class="mi">8</span><span class="p">;}</span><span class="k">return</span> <span class="mi">0</span><span class="p">;}</span><span class="n">I</span> <span class="nf">gf</span><span class="p">(</span><span class="n">AR</span><span class="p">){</span><span class="n">u32</span> <span class="n">w</span><span class="p">,</span><span class="n">ae</span>
<span class="p">;</span><span class="k">switch</span><span class="p">(</span><span class="n">c</span><span class="p">){</span><span class="k">case</span> <span class="mi">1</span><span class="o">:</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">O</span><span class="o">*</span><span class="n">obj</span><span class="o">=</span><span class="n">ao</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">w</span><span class="p">,</span><span class="mi">13</span><span class="p">);</span><span class="n">obj</span><span class="o">-&gt;</span><span class="n">d</span><span class="o">=</span><span class="n">x</span><span class="p">;</span><span class="n">go</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">x</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">c</span><span class="o">=</span><span class="n">w</span><span class="p">;</span><span class="k">return</span> <span class="mi">4</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">4</span><span class="o">:</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">ae</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="k">return</span> <span class="mi">4</span><span class="p">;}</span><span class="k">return</span> <span class="mi">0</span><span class="p">;}</span><span class="n">I</span> <span class="nf">gg</span><span class="p">(</span><span class="n">AR</span><span class="p">){</span><span class="n">I</span> <span class="n">sz</span><span class="p">;</span><span class="k">switch</span><span class="p">(</span><span class="n">c</span><span class="p">){</span><span class="k">case</span> <span class="mi">2</span><span class="o">:</span><span class="n">sz</span><span class="o">=</span><span class="n">rs</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="p">;</span><span class="k">return</span> <span class="n">sz</span><span class="p">;}</span><span class="k">return</span> <span class="mi">0</span><span class="p">;}</span><span class="n">I</span> <span class="nf">gh</span><span class="p">(</span><span class="n">AR</span><span class="p">){</span><span class="n">i32</span> <span class="n">w</span><span class="p">,</span><span class="n">u</span><span class="p">;</span><span class="k">switch</span><span class="p">(</span><span class="n">c</span><span class="p">){</span><span class="k">case</span> <span class="mi">0</span><span class="o">:</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">u</span><span class="p">,</span><span class="mi">4</span>
<span class="p">);</span><span class="n">O</span><span class="o">*</span><span class="n">b</span><span class="o">=</span><span class="n">ao</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">w</span><span class="p">,</span><span class="mi">10</span><span class="p">);</span><span class="n">b</span><span class="o">-&gt;</span><span class="n">e</span><span class="o">=&amp;</span><span class="n">go</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">x</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">e</span><span class="p">[</span><span class="n">u</span><span class="p">];</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">b</span><span class="o">-&gt;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">b</span><span class="o">-&gt;</span><span class="n">h</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">b</span><span class="o">-&gt;</span>
                  <span class="n">s</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">b</span><span class="o">-&gt;</span><span class="n">fmt</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="k">return</span> <span class="mi">24</span><span class="p">;}</span><span class="k">return</span> <span class="mi">0</span><span class="p">;}</span>

<span class="n">G</span> <span class="n">g</span><span class="p">[]</span><span class="o">=</span><span class="p">{{</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="n">ga</span><span class="p">},{</span><span class="s">&#34;wl_shm&#34;</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="n">gb</span><span class="p">},{</span><span class="s">&#34;wl_compositor&#34;</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="n">gc</span><span class="p">},{</span><span class="s">&#34;wl_subcompositor&#34;</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">},</span>
<span class="p">{</span><span class="s">&#34;wl_data_device_manager&#34;</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">0</span><span class="p">},{</span><span class="s">&#34;wl_output&#34;</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">0</span><span class="p">},{</span><span class="s">&#34;wl_seat&#34;</span><span class="p">,</span><span class="mi">7</span><span class="p">,</span><span class="mi">0</span><span class="p">},{</span><span class="s">&#34;xdg_wm_base&#34;</span><span class="p">,</span>
<span class="mi">2</span><span class="p">,</span><span class="n">ge</span><span class="p">},{</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="n">gd</span><span class="p">},{</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">},{</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">},{</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">},{</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="n">gf</span><span class="p">},{</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="n">gg</span><span class="p">},{</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">},{</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="n">gh</span><span class="p">}};</span>

<span class="kt">void</span> <span class="nf">gi</span><span class="p">(</span><span class="n">AR</span><span class="p">){</span><span class="n">u32</span> <span class="n">w</span><span class="p">;</span><span class="k">switch</span><span class="p">(</span><span class="n">c</span><span class="p">){</span><span class="k">case</span> <span class="mi">0</span><span class="o">:</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">wh</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">w</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">write</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">1</span><span class="o">:</span><span class="n">read</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">w</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">ao</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="n">w</span><span class="p">,</span><span class="mi">0</span><span class="p">);</span><span class="k">for</span><span class="p">(</span><span class="n">S</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="n">i</span><span class="o">&lt;</span><span class="k">sizeof</span><span class="p">(</span><span class="n">g</span><span class="p">)</span><span class="o">/</span><span class="k">sizeof</span><span class="p">(</span><span class="n">g</span><span class="p">[</span><span class="mi">0</span><span class="p">]);){</span><span class="n">G</span><span class="o">*</span><span class="n">z</span><span class="o">=&amp;</span><span class="n">g</span><span class="p">[</span><span class="n">i</span><span class="o">++</span><span class="p">];</span>
<span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="n">z</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">)</span><span class="k">continue</span><span class="p">;</span><span class="n">I</span> <span class="n">gl</span><span class="o">=</span><span class="n">strlen</span><span class="p">(</span><span class="n">z</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">)</span><span class="o">+</span><span class="mi">1</span><span class="p">;</span><span class="n">wh</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">w</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">4</span><span class="o">+</span><span class="n">SN</span><span class="p">(</span><span class="n">gl</span><span class="p">)</span><span class="o">+</span><span class="mi">4</span><span class="o">+</span><span class="mi">4</span><span class="p">);</span><span class="n">write</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">i</span><span class="p">,</span><span class="mi">4</span><span class="p">);</span><span class="n">ws</span><span class="p">(</span><span class="n">y</span><span class="p">,</span>
                <span class="n">z</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">);</span><span class="n">write</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="o">&amp;</span><span class="n">z</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">,</span><span class="mi">4</span><span class="p">);}</span><span class="k">break</span><span class="p">;}}</span><span class="kt">void</span> <span class="nf">si</span><span class="p">(){</span><span class="n">c</span><span class="o">=</span><span class="mi">0</span><span class="p">;}</span>

<span class="n">I</span> <span class="nf">main</span><span class="p">(</span><span class="n">I</span> <span class="n">_1</span><span class="p">,</span><span class="kt">char</span><span class="o">**</span><span class="n">_2</span><span class="p">){</span><span class="n">I</span> <span class="n">z</span><span class="o">=</span><span class="n">socket</span><span class="p">(</span><span class="n">AF_UNIX</span><span class="p">,</span><span class="n">SOCK_STREAM</span><span class="p">,</span><span class="mi">0</span><span class="p">);</span><span class="k">struct</span> <span class="n">sockaddr_un</span> <span class="n">y</span><span class="o">=</span><span class="p">{.</span>
<span class="n">sun_family</span><span class="o">=</span><span class="n">AF_UNIX</span><span class="p">};</span><span class="kt">char</span> <span class="o">*</span><span class="n">x</span><span class="o">=</span><span class="n">getenv</span><span class="p">(</span><span class="s">&#34;XDG_RUNTIME_DIR&#34;</span><span class="p">);</span><span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="n">x</span><span class="p">)</span><span class="n">x</span><span class="o">=</span><span class="s">&#34;/tmp&#34;</span><span class="p">;</span><span class="k">do</span><span class="p">{</span><span class="n">sprintf</span><span class="p">(</span>
 <span class="n">y</span><span class="p">.</span><span class="n">sun_path</span><span class="p">,</span><span class="s">&#34;%s/wayland-%d&#34;</span><span class="p">,</span><span class="n">x</span><span class="p">,</span><span class="n">c</span><span class="o">++</span><span class="p">);}</span><span class="k">while</span><span class="p">(</span><span class="n">access</span><span class="p">(</span><span class="n">y</span><span class="p">.</span><span class="n">sun_path</span><span class="p">,</span><span class="n">F_OK</span><span class="p">)</span><span class="o">==</span><span class="mi">0</span><span class="p">);</span><span class="n">bind</span><span class="p">(</span><span class="n">z</span><span class="p">,</span> <span class="p">(</span>
<span class="k">struct</span> <span class="n">sockaddr</span> <span class="o">*</span><span class="p">)</span><span class="o">&amp;</span><span class="n">y</span><span class="p">,</span><span class="k">sizeof</span><span class="p">(</span><span class="n">y</span><span class="p">));</span><span class="n">listen</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="mi">3</span><span class="p">);</span><span class="k">for</span><span class="p">(</span><span class="n">S</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="n">i</span><span class="o">&lt;</span><span class="k">sizeof</span><span class="p">(</span><span class="n">fds</span><span class="p">);</span><span class="o">++</span><span class="n">i</span><span class="p">){</span><span class="n">fds</span><span class="p">[</span><span class="n">i</span><span class="p">].</span>
<span class="n">events</span><span class="o">=</span><span class="n">POLLIN</span><span class="o">|</span><span class="n">POLLHUP</span><span class="p">;</span><span class="n">fds</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">revents</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="n">fds</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">fd</span><span class="o">=</span><span class="mi">0</span><span class="p">;}</span><span class="n">fds</span><span class="p">[</span><span class="n">b</span><span class="o">++</span><span class="p">].</span><span class="n">fd</span><span class="o">=</span><span class="n">z</span><span class="p">;</span><span class="n">signal</span><span class="p">(</span><span class="n">SIGINT</span><span class="p">,</span>
<span class="n">si</span><span class="p">);</span><span class="n">memset</span><span class="p">(</span><span class="o">&amp;</span><span class="n">f</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="k">sizeof</span><span class="p">(</span><span class="n">f</span><span class="p">));</span><span class="k">while</span><span class="p">(</span><span class="n">poll</span><span class="p">(</span><span class="n">fds</span><span class="p">,</span><span class="n">b</span><span class="p">,</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span><span class="o">!=-</span><span class="mi">1</span><span class="o">&amp;&amp;</span><span class="n">c</span><span class="p">){</span><span class="k">if</span><span class="p">(</span><span class="n">fds</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">revents</span><span class="p">){</span><span class="n">I</span> <span class="n">u</span><span class="o">=</span>
<span class="n">accept</span><span class="p">(</span><span class="n">z</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">);</span><span class="n">fds</span><span class="p">[</span><span class="n">b</span><span class="o">++</span><span class="p">].</span><span class="n">fd</span><span class="o">=</span><span class="n">u</span><span class="p">;}</span><span class="k">for</span><span class="p">(</span><span class="n">I</span> <span class="n">i</span><span class="o">=</span><span class="mi">1</span><span class="p">;</span><span class="n">i</span><span class="o">&lt;</span><span class="n">b</span><span class="p">;</span><span class="o">++</span><span class="n">i</span><span class="p">){</span><span class="k">if</span><span class="p">(</span><span class="n">fds</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">revents</span><span class="o">&amp;</span><span class="n">POLLHUP</span><span class="p">){</span>
<span class="n">memmove</span><span class="p">(</span><span class="o">&amp;</span><span class="n">fds</span><span class="p">[</span><span class="n">i</span><span class="p">],</span><span class="o">&amp;</span><span class="n">fds</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">],</span><span class="mi">32</span><span class="o">-</span><span class="n">i</span><span class="p">);</span><span class="n">memset</span><span class="p">(</span><span class="n">f</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span><span class="mi">0</span><span class="p">,</span><span class="mi">128</span><span class="o">*</span><span class="k">sizeof</span><span class="p">(</span><span class="o">**</span><span class="n">f</span><span class="p">));</span><span class="o">--</span><span class="n">b</span><span class="p">;</span><span class="k">continue</span><span class="p">;}</span>
<span class="k">else</span> <span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="n">fds</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">revents</span><span class="p">){</span><span class="k">continue</span><span class="p">;}</span><span class="n">I</span> <span class="n">u</span><span class="o">=</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">;</span><span class="n">I</span> <span class="n">t</span><span class="o">=</span><span class="n">fds</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">fd</span><span class="p">;</span><span class="n">u32</span> <span class="n">s</span><span class="p">,</span><span class="n">r</span><span class="p">;</span><span class="kt">char</span> <span class="n">q</span><span class="p">[</span>
<span class="n">CMSG_SPACE</span><span class="p">(</span><span class="mi">8</span><span class="p">)];</span><span class="k">struct</span> <span class="n">cmsghdr</span> <span class="o">*</span><span class="n">p</span><span class="p">;</span><span class="k">struct</span> <span class="n">iovec</span> <span class="n">n</span><span class="o">=</span><span class="p">{.</span><span class="n">iov_base</span><span class="o">=&amp;</span><span class="n">s</span><span class="p">,.</span><span class="n">iov_len</span><span class="o">=</span><span class="mi">4</span><span class="p">};</span><span class="k">struct</span>
<span class="n">msghdr</span> <span class="n">m</span><span class="o">=</span><span class="p">{</span><span class="mi">0</span><span class="p">};</span><span class="n">m</span><span class="p">.</span><span class="n">msg_iov</span><span class="o">=&amp;</span><span class="n">n</span><span class="p">;</span><span class="n">m</span><span class="p">.</span><span class="n">msg_iovlen</span><span class="o">=</span><span class="mi">1</span><span class="p">;</span><span class="n">m</span><span class="p">.</span><span class="n">msg_control</span><span class="o">=</span><span class="n">q</span><span class="p">;</span><span class="n">m</span><span class="p">.</span><span class="n">msg_controllen</span><span class="o">=</span><span class="k">sizeof</span>
<span class="p">(</span><span class="n">q</span><span class="p">);</span><span class="n">recvmsg</span><span class="p">(</span><span class="n">t</span><span class="p">,</span><span class="o">&amp;</span><span class="n">m</span><span class="p">,</span><span class="mi">0</span><span class="p">);</span><span class="n">p</span><span class="o">=</span><span class="n">CMSG_FIRSTHDR</span><span class="p">(</span><span class="o">&amp;</span><span class="n">m</span><span class="p">);</span><span class="k">if</span><span class="p">(</span><span class="n">p</span><span class="p">){</span><span class="n">d</span><span class="o">=*</span><span class="p">(</span><span class="n">I</span> <span class="o">*</span><span class="p">)</span><span class="n">CMSG_DATA</span><span class="p">(</span><span class="n">p</span><span class="p">);}</span><span class="n">read</span><span class="p">(</span><span class="n">t</span><span class="p">,</span><span class="o">&amp;</span><span class="n">r</span><span class="p">,</span><span class="mi">4</span><span class="p">)</span>
<span class="p">;</span><span class="n">u16</span> <span class="n">o</span><span class="o">=</span><span class="p">((</span><span class="n">r</span><span class="o">&gt;&gt;</span><span class="mi">16</span><span class="p">)</span><span class="o">&amp;</span><span class="mh">0xFFFF</span><span class="p">)</span><span class="o">-</span><span class="mi">8</span><span class="p">,</span><span class="n">c</span><span class="o">=</span><span class="n">r</span><span class="o">&amp;</span><span class="mh">0xFFFF</span><span class="p">;</span><span class="k">if</span><span class="p">(</span><span class="n">s</span><span class="o">==</span><span class="mi">1</span><span class="p">){</span><span class="n">gi</span><span class="p">(</span><span class="n">u</span><span class="p">,</span><span class="n">t</span><span class="p">,</span><span class="n">s</span><span class="p">,</span><span class="n">c</span><span class="p">);}</span><span class="k">else</span><span class="p">{</span><span class="k">for</span><span class="p">(</span><span class="n">S</span> <span class="n">j</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="n">j</span><span class="o">&lt;</span><span class="mi">128</span><span class="p">;</span>
<span class="o">++</span><span class="n">j</span><span class="p">){</span><span class="k">if</span><span class="p">(</span><span class="n">f</span><span class="p">[</span><span class="n">u</span><span class="p">][</span><span class="n">j</span><span class="p">].</span><span class="n">a</span><span class="o">==</span><span class="n">s</span><span class="o">&amp;&amp;</span><span class="n">g</span><span class="p">[</span><span class="n">f</span><span class="p">[</span><span class="n">u</span><span class="p">][</span><span class="n">j</span><span class="p">].</span><span class="n">b</span><span class="p">].</span><span class="n">c</span><span class="p">){</span><span class="n">o</span><span class="o">-=</span><span class="n">g</span><span class="p">[</span><span class="n">f</span><span class="p">[</span><span class="n">u</span><span class="p">][</span><span class="n">j</span><span class="p">].</span><span class="n">b</span><span class="p">].</span><span class="n">c</span><span class="p">(</span><span class="n">u</span><span class="p">,</span><span class="n">t</span><span class="p">,</span><span class="n">s</span><span class="p">,</span><span class="n">c</span><span class="p">);</span><span class="k">break</span><span class="p">;}}</span><span class="k">if</span><span class="p">(</span><span class="n">o</span><span class="o">&gt;</span><span class="mi">0</span><span class="p">)</span>
                     <span class="p">{</span><span class="n">read</span><span class="p">(</span><span class="n">t</span><span class="p">,</span><span class="n">a</span><span class="p">,</span><span class="n">o</span><span class="p">);}}}}</span><span class="n">unlink</span><span class="p">(</span><span class="n">y</span><span class="p">.</span><span class="n">sun_path</span><span class="p">);}</span>
</code></pre></div><p>You&rsquo;re welcome!</p>


guid: https://drewdevault.com/2020/11/17/Better-than-DuckDuckGo.html
title: We can do better than DuckDuckGo
url: https://drewdevault.com/2020/11/17/Better-than-DuckDuckGo.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p><a href="https://duckduckgo.com">DuckDuckGo</a> is one of the long-time darlings of the
technophile&rsquo;s pro-privacy recommendations, and in fact the search engine that I
use myself on the daily. They certainly present a more compelling option than
many of the incumbents, like Google or Bing. Even so, DuckDuckGo is not good
enough, and we ought to do better.</p>
<p>I have three grievances with DuckDuckGo:</p>
<ol>
<li><strong>It&rsquo;s not open source.</strong> Almost all of DDG&rsquo;s software is proprietary, and
they&rsquo;ve demonstrated <a href="https://github.com/duckduckgo/Android/issues/527">gross incompetence</a> in privacy in what little
software they have made open source. Who knows what else is going on in the
proprietary code?</li>
<li><strong>DuckDuckGo is not a search engine</strong>. It&rsquo;s more aptly described as a search
engine frontend. They <em>do</em> handle features like bangs and instant answers
internally, but their actual search results come from third-parties like
Bing. They don&rsquo;t operate a crawler for their search results, and are not
independent.</li>
<li><strong>The search results suck!</strong> The authoritative sources for anything I want to
find are almost always buried beneath 2-5 results from content scrapers and
blogspam. This is also true of other search engines like Google. Search
engines are highly vulnerable to abuse and they aren&rsquo;t doing enough to
address it.</li>
</ol>
<p>There are some FOSS attempts to do better here, but they all fall flat.
<a href="https://github.com/bauruine/searx/">searX</a> is also a false search engine
— that is, they serve someone else&rsquo;s results. <a href="https://yacy.net/">YaCy</a>
has their own crawler, but the distributed design makes results untolerably
slow, poor quality, and vulnerable to abuse, and it&rsquo;s missing strong central
project leadership.</p>
<p>We need a real, working FOSS search engine, complete with its own crawler.</p>
<p>Here&rsquo;s how I would design it.</p>
<p>First, YaCy-style decentralization is <em>way</em> too hard to get right, especially
when a search engine project already has a lot of Very Hard problems to solve.
Federation is also very hard in this situation — queries will have to
consult <em>most</em> instances in order to get good quality results, or a novel
sharding algorithm will have to be designed, and either approach will have to be
tolerant of nodes appearing and disappearing at any time. Not to mention it&rsquo;d be
slow! Several unsolved problems with federation and decentralziation would have
to be addressed on top of building a search engine in the first place.</p>
<p>So, a SourceHut-style approach is better. 100% of the software would be free
software, and third parties would be encouraged to set up their own
installations. It would use standard protocols and formats where applicable, and
accept patches from the community. However, the database would still be
centralized, and even if programmable access were provided, it would not be with
an emphasis on decentralization or shared governance. It might be possible to
design tools which help third-parties bootstrap their indexes, and create a
community of informal index sharing, but that&rsquo;s not the focus here.</p>
<p>It would also need its own crawler, and probably its own indexer. I&rsquo;m not
convinced that any of the existing FOSS solutions in this space are quite right
for this problem. Crucially, I would <em>not</em> have it crawling the entire web from
the outset. Instead, it should crawl a whitelist of domains, or &ldquo;tier 1&rdquo;
domains. These would be the limited mainly to authoritative or high-quality
sources for their respective specializations, and would be weighed upwards in
search results. Pages that these sites link to would be crawled as well, and
given tier 2 status, recursively up to an arbitrary N tiers. Users who want to
find, say, a blog post about a subject rather than the documentation on that
subject, would have to be more specific: &ldquo;$subject blog posts&rdquo;.</p>
<p>An advantage of this design is that it would be easy for anyone to take the
software stack and plop it on their own servers, with their own whitelist of
tier 1 domains, to easily create a domain-specific search engine. Independent
groups could create search engines which specialize in academia, open standards,
specific fandoms, and so on. They could tweak their precise approach to
indexing, tokenization, and so on to better suit their domain.</p>
<p>We should also prepare the software to boldly lead the way on new internet
standards. Crawling and indexing non-HTTP data sources (Gemini?  Man pages?
Linux distribution repositories?), supporting non-traditional network stacks
(Tor? Yggdrasil? cjdns?) and third-party name systems (OpenNIC?), and anything
else we could leverage our influence to give a leg up on.</p>
<p>There&rsquo;s a <em>ton</em> of potential in this domain which is just sitting on the floor
right now. The main problem is: who&rsquo;s going to pay for it? Advertisements or
paid results are <em>not</em> going to fly — conflict of interest. Private, paid
access to search APIs or index internals is one opportunity, but it&rsquo;s kind of
shit and I think that preferring open data access and open APIs would be
exceptionally valuable for the community.</p>
<p>If SourceHut eventually grows in revenue — at least 5-10× its
<a href="https://sourcehut.org/blog/2020-11-11-sourcehut-q3-2020-financial-report/">present revenue</a> — I intend to sponsor this as a public
benefit project, with no plans for generating revenue. I am not aware of any
monetization approach for a search engine which squares with my ethics and
doesn&rsquo;t fundamentally undermine the mission. So, if no one else has figured it
out by the time we have the resources to take it on, we&rsquo;ll do it.</p>


guid: https://drewdevault.com/2020/11/20/A-few-ways-to-make-money-in-FOSS.html
title: A few ways to make money in FOSS
url: https://drewdevault.com/2020/11/20/A-few-ways-to-make-money-in-FOSS.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I work on free and open-source software full time, and I make a comfortable
living doing it. And I don&rsquo;t half-ass it: 100% of my code is free and
open-source. There&rsquo;s no proprietary add-ons, no periodic code dumps, just
100% bona-fide free and open source software. Others have often sought my advice
— how can they, too, make a living doing open source?</p>
<p>Well, there&rsquo;s more than one way to skin a cat. There are many varieties of
software, each with different needs, and many kinds of people, each with
different needs. The exact approach which works for you and your project will
vary quite a bit depending on the nature of your project.</p>
<p>I would generally categorize my advice into two bins:</p>
<ul>
<li>You want to make money from your own projects</li>
<li>You want to make money participating in open source</li>
</ul>
<p>The first one is more difficult. We&rsquo;ll start with the latter.</p>
<h2 id="being-employed-in-foss">Being employed in FOSS</h2>
<p>One way to make money in FOSS is to get someone to pay you to write free
software. There&rsquo;s lots of advantages to this: minimal personal risk, at-market
salaries, benefits, and so on, but at the cost of not necessarily getting to
choose what you work on all the time.</p>
<p>I have a little trick that I often suggest to people who vaguely want to work
&ldquo;in FOSS&rdquo;, but who aren&rsquo;t trying to find the monetization potential in their own
projects. Use git to clone the source repositories for some (large) projects
you&rsquo;re interested in,
<a href="https://drewdevault.com/2020/08/10/How-to-contribute-to-FOSS.html">the kind of stuff you want to work on</a>,
and then run this command:</p>
<pre><code>git log -n100000 --format=&quot;%ae&quot; | cut -d@ -f2 | sort | uniq -c | sort -nr | less
</code></pre><p>This will output a list of the email domains who have committed to the
repository in the last 100,000 commits. This is a good set of leads for
companies who might be interested in paying you to work on projects like this
😉</p>
<p>Another good way is to explicitly seek out large companies known to work a lot
in FOSS, and see if they&rsquo;re hiring in those departments. There are some
companies that specialize in FOSS, such as RedHat, Collabora, and dozens more;
and there are large companies with FOSS-specific teams, such as Intel, AMD, IBM,
and so on.</p>
<h2 id="making-money-from-your-own-foss-work">Making money from your own FOSS work</h2>
<p>If you want to pay for the project infrastructure, and maybe beer money for the
weekend, then donations are an easy way to do that. I&rsquo;ll give it to you
straight, though: you&rsquo;re unlikely to make a living from donations.  Programmers
who do are a small minority. If you want to make a living from FOSS, it&rsquo;s
going to be more difficult.</p>
<p>Start by unlearning what you think you know about startups. The toxic startup
culture around venture capital and endless hyper-growth is more stressful, less
likely to succeed, and socially irresponsible. Building a sustainable business
responsibly takes time, careful planning, and hard work. The fast route —
venture capital funded — is going to impose constraints on your business
that will ultimately make it difficult to remain true to your open-source
mission.</p>
<p>And yes, you are building a <em>business</em>. You need to start thinking of your
project as a business and of yourself as a business owner. This undertaking is
going to require developing business skills in planning, budgeting, scheduling,
resource allocation, marketing &amp; sales, compliance, and more. At times, you will
be forced to embrace your inner
<a href="http://www.catb.org/jargon/html/S/suit.html">suit</a>. Channel your engineering
problem-solving skills into the business problems.</p>
<p>So, you&rsquo;ve got the right mindset. What are some business models that work?</p>
<p><a href="https://sourcehut.org">SourceHut</a>, my company, has two revenue streams. We have
a hosted SaaS product. It&rsquo;s open source, and users can choose to deploy and
maintain it themselves, or they can just buy a hosted account from us. The
services are somewhat complex, so the managed offering saves them a lot of time.
We have skilled sysops/sysadmins, support channels, and so on, for paying users.
Importantly, we don&rsquo;t have a free tier (but we do choose to provide free service
to those who need it, at our discretion).</p>
<p>Our secondary revenue stream is <a href="https://sourcehut.org/consultancy">free software
consulting</a>. Our developers work part-time
writing free and open-source software on contracts. We&rsquo;re asked to help
implement features upstream for various projects, or to develop new open-source
applications or libraries, to share our expertise in operations, and so on, and
charge for these services. This is different from providing paid support or
development on our own projects — we accept contracts to work on <em>any</em>
open source project.</p>
<p>The other approach to consulting is also possible: paid support and development
on your own projects. If there are businesses that rely on your project, then
you may be able to offer them support or develop new features or bugfixes that
they need, on a paid basis. Projects with a large corporate userbase also
sometimes <em>do</em> find success in donations — albeit rebranded as
sponsorships. The largest projects often set up foundations to manage them in
this manner.</p>
<p>These are, in my experience, some of the most successful approaches to
monetizing FOSS. You may have success with a combination of these, or with other
business models as well.  Remember to turn that engineering mind of yours
towards the task of monetization, and experiment with and invent new ways of
making money that best suit the kind of software you want to work on.</p>
<p>Feel free to <a href="mailto:sir@cmpwn.com">reach out</a> if you have some questions or
need a sounding board for your ideas. Good luck!</p>


guid: https://drewdevault.com/2020/12/04/Analytics-and-informed-consent.html
title: Web analytics should at least meet the standards of informed consent
url: https://drewdevault.com/2020/12/04/Analytics-and-informed-consent.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Research conducted on human beings, at least outside of the domain of
technology, has to meet a minimum standard of ethical reasoning called
<a href="https://en.wikipedia.org/wiki/Informed_consent">informed consent</a>. Details
vary, but the general elements of informed consent are:</p>
<ol>
<li>Disclosure of the nature and purpose of the research and its implications
(risks and benefits) for the participant, and the confidentiality of the
collected information.</li>
<li>An adequate understanding of these facts on the part of the participant,
requiring an accessible explanation in lay terms and an assessment of
understanding.</li>
<li>The participant must exercise voluntary agreement, without coercion or fear
of repercussions (e.g. not being allowed to use your website).</li>
</ol>
<p>So, I pose the following question: if your analytics script wouldn&rsquo;t pass muster
at your university&rsquo;s ethics board, then what the hell is it doing on your
website? Can we not meet this basic minimum standard of ethical decency and
respect for our users?</p>
<p>Opt-out is not informed consent. Manually unticking dozens of third-party
trackers from a cookie pop-up is not informed consent. &ldquo;By continuing to use
this website, you agree to&hellip;&rdquo; is not informed consent. &ldquo;Install <a href="https://ublockorigin.com/">uBlock
Origin</a>&rdquo; is not informed consent.</p>
<p>I don&rsquo;t necessarily believe that ethical user tracking is <em>impossible</em>, but I
know for damn sure that most of these &ldquo;pro-privacy&rdquo; analytics solutions which
have been cropping up in the wake of the GDPR don&rsquo;t qualify, either.</p>
<p>Our industry&rsquo;s fundamental failure to respect users, deliberately mining their
data without consent and without oversight for profit, is the reason why we&rsquo;re
seeing legal crackdowns in the form of the GDPR and similar legislation.  Our
comeuppance is well-earned, and I hope that the regulators give it teeth in
enforcement. The industry response — denial and looking for ways to weasel
out of these ethical obligations — is a strategy on borrowed time. The law
is not a computer program, and it is not executed by computers: it is executed
by human beings who can see through your horseshit. You&rsquo;re not going to be able
to seek out some narrow path you can walk to skirt the regulations and keep
spying on people.</p>
<p>You&rsquo;re going to stop spying on people.</p>
<p><em>P.S. If you still want the data you might get from analytics without
compromising on ethics, here&rsquo;s an idea: compensate users for their participation
in your research. Woah, what a wild idea! That&rsquo;s not very growth hacker of you,
Drew.</em></p>


guid: https://drewdevault.com/2020/12/12/Shell-literacy.html
title: Become shell literate
url: https://drewdevault.com/2020/12/12/Shell-literacy.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Shell literacy is one of the most important skills you ought to possess as a
programmer. The Unix shell is one of the most powerful ideas ever put to code,
and should be second nature to you as a programmer. No other tool is nearly as
effective at commanding your computer to perform complex tasks quickly —
or at storing them as scripts you can use later.</p>
<p>In my workflow, I use Vim as my editor, and Unix as my &ldquo;IDE&rdquo;. I don&rsquo;t trick out
<a href="https://git.sr.ht/~sircmpwn/dotfiles/tree/master/.vimrc">my vimrc</a> to add a
bunch of IDE-like features — the most substantial plugin I use on a daily
basis is <a href="https://github.com/ctrlpvim/ctrlp.vim">Ctrl+P</a>, and that just makes it
easier to open files. Being Vim literate is a valuable skill, but an important
detail is knowing when to drop it. My daily workflow involves several open
terminals, generally one with Vim, another to run builds or daemons, and a third
which just keeps a shell handy for anything I might ask of it.</p>
<p><a href="https://l.sr.ht/g_oL.png"><img src="https://l.sr.ht/g_oL.png" alt="Screenshot of my workspace"></a></p>
<p>The shell I keep open allows me to perform complex tasks and answer complex
questions as I work. I find interesting things with <a href="https://git-scm.com/docs/git-grep">git grep</a>,
perform bulk find-and-replace with <a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html#top">sed</a>, answer questions with
<a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html#top">awk</a>, and perform more intricate tasks on-demand with ad-hoc shell
commands and pipelines. I have the freedom to creatively solve problems without
being constrained to the rails laid by IDE designers.</p>
<p>Here&rsquo;s an example of a problem I encountered recently: I had a bunch of changes
in a git repository. I wanted to restore deleted files without dropping the rest
of my changes, but there were hundreds of these. How can I efficiently address
this problem?</p>
<p>Well, I start by getting a grasp of the scale of the issue with git status,
which shows hundreds of deleted files that need to be restored. This scale is
beyond the practical limit of manual intervention, so I switch to git status
-s to get a more pipeline-friendly output.</p>
<pre><code>$ git status -s
 D main/a52dec/APKBUILD
 D main/a52dec/a52dec-0.7.4-build.patch
 D main/a52dec/automake.patch
 D main/a52dec/fix-globals-test-x86-pie.patch
 D main/aaudit/APKBUILD
 D main/aaudit/aaudit
 D main/aaudit/aaudit-common.lua
 D main/aaudit/aaudit-repo
 D main/aaudit/aaudit-server.json
 D main/aaudit/aaudit-server.lua
 ...
</code></pre><p>I can work with this. I add grep '^ D' to filter out any entries which were
not deleted, and pipe it through awk '{ print $2 }' to extract just the
filenames. I&rsquo;ll often run the incomplete pipeline just to check my work and
catch my bearings:</p>
<pre><code>$ git status -s | grep '^ D' | awk '{ print $2 }'
main/a52dec/APKBUILD
main/a52dec/a52dec-0.7.4-build.patch
main/a52dec/automake.patch
main/a52dec/fix-globals-test-x86-pie.patch
main/aaudit/APKBUILD
main/aaudit/aaudit
main/aaudit/aaudit-common.lua
main/aaudit/aaudit-repo
main/aaudit/aaudit-server.json
main/aaudit/aaudit-server.lua
...
</code></pre><p>Very good — we have produced a list of files which we need to address.
Note that, in retrospect, I could have dropped the grep and just used awk to the
same effect:</p>
<pre><code>$ git status -s | awk '/^ D/ { print $2 }'
main/a52dec/APKBUILD
main/a52dec/a52dec-0.7.4-build.patch
main/a52dec/automake.patch
main/a52dec/fix-globals-test-x86-pie.patch
main/aaudit/APKBUILD
main/aaudit/aaudit
main/aaudit/aaudit-common.lua
main/aaudit/aaudit-repo
main/aaudit/aaudit-server.json
main/aaudit/aaudit-server.lua
...
</code></pre><p>However, we&rsquo;re just writing an ad-hoc command here to solve a specific,
temporary problem — finesse is not important. This command isn&rsquo;t going to
be subjected to a code review. Often my thinking in these situations is to solve
one problem at a time: &ldquo;filter the list&rdquo; and &ldquo;reword the list&rdquo;.  Anyway, the
last step is to actually use this list of files to address the issue, with the
help of <a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html#top">xargs</a>.</p>
<pre><code>$ git status -s | awk '/^ D/ { print $2 }' | xargs git checkout --
</code></pre><p>Let&rsquo;s look at some more examples of interesting ad-hoc shell pipelines.
Naturally, I wrote a shell pipeline to find some:</p>
<pre><code>$ history | cut -d' ' -f2- | awk -F'|' '{ print NF-1 &quot; &quot; $0 }' | sort -n | tail
</code></pre><p>Here&rsquo;s the breakdown:</p>
<ul>
<li><code>history</code> prints a list of my historical shell commands.</li>
<li><code>cut -d' ' -f2-</code> removes the first field from each line, using space as a
delimiter. <code>history</code> numbers every command, and this removes the number.</li>
<li><code>awk -F'|' '{ print NF-1 &quot; &quot; $0 }</code> tells awk to use | as the field delimiter
for each line, and print each line prefixed with the number of fields. This
prints every line of my history, prefixed with the number of times the pipe
operator appears in that line.</li>
<li><code>sort -n</code> numerically sorts this list.</li>
<li><code>tail</code> prints the last 10 items.</li>
</ul>
<p>This command, written in the moment, finds, characterizes, filters, and sorts my
shell history by command complexity. Here are a couple of the cool shell
commands I found:</p>
<p>Play the 50 newest videos in a directory with
<a href="https://github.com/mpv-player/mpv">mpv</a>:</p>
<pre><code>ls -tc | head -n50 | tr '\n' '\0' | xargs -0 mpv
</code></pre><p>I use this command all the time. If I want to watch a video later, I will
<a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/touch.html#top">touch</a> the file so it appears at the top of this list. Another command
transmits a tarball of a patched version of <a href="http://www.celestegame.com/">Celeste</a> to a friend using
netcat, minus the (large) game assets, with a progress display via <a href="http://www.ivarch.com/programs/pv.shtml">pv</a>:</p>
<pre><code>find . ! -path './Content/*' | xargs tar -cv | pv | zstd | nc 204:fbf5:... 12345
</code></pre><p>And on my friend&rsquo;s end:</p>
<pre><code>nc -vll :: 12345 | zstdcat | pv | tar -xv
</code></pre><p>tar, by the way, is an under-rated tool for moving multiple files through a
pipeline. It can read and write tarballs to stdin and stdout!</p>
<p>I hope that this has given you a tantalizing taste of the power of the Unix
shell. If you want to learn more about the shell, I can recommend
<a href="http://shellhaters.org/">shellhaters.org</a> as a great jumping-off point into
various shell-related parts of the POSIX specification. Don&rsquo;t be afraid of the
spec — it&rsquo;s concise, comprehensive, comprehensible, and full of examples.
I would also <em>definitely</em> recommend taking some time to learn awk in particular:
<a href="https://ferd.ca/awk-in-20-minutes.html">here&rsquo;s a brief tutorial</a>.</p>


guid: https://drewdevault.com/2020/12/15/Status-update-December-2020.html
title: Status update, December 2020
url: https://drewdevault.com/2020/12/15/Status-update-December-2020.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Happy holidays! I hope everyone&rsquo;s having a great time staying at home and not
spending any time with your families. It&rsquo;s time for another summary of the
month&rsquo;s advances in FOSS development. Let&rsquo;s get to it!</p>
<p>One of my main focuses has been on sourcehut&rsquo;s API 2.0 planning. This month, the
meta.sr.ht and git.sr.ht GraphQL APIs have shipped feature parity with the REST
APIs, and the RFC 6749 compatible OAuth 2.0 implementation has shipped. I&rsquo;ve
broken ground on the todo.sr.ht GraphQL API — it&rsquo;ll be next. Check out the
<a href="https://man.sr.ht/graphql.md">GraphQL docs on man.sr.ht</a> if you want to kick
the tires.</p>
<p>I also wrote a little tool this month called
<a href="https://git.sr.ht/~sircmpwn/mkproof">mkproof</a>, after brainstorming some ways to
allow sourcehut signups over Tor without enabling abuse. The idea is that you
can generate a challenge (mkchallenge), give it to a user who generates a proof
for that challenge (mkproof), and then verify their proof is correct. Generating
the proof is computationally expensive and resistant to highly parallel attacks
(e.g. GPUs), and takes tens of minutes of work — making it unpractical for
spammers to register accounts in bulk, while still allowing Tor users to
register with their anonymity intact.</p>
<p>On the Gemini front, patches from Mark Dain, William Casarin, and Eyal Sawady
have improved <a href="https://git.sr.ht/~sircmpwn/gmnisrv">gmnisrv</a> in several respects
— mainly bugfixes — and <a href="https://git.sr.ht/~sircmpwn/gmni">gmnlm</a>
has grown the &ldquo;&lt;n&gt;|&rdquo; command, which pipes the Nth link into a shell
command. Thanks are due to Alexey Yerin as well, who sent a little bugfix with
redirect handling.</p>
<p>The <a href="https://datatracker.ietf.org/doc/draft-devault-bare/">second draft of the BARE
specification</a> was
submitted to the IETF this month. Will revisit it again in several weeks. John
Mulligan has also sent several patches improving go-bare — thanks!</p>
<p>scdoc 1.11.0 was released this month, with only minor bug fixes.</p>
<p>That&rsquo;s all for now! I&rsquo;ll see you in a month.</p>
<details>
  <summary>...</summary>
  <p>
  The secret project has slowed down a bit as we've started on a new phase of
  development: writing the specification, and new compiler which implements it
  from the ground up. Progress on this is good, but won't introduce anything
  groundbreaking for a while. Stay tuned.
</details>


guid: https://drewdevault.com/2020/12/18/godocs.io.html
title: godocs.io is now available
url: https://drewdevault.com/2020/12/18/godocs.io.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Due to the coming sunsetting of godoc.org in favor of pkg.go.dev, I&rsquo;m happy to
announce that <a href="https://godocs.io">godocs.io</a> is now available as a replacement.
We have <a href="https://sr.ht/~sircmpwn/godocs.io">forked the codebase</a> and cleaned
things up quite a bit, removing lots of dead or obsolete features, cleaning out
a bunch of Google-specific code and analytics, reducing the JavaScript
requirements, and rewriting the search index for Postgres. We will commit to its
maintenance going forward for anyone who prefers the original godoc.org
experience over the new website.</p>
<p>Notice: this article was rewritten on 2021-01-19. The original article has a lot
of unnecessary salt. <a href="https://git.sr.ht/~sircmpwn/drewdevault.com/tree/fa8799c882f6555606e0a906bd8e0a484bb51398/item/content/blog/godocs.io.md">You can read the original here</a>.</p>


guid: https://drewdevault.com/2020/12/25/How-to-design-a-new-programming-language.html
title: How to design a new programming language from scratch
url: https://drewdevault.com/2020/12/25/How-to-design-a-new-programming-language.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>There is a long, difficult road from vague, pie-in-the-sky ideas about what
would be cool to have in a new programming language, to a robust,
self-consistent, practical implementation of those ideas. Designing and
implementing a new programming language from scratch is one of the most
challenging tasks a programmer can undertake.</p>
<p>Note: this post is targeted at motivated programmers who want to make a
serious attempt at designing a useful programming language. If you just want to
make a language as a fun side project, then you can totally just wing it. Taking
on an unserious project of that nature is also a good way to develop some
expertise which will be useful for a serious project later on.</p>
<p>Let&rsquo;s set the scene. You already know a few programming languages, and you know
what you like and dislike about them — these are your influences. You have
some cool novel language design ideas as well. A good first step from here is to
dream up some pseudocode, putting some of your ideas to paper, so you can get an
idea of what it would actually feel like to write or read code in this
hypothetical language. Perhaps a short write-up or a list of goals and ideas is
also in order. Circulate these among your confidants for discussion and
feedback.</p>
<p>Ideas need to be proven in the forge of implementations, and the next step is to
write a compiler (or interpreter — everything in this article applies
equally to them). We&rsquo;ll call this the sacrificial implementation, because you
should be prepared to throw it away later. Its purpose is to prove that your
design ideas work and can be implemented efficiently, but <em>not</em> to be the
production-ready implementation of your new language. It&rsquo;s a tool to help you
refine your language design.</p>
<p>To this end, I would suggest using a parser generator like yacc to create your
parser, even if you&rsquo;d prefer to ultimately use a different design (e.g.
recursive descent). The ability to quickly make changes to your grammar, and the
side-effect of having a formal grammar written as you work, are both valuable to
have at this stage of development. Being prepared to throw out the rest of the
compiler is helpful because, due to the inherent difficulty of designing and
implementing a programming language at the same time, your first implementation
will probably be shit. You don&rsquo;t know what the language will look like, you&rsquo;ll
make assumptions that you have to undo later, and it&rsquo;ll undergo dozens of
refactorings. It&rsquo;s gonna suck.</p>
<p>However, shit as it may be, it will have done important work in validating your
ideas and refining your design. I would recommend that your next step is to
start working on a formal specification of the language (something that I
believe all languages should have). You&rsquo;ve proven what works, and writing it up
formally is a good way to finalize the ideas and address the edge cases. Gather
a group of interested early adopters, contributors, and subject matter experts
(e.g. compiler experts who work with similar languages), and hold discussions on
the specification as you work.</p>
<p>This is also a good time to start working on your second implementation. At this
point, you will have a good grasp on the overall compiler design, the flaws from
your original implementation, and better skills as a compiler programmer.
Working on your second compiler and your specification at the same time can help
as both endeavours inform the others — a particularly difficult detail to
implement could lead to a simplification in the spec, and an under-specified
detail getting shored up could lead to a more robust implementation.</p>
<p>Don&rsquo;t get carried away — keep this new compiler simple and small. Don&rsquo;t go
crazy on nice-to-have features like linters and formatters, an exhaustive test
suite, detailed error messages, a sophisticated optimizer, and so on. You want
it to implement the specification as simply as possible, so that you can use it
for the next step: the hosted compiler. You need to write a third
implementation, using your own language to compile itself.</p>
<p>The second compiler, which I hope you wrote in C, is now the bootstrap compiler.
I recommend keeping it up-to-date with the specification and maintaining it
perpetually as a convenient path to bootstrap your toolchain from scratch
(looking at you, Rust). But it&rsquo;s not going to be the final implementation: any
self-respecting general-purpose programming language is implemented in itself.
The next, and final step, is to implement your language for a third time.</p>
<p>At this point, you will have refined and proven your language design. You will
have developed and applied compiler programming skills. You will have a robust
implementation for a complete and self-consistent programming language,
developed carefully and with the benefit of hindsight. Your future community
will thank you for the care and time you put into this work, as your language
design and implementation sets the ceiling on the quality of programs written in
it.</p>


guid: https://drewdevault.com/2021/01/01/Megacorps-are-not-your-dream-job.html
title: A megacorp is not your dream job
url: https://drewdevault.com/2021/01/01/Megacorps-are-not-your-dream-job.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Megacorporations<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> <em>do not</em> care about you. You&rsquo;re worth nothing to them.
Google made $66 billion in 2014 — even if you made an exorbitant $500K
salary, you only cost them .00075% of that revenue. They are not invested in
you. Why should you invest in them? Why should you give a company that isn&rsquo;t
invested in you 40+ hours of your week, half your waking life, the <em>only</em> life
you get?</p>
<p>You will have little to no meaningful autonomy, impact, or influence. Your
manager&rsquo;s manager&rsquo;s manager&rsquo;s manager (1) will exist, and (2) will not know your
name, and probably not your manager&rsquo;s name either. The company will be good at
advertising their jobs, especially to fresh grads, and you will no doubt have
dozens of cool project in mind that you&rsquo;re itching to get involved with. You
won&rsquo;t be assigned any of them — all vacancies are already filled by
tenured staff and nepotism. You&rsquo;re more likely to work on a product you have
hardly ever heard of or used, doing work that doesn&rsquo;t interest you or
meaningfully impact anyone you know.</p>
<p>A business doesn&rsquo;t get a billion-dollar valuation (or&hellip; ugh&hellip; a
trillion-dollar valuation) by having a productive team which takes good care of
its employees, rewarding them with interesting projects, or quickly correcting
toxic work environments. A business might get millions of dollars, at most, with
that approach. The megacorps got their 10th figure with another strategy:
ruthlessness. They create and exploit monopolies, and bribe regulators to look
the other way. They acquire and dismantle competitors. They hire H1B&rsquo;s and
subject them to payroll fraud and workplace abuse, confident that they can&rsquo;t
quit without risking their visa. Megacorps are a faceless machine which is
interested only in making as much money as possible with any resources at their
disposal, among those being a budget which exceeds most national GDPs.<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup></p>
<p>If anything goes wrong in this heartless environment, you&rsquo;re going to be in a
very weak position. If you go to HR<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> for almost any dispute, they are
unlikely to help. If you quit, remember that they will have forced you to sign
an NDA and a non-compete.  You&rsquo;re rolling the dice on whether or not they&rsquo;ll
decide that you&rsquo;ve overstepped (and they can <em>decide</em> that — the terms are
virtually impossible not to breach). That .00075% of their annual revenue you
took home? They could easily spend 100x that on lawyers without breaking a
sweat, and money is justice in the United States. You will likely have no
recourse if they wrong you.</p>
<p>They may hurt you, but even worse, they will make you hurt others. You will be
complicit in their ruthlessness. Privacy dragnets, union busting, monopolistic
behavior and lobbying, faux-slavery of gig workers in domestic warehouses and
actual-slavery of workers in foreign factories, answering to <a href="https://en.wikipedia.org/wiki/Uyghur_genocide">nations committing
actual ongoing genocide</a> —
this is only possible because highly skilled individuals like yourself chose to
work for them, build their war chest, or even directly contribute to these
efforts. Your salary may be a drop in the bucket to them, but consider how much
that figure means to you. If you make that $500K, they spend 1.5× that
after overhead, and they&rsquo;d only do it if they expect a return on that
investment. Would you give a corporation with this much blood on its hands
$750K of your worth? Pocket change to them, maybe, but a lot of value to <em>you</em>,
value that you could be adding somewhere else.</p>
<p>They won&rsquo;t care about you. They won&rsquo;t be invested in you. They won&rsquo;t give you
interesting work. You will have no recourse if things go wrong, and things are
primed to go wrong. They could hurt you, and they could make you hurt others.
Don&rsquo;t fall for their propaganda.</p>
<p>Megacorps are, in fact, in the minority. There are tens of thousands of other
tech companies that could use your help. Tech workers are in high demand —
you have choices! You will probably be much happier at a small to mid-size
company. The &ldquo;dream job&rdquo; megacorps have sold you on is just good marketing.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>EDIT @ 23:37 UTC: It bears clarifying that I&rsquo;m referring to extremely large companies, at or near the scale of FAANG (Facebook, Apple, Amazon, Netflix, Google). Hundreds of billions of dollars or more in market cap. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Political side thought: Amazon&rsquo;s revenue in 2019 alone exceeds the GDP of 150 sovereign nations. Is undemocratic ownership of resources and power on that scale just? <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Quick reminder that HR&rsquo;s job is to protect the company, not you. This applies to any company, not just megacorps. If you have a problem that you need to bring to HR, you should have a lawyer draft that letter, and you should polish up your resume first. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2021/01/04/A-culture-of-stability-and-reliability.html
title: Fostering a culture that values stability and reliability
url: https://drewdevault.com/2021/01/04/A-culture-of-stability-and-reliability.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>There&rsquo;s an idea which encounters a bizarre level of resistance from the broader
software community: that software can be completed. This resistance manifests in
several forms, perhaps the most common being the notion that a git repository
which doesn&rsquo;t receive many commits is abandoned or less worthwhile. For my part,
I consider software that aims to be <em>completed</em> to be more worthwhile most of
the time.</p>
<p>There are two sources of change which projects are affected by: external and
internal. An internal source of change is, for example, a planned feature, or a
discovered bug. External sources of change are, say, when a dependency makes a
breaking change and your software has to be updated accordingly. Some projects
will necessarily have an indefinite source of external change to consider, often
as part of their value proposition. <a href="https://youtube-dl.org/">youtube-dl</a> will always evolve to add
new sites and workarounds, <a href="https://github.com/swaywm/wlroots">wlroots</a> will continue to grow to take advantage
of new graphics and input hardware features, and so on.</p>
<p>Any maintained program will naturally increase in stability over time as bug
fixes accumulate, towards some finite maximum. However, change drives this trend
in reverse. Introducing new features, coping with external change factors, even
fixing bugs, all of this often introduce new problems. If you want to produce
software which is reliable, robust, and stable, then managing change is an
essential requirement.</p>
<p>To this end, software projects can, and often should, draw a finish line. Or, if
not a finish line, a curve for gradually backing off on feature introduction,
raising the threshold of importance by which a new feature is considered.</p>
<p><a href="https://github.com/swaywm/sway">Sway</a>, for instance, was &ldquo;completed&rdquo; some time
ago. We stopped accepting most major feature requests, preferring only to
implement changes which were made necessary by external sources: notably,
features implemented in i3, the project sway aimed to replace. The i3 project
<a href="https://old.reddit.com/r/i3wm/comments/kn8pa2/an_update_on_the_future_of_i3/">announced this week</a> that it was adopting a similar policy regarding new
features, and thus sway&rsquo;s change management is again reduced in scope to only
addressing bugs and performance. Sway has completed its value proposition, and
now our only goal is to become more and more stable and reliable at delivering
it.</p>
<p><a href="https://sr.ht/~sircmpwn/scdoc">scdoc</a> is another project which has met its
stated goals. Its primary external source of change is roff — which is
almost 50 years old. Therefore, it has accumulated mainly bugfixes and
robustness over the past few years since its release, and users enjoy a great
deal of reliability and stability from it. Becoming a tool which &ldquo;just works&rdquo;
and can be depended on without a second thought is the only goal going forward.</p>
<p>Next time you see a git repo which is only getting a slow trickle of commits,
don&rsquo;t necessarily write it off as abandoned. A slow trickle of commits is the
ultimate fate of software which aims to be stable and reliable. And, as a
maintainer of your own projects, remember that turning a critical eye to new
feature requests, and evaluating their cost in terms of complexity and
stability, is another responsibility that your users are depending on you for.</p>


guid: https://drewdevault.com/2021/01/06/Invented-or-discovered.html
title: Was $X invented, or discovered?
url: https://drewdevault.com/2021/01/06/Invented-or-discovered.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>
Let's do a little exercise. Check the term which you think best describes the
nature of advances in each domain.

<em>Note: this page is interactive. JavaScript is not required.</em>

<style>
.idea {
  position: relative;
  text-align: center;
  border: 1px solid #333;
  background: #eee;
  margin-bottom: 1.5rem;
  display: grid;
  grid-template-rows: auto auto auto;
  grid-template-columns: 1fr 1fr;
}

.idea h3 {
  grid-column-start: 1;
  grid-column-end: 3;
  margin: 0.25rem;
}

.idea small {
  grid-row-start: 2;
  grid-row-end: 3;
  grid-column-start: 1;
  grid-column-end: 3;
}

.idea label {
  grid-row-start: 3;
  grid-row-end: 3;
  padding: 0.25rem 0;
  background: white;
  border-top: 1px solid #333;
}

.idea label:last-child {
  border-left: 1px solid #333;
}

button {
  padding: 0.25rem;
  font-size: 1.2rem;
  display: block;
  margin: 1rem auto;
}

form p {
  text-align: center;
}
</style>

<form method="POST" action="https://drewdevault.com/discoveries/submit">
  <div class="idea">
    <h3>Mathematics is...</h3>
    <label><input type="radio" name="math" value="0" /> Discovered</label>
    <label><input type="radio" name="math" value="1" /> Invented</label>
  </div>

  <div class="idea">
    <h3>Algorithms are...</h3>
    <label><input type="radio" name="algos" value="0" /> Discovered</label>
    <label><input type="radio" name="algos" value="1" /> Invented</label>
  </div>

  <div class="idea">
    <h3>Programming paradigms<sup><a href="#ft-1" id="ref-1">1</a></sup> are...</h3>
    <label><input type="radio" name="paradigms" value="0" /> Discovered</label>
    <label><input type="radio" name="paradigms" value="1" /> Invented</label>
  </div>

  <div class="idea">
    <h3>High-level software is...</h3>
    <label><input type="radio" name="software" value="0" /> Discovered</label>
    <label><input type="radio" name="software" value="1" /> Invented</label>
  </div>

  <p>
  Back to the fundamentals now:

  <div class="idea">
    <h3>Physics is...</h3>
    <label><input type="radio" name="physics" value="0" /> Discovered</label>
    <label><input type="radio" name="physics" value="1" /> Invented</label>
    <span class="result"></span>
  </div>

  <div class="idea">
    <h3>Chemistry is...</h3>
    <label><input type="radio" name="chemistry" value="0" /> Discovered</label>
    <label><input type="radio" name="chemistry" value="1" /> Invented</label>
  </div>

  <div class="idea">
    <h3>Biology is...</h3>
    <label><input type="radio" name="biology" value="0" /> Discovered</label>
    <label><input type="radio" name="biology" value="1" /> Invented</label>
  </div>

  <div class="idea">
    <h3>Pharmaceuticals are...</h3>
    <label><input type="radio" name="pharma" value="0" /> Discovered</label>
    <label><input type="radio" name="pharma" value="1" /> Invented</label>
  </div>

  <div class="idea">
    <h3>Medical treatments are...</h3>
    <label><input type="radio" name="medicine" value="0" /> Discovered</label>
    <label><input type="radio" name="medicine" value="1" /> Invented</label>
  </div>

  <p>
  How about this?

  <div class="idea">
    <h3>Psychology is...</h3>
    <label><input type="radio" name="psychology" value="0" /> Discovered</label>
    <label><input type="radio" name="psychology" value="1" /> Invented</label>
  </div>

  <div class="idea">
    <h3>Sociology is...</h3>
    <label><input type="radio" name="sociology" value="0" /> Discovered</label>
    <label><input type="radio" name="sociology" value="1" /> Invented</label>
  </div>

  <div class="idea">
    <h3>Language is...</h3>
    <label><input type="radio" name="language" value="0" /> Discovered</label>
    <label><input type="radio" name="language" value="1" /> Invented</label>
  </div>

  <div class="idea">
    <h3>Culture is...</h3>
    <label><input type="radio" name="culture" value="0" /> Discovered</label>
    <label><input type="radio" name="culture" value="1" /> Invented</label>
  </div>

  <div class="idea">
    <h3>Art is...</h3>
    <label><input type="radio" name="art" value="0" /> Discovered</label>
    <label><input type="radio" name="art" value="1" /> Invented</label>
  </div>

  <button type="submit">Submit answers</button>
  <p>or <a href="https://drewdevault.com/discoveries/results">view results</a>
</form>

<section class="footnotes">
  <hr />
  <ol>
    <li id="ft-1">
      Functional, imperative, object-oriented, etc. Remember that these
      paradigms arise from fundamental computer science concepts like Turing
      machines and Lambda Calculus.
      <a href="#ref-1">↩︎</a>
    </li>
  </ol>
</section>


guid: https://drewdevault.com/2021/01/07/History-will-not-remember-us-fondly.html
title: History will not remember us fondly
url: https://drewdevault.com/2021/01/07/History-will-not-remember-us-fondly.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Today, we recall the Middle Ages as an unenlightened time (quite literally, in
fact). We view the Middle Ages with a critical eye towards its brutality, lack
of individual freedoms, and societal and technological regression. But we rarely
turn that same critical lens on ourselves to consider how we&rsquo;ll be perceived by
future generations. I expect the answer, upsetting as it may be, is this: the
future will think poorly of us.</p>
<p>We possess the resources and production necessary to provide every human being
on Earth with a comfortable living: adequate food, housing, health, and
happiness. We have decided not to do so. We have achieved what one may consider
the single unifying goal of the entire history of humanity: we have eliminated
natural scarcity for our basic resources. We have done this, and we choose to
deny our fellow humans their basic needs, in the cruel pursuit of profit. We
have more empty homes than we have homeless people. America alone throws away
enough food to feed the entire world population. And we choose to let our peers
die of hunger and exposure.</p>
<p>We are politically destitute. Profits again drive everything — in the
United States, Citizens United gave corporations unfettered access to buy and
sell political will, and in the time since they have successfully installed
politicians favorable to the elite class. Our corporations possess obscene
wealth, coffers that rival those of nation-states, and rule over our people via
their proxies in political office. Princeton published <a href="https://www.cambridge.org/core/journals/perspectives-on-politics/article/testing-theories-of-american-politics-elites-interest-groups-and-average-citizens/62327F513959D0A304D4893B382B992B">a study</a> in 2014
which showed that the opinions of the average American citizen has a
statistically negligible effect on political outcomes, while the opinions of the
elite can all but decide the same outcomes. Our capitalist owners have
unchallenged rule over society, and they rule it with the single-minded
obsession to create profit at any cost, including lives.</p>
<p>The US Capitol was overrun by armed seditionists yesterday. Armed seditionists,
who, by the way, were radicalized on the internet. As a computer engineer, I am
complicit in this radicalization. The early internet was a sea of optimism, full
of enthusiasm about the growing connectivity between people which had the
potential to unite humanity like never before. We early adopters felt like world
citizens: making friends, collaborating, and uniting with no respect for borders
or ideology.  What we hadn&rsquo;t realized is that we were also building the most
powerful tool the world has ever seen for censorship, propaganda, and
radicalization.</p>
<p>The companies which built this technology are modern slave drivers, broadly
eroding worker freedoms in the first world, and in the third world seeking to
exploit the cheapest slave labor they can find. We are developing technology
which facilitates the authoritarian and <em>genocidal</em> policies of China. Anyone
who speaks out is fired, corrections are quickly issued, and a statement of
unconditional support for the profit generating, population murdering thugs is
proclaimed. I speak passionately to my peers in my field, begging them to fight
back, but many lack the courage, and most don&rsquo;t care — so long as their
exorbitant paychecks keep coming in. Money, money, money. We are at one end of a
process which launders money to wash off the blood. Morals are dead.</p>
<p>It&rsquo;s not just America — democracy is on the decline world-wide. A friend
in France recently took to the streets to protest against the introduction of
laws protecting the police from citizen oversight. Populist traitors tore the
UK out of the EU, effective last week, dooming their people to economic and
political destitution. The Greek economy has failed, right-wingers are passing
discriminatory laws against LGBT Poles, and conservative populism has taken hold
of much of Italy, just to name a few more. Social and political systems are
regressing worldwide.</p>
<a href="https://l.sr.ht/Gd_r.png">
  <img
    style="max-width: 100%"
    alt="A visualization of democratic decline across Europe and Eurasia, showing that declines have outweighed gains in each of the past 10 years, demonstrating a persistent net decline in democracy."
    src="https://l.sr.ht/Gd_r.png" />
</a>
<p><small style="display: block; text-align: center">Source: <a href="https://freedomhouse.org/report/nations-transit/2020/dropping-democratic-facade">Freedom House</a></small></p>
<p>Our entire society boils down to one measure: profit. We are being eaten alive
by capitalism. Americans have been brainwashed into a national ethos which is
<em>defined</em> by capitalism. In the relentless pursuit of profits, we have eroded
all political and social freedoms and created a system defined by its remarkable
cruelty in a time when we have access to greater wealth and resources than at
any other time in history.</p>
<p>Perhaps future generations won&rsquo;t remember us after all, considering that in that
same relentless pursuit of profits we are vigorously rendering the Earth
uninhabitable. But, if they do live to remember us, they will remember us as a
wicked, cruel, and unempathetic lot. We will be remembered in disgrace.</p>


guid: https://drewdevault.com/2021/01/15/Status-update-January-2021.html
title: Status update, January 2021
url: https://drewdevault.com/2021/01/15/Status-update-January-2021.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>
Hello from the future! My previous status update was last year, but it feels
like it was only a month ago. I hope you didn't miss my crappy jokes too much
during the long wait.

<p>
One of the advancements that I would like to mention this month is the general
availability of <a href="https://godocs.io">godocs.io</a>, which is a replacement for the
soon-to-be-obsolete godoc.org, based on a fork of their original codebase.
<a href="https://sr.ht/~sircmpwn/godocs.io">Our fork</a> has already attracted
interest from many contributors who wanted to work on godoc.org, but found the
Google CLA distasteful. We've been hard at work excising lots of Google crap,
rewriting the indexer to use PostgreSQL instead of GCP, and making the little
JavaScript bits more optional &amp; more conservative in their implementation.
We also plan to update it with first-class support for Go modules, which was
never added to the upstream gddo codebase. Beyond this, we do not plan on
making any large-scale changes: we just want godoc.org to keep being a thing.
Enjoy!

<p>
On SourceHut, the first point of note is the new dark theme, which is
automatically enabled when your user-agent configures
<code>prefers-color-scheme: dark</code>. It has gone through a couple of
iterations of refinement, and I have a few more changes queued up for my next
round of improvements. Please let me know if you notice anything unusual!
Additionally, I broke ground on the todo.sr.ht API 2.0 implementation this
month. It required some minor changes to our underlying GraphQL approach, but
in general it should be fairly straightforward &mdash; albeit time consuming
&mdash; to implement. Ludovic has also started working on an API 2.0 branch for
hg.sr.ht, which I plan on reviewing shortly.

<p>
Small projects have enjoyed some improvements as well.
<a href="https://sr.ht/~sircmpwn/mkproof/">mkproof</a> grew multi-processor
support and had its default difficulty tweaked accordingly &mdash; thanks, Tom!
Zach DeCook and Nolan Prescott also sent some bugfixes for
<a href="https://sr.ht/~sircmpwn/gmnisrv/">gmnisrv</a>, and René Wagner and
Giuseppe Lumia both helped fix some issues with
<a href="https://sr.ht/~sircmpwn/gmni">gmni</a> as well. Jason Phan sent an
improvement for <a href="https://sr.ht/~sircmpwn/dowork">dowork</a> which adds
random jitter to the exponential backoff calculation. Thanks to all of these
folks for their help!

<p>
That's all for today. Thanks again for your support and attention, and I'll see
you again soon!

<details>
<summary>...</summary>
<p>I have actually been working on this a lot this month. Progress is good.
<pre>fn measurements() void = {
	const x = "Hello!";
	assert(len(x) == 6z);
	assert(size(str) == size(*u8) + size(size) * 2z);
	const align: size =
		if (size(*u8) &gt; size(size)) size(*u8)
		else size(size);
	assert(&amp;x: uintptr: size % align == 0z);
};

fn charptr() void = {
	const x = "Hello!";
	const y = x: *const char;
	const z = y: *[*]u8;
	const expected = ['H', 'e', 'l', 'l', 'o', '!', '\0'];
	for (let i = 0z; i &lt; len(expected); i += 1z) {
		assert(z[i] == expected[i]: u32: u8);
	};
};

fn storage() void = {
	const string = "こんにちは";
	const ptr = &amp;string: *struct {
		data: *[*]u8,
		length: size,
		capacity: size,
	};
	assert(ptr.length == 15z &amp;&amp; ptr.capacity == 15z);

	// UTF-8 encoded
	const expected = [
		0xE3u8, 0x81u8, 0x93u8, 0xE3u8, 0x82u8, 0x93u8, 0xE3u8, 0x81u8,
		0xABu8, 0xE3u8, 0x81u8, 0xA1u8, 0xE3u8, 0x81u8, 0xAFu8, 0x00u8,
	];
	for (let i = 0z; i &lt; len(expected); i += 1z) {
		assert(ptr.data[i] == expected[i]);
	};
};

export fn main() void = {
	measurements();
	charptr();
	storage();
};</pre>
</details>


guid: https://drewdevault.com/2021/01/19/Spooky-code-at-a-distance.html
title: Spooky action at a distance
url: https://drewdevault.com/2021/01/19/Spooky-code-at-a-distance.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Einstein famously characterized the strangeness of quantum mechanics as &ldquo;spooky
action at a distance&rdquo;, which, if I had to pick one phrase about physics to be my
favorite, would be a strong contender. I like to relate this to programming
language design: there are some language features which are similarly spooky.
Perhaps the most infamous of these is operator overloading. Consider the
following:</p>
<pre><code>x + y
</code></pre><p>If this were written in C, without knowing anything other than the fact that
this code compiles correctly, I can tell you that x and y are numeric types, and
the result is their sum. I can even make an educated guess about the CPU
instructions which will be generated to perform this task. However, if this were
a language with operator overloading&hellip; who knows? What if x and y are some kind
of some Vector class? It could compile to this:</p>
<pre><code>Vector::operator_plus(x, y)
</code></pre><p>The performance characteristics, consequences for debugging, and places to look
for bugs are considerably different than the code would suggest on the surface.
This function call is the &ldquo;spooky action&rdquo; — and the distance between the
&ldquo;+&rdquo; operator and the definition of its behavior is the &ldquo;distance&rdquo;.</p>
<p>Also consider if x and y are strings: maybe &ldquo;+&rdquo; means concatenation?
Concatenation often means allocation, which is a pretty important side-effect to
consider. Are you going to thrash the garbage collector by doing this? <em>Is</em>
there a garbage collector, or is this going to leak? Again, using C as an
example, this case would be explicit:</p>
<pre><code>char *new = malloc(strlen(x) + strlen(y) + 1);
strcpy(new, x);
strcat(new, y);
</code></pre><p>If the filename of the last file you had open in your text editor ended in
<code>.rs</code>, you might be frothing at the mouth after reading this code. Strictly for
the purpose of illustrating my point, however, consider that everything which
happens here is explicit, opt-in to the writer, and obvious to the reader.</p>
<p>That said, C doesn&rsquo;t get off scott-free in this article. Consider the
following code:</p>
<pre><code>int x = 10, y = 20;
int z = add(x, y);
printf(&quot;%d + %d = %d\n&quot;, x, y, z);
</code></pre><p>You may expect this to print out <code>10 + 20 = 30</code>, and you would be forgiven for
your naivety.</p>
<pre><code>$ cc -o test test.c
$ ./test
30 + 20 = 30
</code></pre><p>The savvy reader may have already figured out the catch: add is not a function.</p>
<pre><code>#define add(x, y) x += y
</code></pre><p>The spooky action is the mutation of x, and the distance is between the apparent
&ldquo;callsite&rdquo; and the macro definition. This is spooky because it betrays the
reader&rsquo;s expectations: it looks and smells like a function call, but it does
something which breaks the contract of function calls. Some languages do this
better, by giving macros an explicit syntax like <code>name!(args...)</code>, but,
personally, I still don&rsquo;t like it.</p>
<p>Language features like this are, like all others, a trade-off. But I&rsquo;m of the
opinion that this trade is unwise: you&rsquo;re wagering readability, predictability,
debuggability, and more. These features are toxic to anyone seeking stable,
robust code. They certainly have no place in systems programming.</p>


guid: https://drewdevault.com/2021/01/19/Elasticsearch-does-not-belong-to-Elastic.html
title: Elasticsearch does not belong to Elastic
url: https://drewdevault.com/2021/01/19/Elasticsearch-does-not-belong-to-Elastic.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Elasticsearch belongs to its 1,573 contributors, who retain their copyright, and
granted Elastic a license to distribute their work without restriction. This is
the loophole which Elastic exploited when they decided that Elasticsearch <a href="https://www.elastic.co/blog/licensing-change">would
no longer be open source</a>, a
loophole that they introduced with this very intention from the start. When you
read their announcement, don&rsquo;t be gaslit by their deceptive language: Elastic is
no longer open source, and this is a move against open source. It is not
&ldquo;doubling down on open&rdquo;. <strong>Elastic has spit in the face of every single one of
1,573 contributors, and everyone who gave Elastic their trust, loyalty, and
patronage</strong>. This is an Oracle-level move.</p>
<iframe
  width="560"
  height="315"
  src="https://www.youtube-nocookie.com/embed/-zRN7XLCRhc?start=2483"
  frameborder="0"
  allow="accelerometer; autoplay; gyroscope; picture-in-picture"
  allowfullscreen></iframe>
<p>
<a
  style="display: block; text-align: center"
  href="https://youtu.be/-zRN7XLCRhc?t=2483"
><small>Bryan Cantrill on OpenSolaris &mdash; YouTube</small></a>
<p>Many of those contributors were there because they believe in open source. Even
those who work for Elastic as their employees, who had their copyright taken
from them by their employer, work there because they believe in open source. I
am frequently asked, &ldquo;<a href="https://drewdevault.com/2020/11/20/A-few-ways-to-make-money-in-FOSS.html">how can I get paid to work in open source</a>&rdquo;, and one
of my answers is to recommend a job at companies like Elastic. People seek these
companies out because they want to be involved in open source.</p>
<p>Elastic was not having their lunch eaten by Amazon. They cleared half a billion
dollars last year. Don&rsquo;t gaslight us. Don&rsquo;t call your product &ldquo;free &amp; open&rdquo;,
deliberately misleading users by aping the language of the common phrase &ldquo;free &amp;
open source&rdquo;. You did this to get even more money, you did it to establish a
monopoly over Elasticsearch, and you did it in spite of the trust your community
gave you. Fuck you, Shay Banon.</p>
<p>I hope everyone reading will remember this as yet another lesson in the art of
<a href="https://drewdevault.com/2018/10/05/Dont-sign-a-CLA.html">never signing a CLA</a>.
Open source is a community endeavour. It&rsquo;s a committment to enter your work into
the commons, and to allow the community to collectively benefit from it —
even financially. Many people built careers and businesses out of Elasticsearch,
independently of Elastic, and were entitled to do so under the social contract
of open source. Including Amazon.</p>
<p>You don&rsquo;t own it. Everyone owns it. This is <em>why open source is valuable</em>. If
you want to play on the FOSS playing field, then you play by the goddamn rules.
If you aren&rsquo;t interested in that, then you&rsquo;re not interested in FOSS. You&rsquo;re
free to distribute your software any way you like, including under proprietary
or source-available license terms. But if you choose to make it FOSS, that means
something, and you have the moral obligation to uphold.</p>


guid: https://drewdevault.com/2021/01/20/FOSS-is-to-surrender-your-monopoly.html
title: Open source means surrendering your monopoly over commercial exploitation
url: https://drewdevault.com/2021/01/20/FOSS-is-to-surrender-your-monopoly.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Participation in open source requires you to surrender your monopoly over
commercial exploitation. This is a profound point about free and open source
software which seems to be causing a lot of companies to struggle with their
understanding of the philosophy of FOSS, and it&rsquo;s worth addressing on its own.
It has been apparent for some years now that FOSS is eating the software world,
and corporations are trying to figure out their relationship with it. One fact
that you will have to confront in this position is that you cannot monopolize
the commercial potential of free and open source software.</p>
<p>The term &ldquo;open source&rdquo; is broadly accepted as being defined by the <a href="https://opensource.org/osd">Open Source
Definition</a>, and its very first requirement is the following:</p>
<blockquote>
<p>[The distribution terms of open-source software] shall not restrict any party
from selling or giving away the software as a component of an aggregate
software distribution containing programs from several different sources. The
license shall not require a royalty or other fee for such sale.</p>
</blockquote>
<p>That covers the &ldquo;OSS&rdquo; in &ldquo;FOSS&rdquo;. The &ldquo;F&rdquo; refers to &ldquo;free software&rdquo;, and is
covered by <a href="https://www.gnu.org/philosophy/free-sw.html">this Free Software Foundation resource</a>:</p>
<blockquote>
<p>[A program is free software if the program&rsquo;s users have] the freedom to run
the program as they wish, for any purpose, [&hellip; and to &hellip;] redistribute
copies.</p>
</blockquote>
<p>It further clarifies the commercial aspect of this freedom explicitly:</p>
<blockquote>
<p>“Free software” does not mean “noncommercial”. A free program must be
available for commercial use, commercial development, and commercial
distribution. [&hellip;] Regardless of how you got your copies, you always have the
freedom to copy and change the software, [and] to sell copies.</p>
</blockquote>
<p>This is an essential, non-negotiable requirement of free and open-source
software, and a reality you must face if you want to reap the benefits of the
FOSS ecosystem. <em>Anyone</em> can monetize your code. That includes you, and me, all
of your contributors, your competitors, Amazon and Google, and everyone else.
This is a rejection of how intellectual property typically works —
copyright laws exist for the express purpose of creating an artificial monopoly
for your business, and FOSS licenses exist for the express purpose of breaking
it. If you&rsquo;re new to FOSS, it is going to be totally alien to your understanding
of IP ownership.</p>
<p>It&rsquo;s quite common for people other than you to make money from your free and
open source software works. Some will incorporate them into their own products
to sell, some will develop an expertise with it and sell their skills as a
consultant, some will re-package it in an easy-to-use fashion and charge people
for the service. Others might come up with even more creative ways to monetize
the software, like writing books about it. It will create wealth for everyone,
not just the original authors. And if you want it to create wealth for <em>you</em>,
you are responsible for figuring out how. Building a business requires more work
than just writing the software.</p>
<p>This makes sense in terms of karmic justice, as it were. One of the most
important advantages of making your software FOSS is that the global community
can contribute improvements back to it. The software becomes more than your
organization can make it alone, both through direct contributions to your code,
and through the community which blossoms around it. If the sum of its value is
no longer entirely accountable to your organization, is it not fair that the
commercial exploitation of that value shouldn&rsquo;t be entirely captured by your
organization, either? This is the deal that you make when you choose FOSS.</p>
<p>There are ways that you can influence how others use your FOSS software, mainly
having to do with making sure that everyone else keeps this same promise. You
cannot stop someone from making money from your software, but you <em>can</em> obligate
them to share their improvements with everyone else, which you can incorporate
back into the original product to make it more compelling for everyone. The <a href="https://www.gnu.org/licenses/">GPL
family of licenses</a> is designed for this purpose.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p>
<p>Furthermore, if your business is a consumer of free and open source software,
rather than a producer, you need to be aware that you may be subject to those
obligations. It&rsquo;s not a free lunch: you may be required to return your
improvements to the community. FOSS licenses are important, and you should make
it your business to understand them, both as a user, contributor, and author of
free and open source software.</p>
<p>FOSS is eating the world, and it&rsquo;s a very attractive choice for businesses for a
good reason. This is the reason. It increases wealth for everyone. Capitalism
concerns itself with making monopolies — FOSS instead concerns itself with
the socialized creation of software wealth.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>If you want a brief introduction to GPL licenses, I have written <a href="https://man.sr.ht/license.md">a short guide for SourceHut users</a>. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2021/01/28/Use-open-platforms-or-else.html
title: Use open platforms — or else
url: https://drewdevault.com/2021/01/28/Use-open-platforms-or-else.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>The ongoing events around <a href="https://old.reddit.com/r/wallstreetbets">/r/wallstreetbets</a> teaches us, once again, about
the value of open platforms, and the tremendous <em>risk</em> involved in using
proprietary platforms. The economic elites who control those proprietary
platforms, backed by their venture capital interests, <em>will</em> shut us down if we
threaten them. We&rsquo;re taking serious risk by casting our lot with them.</p>
<p><a href="https://en.wikipedia.org/wiki/Discord_(software)">Discord</a>, a proprietary instant messaging and VoIP platform, kicked out the
/r/WSB community yesterday. They claimed it was due to spam and abuse from bots.
These are convenient excuses when considered in the broader context of Discord&rsquo;s
conflict of interest, between its retail investor users and its wall-street
investor backers. However, even if we take their explanation at face value, we
can easily question Discord&rsquo;s draconian policies about its proprietary chat
protocol. They have a history of cracking down on third-party bots and clients
with the same excuses of preventing spam and abuse. If Discord accepts
responsibility for preventing spam and abuse, then why are they deplatforming
users when they, Discord, failed to prevent it?</p>
<p>It&rsquo;s all a lie. They use a proprietary protocol and crack down on third-party
implementations because they demand total control over their users. They
deplatformed /r/WSB because they were financially threatened by them. Discord
acts in their own interests, including when they are against the interests of
their users. In the words of Rohan Kumar, they&rsquo;re trying to <a href="https://seirdy.one/2021/01/27/whatsapp-and-the-domestication-of-users.html">domesticate their
users</a>. It&rsquo;s the same with every corporate-operated platform. Betting that
Reddit will ultimately shut down /r/WSB is probably a stronger bet than buying
GME!</p>
<p>But there is another way: free and open platforms, protocols, and standards.
Instead of Discord, I could recommend <a href="https://matrix.org">Matrix</a>,
<a href="https://en.wikipedia.org/wiki/Internet_Relay_Chat">IRC</a>, or
<a href="https://www.mumble.info">Mumble</a>. These are not based on central corporate
ownership, but instead on publicly available standards that anyone can build
on top of. The ownership of these platforms is distributed between its users,
and thus aligned with their incentives.</p>
<p>Federation is also a really compelling solution. Unlike Discord and Reddit,
which are centrally owned and operated, federated software calls for many
independent server operators to run instances which are responsible for tens or
hundreds of users each. Each of these servers then use standardized protocols to
communicate with each other, forming one cohesive, distributed social network.
Matrix and IRC are both federated protocols, for example. Others include
<a href="https://joinmastodon.org/">Mastodon</a>, which is similar to Twitter in function;
<a href="https://joinpeertube.org/en">PeerTube</a>, for hosting videos and live streams;
and <a href="https://join.lemmy.ml/">Lemmy</a>, which is a federated equivalent of Reddit.</p>
<p>These are the alternatives. These platforms lack that crucial conflict of
interest which is getting us kicked off of the corporate owned platforms. These
are the facts: open platforms are the only ones align with the interests of
their users, and closed platforms exploit their users. Once you recognize this,
you should jump ship <em>before</em> you&rsquo;re deplatformed, or else you&rsquo;re risking your
ability to organize yourselves to move to another platform. Use open platforms —
or else. Do it today.</p>


guid: https://drewdevault.com/2021/02/09/How-to-make-your-downstreams-happy.html
title: How to make your downstream users happy
url: https://drewdevault.com/2021/02/09/How-to-make-your-downstreams-happy.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>There are a number of things that your FOSS project can be doing which will make
the lives of your downstream users easier, particularly if you&rsquo;re writing a
library or programmer-facing tooling. Many of your downstreams (Linux distros,
pkgsrc, corporate users, etc) are dealing with lots of packages, and some minor
tweaks to your workflow will help them out a lot.</p>
<p>The first thing to do is <em>avoid</em> using any build system or packaging system
which is not the norm for your language. Also avoid incorporating information
into your build which relies on being in your git repo — most packagers
prefer to work with tarball snapshots, or to fetch your package from e.g. PyPI.
These two issues are definitely the worst offenders. If you do have to use a
custom build system, take your time to document it thoroughly, so that users who
run into problems are well-equipped to address them. The typical build system or
packaging process in use for your language already addressed most of those edge
cases long ago, which is why we like it better. If you must fetch, say, version
information from git, then please add a fallback, such as an environment
variable.</p>
<p>Speaking of environment variables, another good one to support is
<a href="https://reproducible-builds.org/docs/source-date-epoch/">SOURCE_DATE_EPOCH</a>,
for anything where the current date or time is incorporated into your build
output. Many distros are striving for <em>reproducible</em> builds these days, which
involves being able to run a build twice, or by two independent parties, and
arrive at an identical checksum-verifiable result. You can probably imagine some
other ways to prevent issues here — don&rsquo;t incorporate the full path to
each file in your logs, for instance. There are more recommendations on the
website linked earlier.</p>
<p>Though we don&rsquo;t like to rely on it as part of the formal packaging process, a
good git discipline will also help us with the informal parts. You may already
be using <a href="https://git-scm.com/docs/git-tag">git tags</a> for your releases — consider putting a changelog
into your annotated tags (git tag -a). If you have <a href="https://drewdevault.com/2019/02/25/Using-git-with-discipline.html">good commit discipline</a>
in your project, then you can easily use <a href="https://git-scm.com/docs/git-shortlog">git shortlog</a> to generate such a
changelog from your commit messages. This helps us understand what we can expect
when upgrading, which helps incentivize us to upgrade in the first place. In
<a href="https://drewdevault.com/2019/10/12/how-to-fuck-up-releases.html">How to fuck up software releases</a>, I wrote about my <a href="https://git.sr.ht/~sircmpwn/dotfiles/tree/master/bin/semver">semver</a> tool, which
you may find helpful in automating this process. It can also help you avoid
forgetting to do things like update the version number somewhere in the code.</p>
<iframe src="https://asciinema.org/a/nzBvuMXjUMsoewLnrbTm7E28O/embed?" id="asciicast-iframe-nzBvuMXjUMsoewLnrbTm7E28O" name="asciicast-iframe-nzBvuMXjUMsoewLnrbTm7E28O" scrolling="no" allowfullscreen="true" style="overflow: hidden; border: 0px; width: 540px; float: none; visibility: visible; height: 404px;"></iframe>
<p>In short, to make your downstreams happy:</p>
<ol>
<li>Don&rsquo;t rock the boat on builds and packaging.</li>
<li>Don&rsquo;t expect your code to always be in a git repo.</li>
<li>Consider reproducible builds.</li>
<li>Stick a detailed changelog in your annotated tag — which is easy if you
have good commit discipline.</li>
</ol>
<p>Overall, this is pretty easy stuff, and good practices which pay off in other
respects as well. Here&rsquo;s a big &ldquo;thanks&rdquo; in advance from your future downstreams
for your efforts in this regard!</p>


guid: https://drewdevault.com/2021/02/15/Status-update-February-2021.html
title: Status update, February 2021
url: https://drewdevault.com/2021/02/15/Status-update-February-2021.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>
Salutations! It's officially a year of pandemic life. I hear the vaccine
distribution is going well, so hopefully there won't be another year of this. In
the meanwhile, I've been working hard on free software, what with having little
else to do. However, I'm afraid I cannot tell you about most of it!

<p>
I've been working on todo.sr.ht's GraphQL API, and it's going quite well. I hope
to ship a working read-only version later this month. There have been a number
of bug fixes and rote maintenance work on sr.ht as well, but nothing
particularly exciting. We did upgrade everything for Alpine 3.13, which went off
without a hitch. Anyway, I'll go over the minor details in the sr.ht "what's
cooking" post later today.

<p>
The rest of the progress was made in secret. Secret! You will have to live in
ignorance for now. Sorry!

<details>
<summary>(unless you click this)</summary>
<p>Here's a peek at our progress:</p>
<pre>
use fmt;
use io;
use os;

export fn main() void = {
        if (len(os::args) == 1) match (io::copy(os::stdout, os::stdin)) {
                err: io::error =&gt; fmt::fatal("Error copying &lt;stdin&gt;: {}",
                        io::errstr(err)),
                size =&gt; return,
        };

        for (let i = 1z; i &lt; len(os::args); i += 1) {
                let f = match (os::open(os::args[i], io::mode::RDONLY)) {
                        s: *io::stream =&gt; s,
                        err: io::error =&gt; fmt::fatal("Error opening {}: {}",
                                os::args[i], io::errstr(err)),
                };
                defer io::close(f);

                match (io::copy(os::stdout, f)) {
                        err: io::error =&gt; fmt::fatal("Error copying {}: {}",
                                os::args[i], io::errstr(err)),
                        size =&gt; void,
                };
        };
};
</pre>
<p>
I'm looking for a few volunteers to get involved and help flesh out the standard
library. If you are interested, please email
<a href="mailto:sir@cmpwn.com">sir@cmpwn.com</a> to express your interest,
along with your sr.ht username and a few words about your systems programming
experience &mdash; languages you're comfortable with, projects you've worked
on, platforms you grok, etc.
</p>
</details>


guid: https://drewdevault.com/2021/02/21/On-the-traits-of-good-replacements.html
title: A great alternative is rarely fatter than what it aims to replace
url: https://drewdevault.com/2021/02/21/On-the-traits-of-good-replacements.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>This is not always true, but in my experience, it tends to hold up. We often
build or evaluate tools which aim to replace something kludgy^Wvenerable.
Common examples include shells, programming languages, system utilities, and so
on. Rust, Zig, etc, are taking on C in this manner; so too does zsh, fish, and
oil take on bash, which in turn takes on the Bourne shell. There are many
examples.</p>
<p>All of these tools are fine in their own respects, but they have all failed to
completely supplant the software they&rsquo;re seeking to improve upon.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> What these
projects have in common is that they <em>expand</em> on the ideas of their
predecessors, rather than <em>refining</em> them. A truly great alternative finds the
nugget of truth at the center of the idea, cuts out the cruft, and solves the
same problem with less.</p>
<p>This is one reason I like Alpine Linux, for example. It&rsquo;s not really aiming to
replace any distro in particular so much as it competes with the Linux ecosystem
as a whole. Alpine does this by being <em>simpler</em> than the rest: it&rsquo;s the only
Linux system I can fit more or less entirely in my head. Compare this to the
most common approach: &ldquo;let&rsquo;s make a Debian derivative!&rdquo; It kind of worked for
Ubuntu, less so for everyone else. The C library Alpine ships, <a href="https://musl.libc.org">musl
libc</a>, is another example: it aims to replace glibc by
being leaner and meaner, <a href="https://drewdevault.com/2020/09/25/A-story-of-two-libcs.html">and I&rsquo;ve talked about its success in this respect
before</a>.</p>
<p>Go is a programming language which has done relatively well in this respect. It
aimed to fill a bit of a void in the high-performance internet infrastructure
systems programming niche,<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> <sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> and it is markedly simpler than most of the
other tools in its line of work. It takes the opportunity to add a few
innovations — its big risk is its novel concurrency model — but Go
balances this with a level of simplicity in other respects which is unchallenged
among its contemporaries,<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup> and a commitment to that simplicity which has
endured for years.<sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup></p>
<p>There are many other examples. UTF-8 is a simple, universal approach which
smooths over the idiosyncrasies of the encoding zoo which pre-dates it, and has
more-or-less rendered its alternatives obsolete. JSON has almost completely
replaced XML, and its grammar famously fits on a business card.<sup id="fnref:6"><a href="#fn:6" class="footnote-ref" role="doc-noteref">6</a></sup> On the other
hand, when zsh started as a superset of bash, it crippled its ability to compete
on &ldquo;having less warts than bash&rdquo;.</p>
<p>Rust is more vague in its inspirations, and does not start as a superset of
anything. It has, however, done a poor job of scope management, and is
significantly more complex than many of the languages it competes with, notably
C and Go. For this reason, it struggles to root out the hold-outs in those
domains, and it suffers for the difficulty in porting it to new platforms, which
limits its penetration into a lot of domains that C is still thriving in.
However, it succeeds in being much simpler than C++, and I expect that it will
render C++ obsolete in the coming years as such.<sup id="fnref:7"><a href="#fn:7" class="footnote-ref" role="doc-noteref">7</a></sup></p>
<p>In computing, we make do with a hodge podge of hacks and kludges which, at best,
approximate the solutions to the problems that computing presents us. If you
start with one such hack as the basis of a supposed replacement and build <em>more</em>
on top of it, you will inherit the warts, and you may find it difficult to rid
yourself of them. If, instead, you question the premise of the software,
interrogate the underlying problem it&rsquo;s trying to solve, and apply your
insights, plus a healthy dose of hindsight, you may isolate what&rsquo;s right from
what&rsquo;s superfluous, and your simplified solution just might end up replacing the
cruft of yore.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Some of the listed examples have not given up and would prefer that I say something to the effect of &ldquo;but the jury is still out&rdquo; here. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>That&rsquo;s a lot of adjectives! <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>More concisely, I think of Go as an &ldquo;internet programming language&rdquo;, distinct from the systems programming languages that inspired it. Its design shines especially in this context, but its value-add is less pronounced for other tasks in the systems programming domain - compilers, operating systems, etc. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p><a href="https://golang.org/ref/spec">The Go spec</a> is quite concise and has changed very little since Go&rsquo;s inception. Go is also unique among its contemporaries for (1) writing a spec which (2) supports the development of multiple competing implementations. <a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>Past tense, unfortunately, now that Go 2 is getting stirred up. <a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>It is possible that JSON has achieved <em>too much</em> success in this respect, as it has found its way into a lot of use-cases for which is less than ideal. <a href="#fnref:6" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>Despite my infamous distaste for Rust, long-time readers will know that where I have distaste for Rust, I have passionate scorn for C++. I&rsquo;m quite glad to see Rust taking it on, and I hope very much that it succeeds in this respect. <a href="#fnref:7" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2021/02/25/Gmail-is-a-huge-source-of-spam.html
title: Gmail is a huge source of spam
url: https://drewdevault.com/2021/02/25/Gmail-is-a-huge-source-of-spam.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>5× as many spam registrations on sourcehut are from gmail than from the
second-largest offender.</p>
<pre><code># SELECT
  SPLIT_PART(email, '@', 2) as domain, count(*) as count
  FROM &quot;user&quot;
  WHERE user_type = 'suspended'
  GROUP BY domain
  ORDER BY count DESC;
          domain           | count
---------------------------+-------
 gmail.com                 |   119
 qq.com                    |    26
 mail.ru                   |    17
 mailinator.com            |    10
 yopmail.com               |     6
 aol.com                   |     6
 yahoo.com                 |     6
[...more omitted...]
</code></pre><p>This is just the ones which got through: most spam registrations are detected
and ignored before they make it to the database.</p>
<p>A huge number of spam emails I recieve in my personal inbox originate from
@gmail.com, and often they arrive in my inbox unscathed (as opposed to going to
Junk) because Gmail is considered a reputable mail provider. My colleague
estimates that between 15% and 25% of the spam emails sent to a mailing list he
administrates comes from Gmail.</p>
<p>One might argue that, because Gmail is the world&rsquo;s largest email provider, it&rsquo;s
natural to expect that they would have the largest volume of spam simply because
they have proportionally more users who might use it for spam. I would argue
that this instead tells us that they have the largest responsibility to curtail
spam on their platform.</p>
<p>I&rsquo;ve forwarded many, many reports to <a href="mailto:abuse@gmail.com">abuse@gmail.com</a>, but they&rsquo;ve never followed
up and the problem has not become any better. I have had half a mind to block
Gmail registrations on sourcehut outright, but about 41% of all registrations
use Gmail.</p>
<p>It bears repeating that anyone with any level of technical expertise ought to
know better than to use Gmail. I usually recommend
<a href="https://www.migadu.com">Migadu</a><sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>, but there are many options to choose from.
If you&rsquo;re worried about mail deliverability issues, don&rsquo;t be — it&rsquo;s more
or less a myth in $CURRENTYEAR. If you set up
<abbr title="DomainKeys Identified Mail, an means of verifying message authenticity">DKIM</abbr>
properly and unlist your IP address from the
<abbr title="DNS blocklists">DNSBL</abbr>s (a simple process), then your mails
will get through.</p>
<p>In case you&rsquo;re wondering, the dis-award for second-worst goes to Amazon SES.
They don&rsquo;t register on sourcehut (it&rsquo;s outgoing only, so that makes sense), but
I see them often in my personal inbox. However, SES only appears at a rate of
about a tenth of the gmail spam, and they appear to actually listen to my abuse
reports, so I can more or less forgive them for it.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Full disclosure: sourcehut has a business relationship with Migadu, though I&rsquo;ve recommended them since long before we met. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2021/03/03/To-make-money-in-FOSS-build-a-business.html
title: To make money in FOSS, build a business first
url: https://drewdevault.com/2021/03/03/To-make-money-in-FOSS-build-a-business.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I&rsquo;ve <a href="https://drewdevault.com/2020/11/20/A-few-ways-to-make-money-in-FOSS.html">written about</a> making money in free and open source software before,
but it&rsquo;s a deep topic that merits additional discussion. While previously I
focused on what an individual can do in order to build a career in FOSS, but
today I want to talk about how you can build a sustainable business in FOSS.</p>
<p>It&rsquo;s a common mistake to do this the wrong way around: build the software, then
the business. Because FOSS <a href="https://drewdevault.com/2021/01/20/FOSS-is-to-surrender-your-monopoly.html">requires you to surrender your sole monetization
rights</a>, building the software first and worrying about the money later puts
you at a huge risk of losing your first-mover advantage. If you&rsquo;re just making a
project which is useful to you and you don&rsquo;t want the overhead of running a
business, then that may be totally okay — you can just build the software
without sweating the business issues. If you choose this path, however, be aware
that the promise of free and open source software entitles anyone else to build
that business without you. If you lapse in your business-building efforts and
your software project starts making someone else money, then they&rsquo;re not at
fault for &ldquo;taking your work&rdquo; — you gave it to them.</p>
<p>I&rsquo;ve often said that you can make money in FOSS, but not usually by accident.
Don&rsquo;t just build your project and wait for the big bucks to start rolling in.
You need to take the business-building seriously from the start. What is the
organization of your company? Who will you work with? What kind of clients or
customers will you court? Do you know how to reach them? How much they&rsquo;re
willing to pay? What you will sell? Do you have a budget? If you want to make
money from your project, sit down and answer these questions seriously.</p>
<p>Different kinds of software projects make money in different ways. Some projects
with enterprise-oriented software may be able to sell support contracts. Some
can sell consultants to work on integration and feature development. Maybe you
can write books about your software, or teach courses on it. Perhaps your
software, like the kind <a href="https://sourcehut.org">my company</a> builds, is
well-suited to being sold as a service. Some projects simply solicit donations,
but this is the most difficult approach.</p>
<p>Whatever you choose to do, you need to choose it deliberately. You need to
incorporate your business, hire an accountant, and do a lot of boring stuff
which has nothing to do with the software you want to write. And if you skip
this step, someone else is entitled to do all of this boring work, then stick
your software on top of it and make a killing without you.</p>


guid: https://drewdevault.com/2021/03/06/Corporate-surveillance-murder.html
title: The corporate surveillance machine is killing people
url: https://drewdevault.com/2021/03/06/Corporate-surveillance-murder.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I have never been angrier about the corporate surveillance complex, which I have
rallied against for <em>years</em>, than I am today. Buying and selling user&rsquo;s private
information on the open market is bad enough for the obvious reasons, but today,
I learned that the depths of depravity this market will descend to are without
limit. Today I am more angry and ashamed at this industry than I have ever been.
Corporate surveillance and adtech has turned your phone into an informant
against you and brought about the actual <strong>murder</strong> of the user.</p>
<p><a href="https://www.vice.com/en/article/y3g97x/location-data-apps-drone-strikes-iowa-national-guard">Vice: Military Unit That Conducts Drone Strikes Bought Location Data From Ordinary Apps</a></p>
<p>Say you&rsquo;re a Muslim. You download some apps for reading the Quran and
participating in Muslim-oriented social networks. These ads steal whatever
personal information it can get its hands on, through any means available, and
sell it to <a href="https://trademarks.justia.com/874/53/locate-87453515.html">Locate X</a>, who stores every GPS location your phone has visited
and tags it as being associated with a Muslim. This is used, say, to place
Muslim-targeted ads on billboards in Muslim-dense areas. It&rsquo;s also sold to the
Iowa National Guard, who uses it to conduct drone strikes. The app you installed
is selling your GPS data so it can be used to <em>kill you</em>.</p>
<p>For a long time, I have preached &ldquo;respect the user&rdquo;. I want us, as programmers,
to treat the user with the same standards of common decency and respect we&rsquo;d
afford to our neighbors. It seems I have to revise my sermon to &ldquo;don&rsquo;t murder
the user&rdquo;! If you work at a company which surveilles its users, <a href="https://drewdevault.com/2020/05/05/We-are-complicit-in-our-employers-deeds.html">you are
complicit</a> in these murders. You have written software which is used to
<em>murder people</em>.</p>
<p>This industry is in severe need of a moral health check. You, the reader of
this article, need to take personal responsibility for what your code is doing.
Your boss isn&rsquo;t going to. Do you really know what that database is being used
for, or who it&rsquo;s being sold to, or who it <em>might</em> be sold to in the future? Most
companies include their hoard of private, personal information about their users
as part of their valuation. Do you have stock options, by the way?</p>
<p>I&rsquo;ve often heard the excuse that employees of large surveillance companies &ldquo;want
to feed their families, like anyone else&rdquo;. Well, thanks to your work, a child
you&rsquo;ve never met was orphaned, and doesn&rsquo;t have a family anymore. Who&rsquo;s going to
feed them? Is there really no other way for you to support your family?</p>
<p>Don&rsquo;t fucking kill your users.</p>


guid: https://drewdevault.com/2021/03/15/Status-update-March-2021.html
title: Status update, March 2021
url: https://drewdevault.com/2021/03/15/Status-update-March-2021.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>After the brief illusion of spring, this morning meets us with a cold apartment
indoors and fierce winds outdoors. Today concludes a productive month, mainly
for the secret project and for sourcehut, but also marked by progress in some
smaller projects as well. I&rsquo;ll start with those smaller projects.</p>
<p>I have written a feed reader for Gemini, which is (1) <a href="https://sr.ht/~sircmpwn/gemreader">free
software</a>, and (2) <a href="gemini://feeds.drewdevault.com">available as a free
hosted service</a>. Big thanks to adnano, the
author of the <a href="https://sr.ht/~adnano/go-gemini">go-gemini</a> library, which has
been very helpful for many of my Gemini-related exploits, and who has been a
great collaborator. I also used it to provide Gemini support for the new
<a href="https://srht.site">pages.sr.ht</a>, which offers static web and gemini hosting for
sr.ht users. I also updated <a href="https://sr.ht/~sircmpwn/gmni">gmni</a> to use BearSSL
instead of OpenSSL this month.</p>
<p><a href="https://godocs.io">godocs.io</a> has been enjoying continued improvements, mainly
thanks again to adnano. Heaps of obsolete interfaces and cruft have been
excised, not only making it lighter for godocs.io, but also making our <a href="https://git.sr.ht/~sircmpwn/gddo">gddo
fork</a> much easier for you to run yourself.
Adnan hopes to have first-class support for Go modules working soon, which will
bring us up to feature parity with pkg.go.dev.</p>
<p>There&rsquo;s some sourcehut news as well, but I&rsquo;ll leave that for the &ldquo;What&rsquo;s
cooking&rdquo; later today. Until next time!</p>
<details>
  <summary>...</summary>
<p>Progress on the secret project has been phenomenal. In the last month, the
standard library has doubled in size, and this weekend, we finished the
self-hosted build driver. We are about 1,000 lines of code shy of having more
code written in <span style="background: black;">xxxx</span>
than in C. Here&rsquo;s the build driver compiling and running itself several times:</p>
<pre>
$ <span style="background: black;">    </span> run ./cmd/<span style="background: black;">    </span> run ./cmd/<span style="background: black;">    </span> run -h
run: compiles and runs <span style="background: black;">    </span> programs

Usage: run [-v]
	 [-D &lt;ident:type=value&gt;]
	 [-j &lt;jobs&gt;]
	 [-l &lt;name&gt;]
	 [-T &lt;tags...&gt;]
	 [-X &lt;tags...&gt;]
	 path args...

-v: print executed commands
-D &lt;ident:type=value&gt;: define a constant
-j &lt;jobs&gt;: set parallelism for build
-l &lt;name&gt;: link with a system library
-T &lt;tags...&gt;: set build tags
-X &lt;tags...&gt;: unset build tags</pre>
<p>The call for help last month was swiftly answered, and we have 7 or 8 new
people working on the project now. We&rsquo;ve completed enough work to unblock many
workstreams, which will allow these new contributors to work in parallel on
different areas of interest, which should substantially speed up progress.</p>
</details>


guid: https://drewdevault.com/2021/03/19/A-new-systems-language.html
title: We are building a new systems programming language
url: https://drewdevault.com/2021/03/19/A-new-systems-language.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>It&rsquo;s an open secret: the &ldquo;secret project&rdquo; I&rsquo;ve been talking about is a new
systems programming language. It&rsquo;s been underway since December &lsquo;19, and we hope
to release the first version in early 2022. The language is pretty small —
we have a mostly complete specification which clocks in at 60 pages. It has
manual memory management, no runtime, and it uses a superset of the C ABI,
making it easy to link with libraries and C code. It should be suitable almost
anywhere C is useful: compilers, system utilities, operating systems, network
servers and clients, and so on.</p>
<!--
Hello, clever reader :) I was too lazy to make a dedicated build of Hugo with a
fake name for the language. Since you bothered to check, head to
https://harelang.org for the rest of the details. Please keep it to yourself!
-->
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="cp">use io;</span>

<span class="k">export</span> <span class="k">fn</span> <span class="n">main</span><span class="p">()</span> <span class="kt">void</span> <span class="o">=</span> <span class="p">{</span>
	<span class="k">const</span> <span class="n">greetings</span> <span class="o">=</span> <span class="p">[</span>
		<span class="s">&#34;Hello, world!&#34;</span><span class="p">,</span>
		<span class="s">&#34;¡Hola Mundo!&#34;</span><span class="p">,</span>
		<span class="s">&#34;Γειά σου Κόσμε!&#34;</span><span class="p">,</span>
		<span class="s">&#34;Привет мир!&#34;</span><span class="p">,</span>
		<span class="s">&#34;こんにちは世界！&#34;</span><span class="p">,</span>
	<span class="p">];</span>
	<span class="k">for</span> <span class="p">(</span><span class="k">let</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="n">z</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="k">len</span><span class="p">(</span><span class="n">greetings</span><span class="p">);</span> <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
		<span class="n">io</span><span class="o">::</span><span class="n">println</span><span class="p">(</span><span class="n">greetings</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span>
	<span class="p">};</span>
<span class="p">};</span>
</code></pre></div><p>We could compare our language to many other languages, but let&rsquo;s start with how
it compares to C:</p>
<ul>
<li>More robust error handling via tagged unions</li>
<li>Improved, Unicode-aware string support</li>
<li>Memory safe array, slice, and pointer types (and unsafe versions, if needed)</li>
<li>Direct compatibility with the C ABI for trivial C interop</li>
<li>A simpler, context-free, expression-oriented syntax</li>
<li>A standard library free of the constraints of POSIX or the C standard</li>
</ul>
<p>Our language currently supports Linux on x86_64 or aarch64, and we plan on
expanding this to the BSDs, Haiku, and Plan 9; as well as i686, riscv64 and
riscv32, and ppc64 before the release.</p>
<p>I plan to continue keeping the other details a secret until the release —
we want the first release to be a complete, stable, production-ready programming
language with all of the trimmings. The first time most people will hear about
this language will also be the first time they can ship working code with it.</p>
<p>However, if you want to get involved sooner, there&rsquo;s a way: we need your help.
So far, we&rsquo;ve written most of the spec, the first of two compilers, and about
15,000 lines of the standard library. The standard library is what needs the
most help, and I&rsquo;m seeking volunteers to get involved.</p>
<p>The standard library mandate begins with the following:</p>
<blockquote>
<p>The <span style="color: transparent">xxxx</span> standard library shall provide:</p>
<ol>
<li>Useful features to complement <span style="color: transparent">xxxx</span> language features</li>
<li>An interface to the host operating system</li>
<li>Implementations of broadly useful algorithms</li>
<li>Implementations of broadly useful formats and protocols</li>
<li>Introspective meta-features for <span style="color: transparent">xxxx</span>-aware programs</li>
</ol>
<p>Each of these services shall:</p>
<ol>
<li>Have a concise and straightforward interface</li>
<li>Correctly and completely implement the useful subset of the required behavior</li>
<li>Provide complete documentation for each exported symbol</li>
<li>Be sufficiently tested to provide confidence in the implementation</li>
</ol>
</blockquote>
<p>We have a number of focus areas for standard library development. I expect most
contributors, at least at first, to stick to one or two of these areas. The
focus areas we&rsquo;re looking into now are:</p>
<dl>
  <dt>Algorithms</dt>
  <dd>Sorting • compression • math • etc</dd>
  <dt>Cryptography</dt>
  <dd>Hashing • encryption • key derivation • TLS • etc</dd>
  <dt>Date & time support</dt>
  <dd>Parsing • formatting • arithmetic • timers • etc</dd>
  <dt>Debugging tools</dt>
  <dd>ELF and DWARF support • vDSO • dynamic loading • etc</dd>
  <dt>Formats & encodings</dt>
  <dd>JSON • XML • HTML • MIME • RFC 2822 • tar • etc</dd>
  <dt><span style="color: transparent">xxxx</span> language support</dt>
  <dd>Parsing • type checker • hosted toolchain • etc</dd>
  <dt>Networking</dt>
  <dd>IP & CIDR handling • sockets • DNS resolver • HTTP • etc</dd>
  <dt>Platform support</dt>
  <dd>New platforms and architectures • OS-specific features</dd>
  <dt>String manipulation</dt>
  <dd>Search, replace • Unicode • Regex • etc</dd>
  <dt>Unix support</dt>
  <dd>chmod • mkfifo • passwd • setuid • TTY management • etc</dd>
</dl>
<p>If any of this sounds up your alley, we&rsquo;d love your help! Please <a href="mailto:sir@cmpwn.com">write me an
email</a> describing your interest areas and previous systems
programming experience.</p>
<p><strong>Update 2021-03-20</strong>: We&rsquo;re targeting the first release in early 2022, not 2021.</p>


guid: https://drewdevault.com/2021/03/23/Open-sourcing-video-games.html
title: The complete guide for open sourcing video games
url: https://drewdevault.com/2021/03/23/Open-sourcing-video-games.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Video games are an interesting class of software. Unlike most software, they are
a creative endeavour, rather than a practical utility. Where most software
calls for new features to address practical needs of their users, video games
call for new features to serve the creative vision of their makers. Similarly,
matters like refactoring and paying down tech debt are often heavily
de-prioritized in favor of shipping something ASAP. Many of the collaborative
benefits of open source are less applicable to video games. It is perhaps for
these reasons that there are very few commercial open source games.</p>
<p>However, there are some examples of such games, and they have had a great deal
of influence on gaming.  Id is famous for this, having released the source code
for several versions of DOOM. The Quake engine was also released under the GPL,
and went on to be highly influential, serving as the basis for dozens of games,
including time-honored favorites such as the Half Life series. Large swaths of
the gaming canon were made possible thanks to the generous contributions of open
source game publishers.</p>
<p>Publishing open source games is also a matter of historical preservation.
Proprietary games tend to atrophy. Long after their heyday, with suitable
platforms scarce and physical copies difficult to obtain, many games die a slow
and quiet death, forgotten to the annals of time. Some games have overcome this
by releasing their source code, making it easier for fans to port the game to
new platforms and keep it alive.</p>
<p>What will your game&rsquo;s legacy be? Will it be forgotten entirely, unable to run on
contemporary platforms? Will it be source-available, occasionally useful to the
devoted player, but with little reach beyond? Perhaps it goes the way of DOOM,
living forever in ports to hundreds of devices and operating systems. Maybe it
goes the way of Quake, its soul forever a part of the beloved classics of the
future. If you keep the source code closed, the only conclusion is the first:
enjoyed once, now forgotten.</p>
<p>With this in mind, how do you go about securing your game&rsquo;s legacy?</p>
<h2 id="source-available-the-bare-minimum">Source available: the bare minimum</h2>
<p>The bare minimum is to make your game &ldquo;source available&rdquo;. Be aware that this is
not the same thing as making it open source! Some of your famous peers in this
category include Alien 3, Civilization IV and V, Crysis, Deus Ex, Prince of
Persia, Unreal Tournament, and VVVVVV.</p>
<p>This approach makes your source code available to view and perhaps to compile
and run, but prohibits derivative works. This is definitely better than leaving
it closed source: it provides helpful resources for modders, speedrunners, and
other fans; and devoted players may be able to use it as the basis for getting
the game running on future platforms, albeit alone and unable to share their
work.</p>
<p>If you choose a minimal enforcement approach, then some players might ultimately
share their work, but you&rsquo;re leaving them on tenuous legal grounds. I would
recommend this if you&rsquo;re very protective of your IP, but know that you&rsquo;re
limiting the potential second life of your game if you take this approach.</p>
<h2 id="copyleft-with-proprietary-assets">Copyleft with proprietary assets</h2>
<p>The next step up is to make your game open source using a <em>copyleft</em> license,
but refraining from extending the license to the assets — anyone who wants
to get the source code working would either need to buy the game from you and
extract the assets, or supply their own community-made assets. This is a popular
approach among open source games, and gives you most of the benefits and few of
the drawbacks. You&rsquo;ll join the ranks of our DOOM and Quake examples, as well as
Amnesia: the Dark Descent, System Shock, Duke Nukem 3D, and Wolfenstein 3D.</p>
<p>Games like this enjoy a long life as their software is more easily ported to new
platforms and shared with other users. DOOM runs on phones, digital cameras,
ATMs, even toasters! Its legacy is secure without any ongoing commitment from
the original developers. This also allows derivatives works — new games
based on your code — though it may turn some developers away. Using a
copyleft license like the <a href="https://www.gnu.org/licenses/gpl-3.0.en.html">GPL</a>
requires derivative works to <em>also</em> be made open source. The community generally
has no problem with this, but it may affect the willingness of future developers
to incorporate your work into their own commercial games. I personally think
that the proliferation of open source software that&rsquo;s implied in the use of a
copyleft license is a positive thing — but you may want to use another
approach.</p>
<h2 id="permissive-license-proprietary-assets">Permissive license, proprietary assets</h2>
<p>If you want to allow your source code to find its way into as many future games
as possible, a permissive open source license like <a href="https://opensource.org/licenses/MIT">MIT</a> is the way to go.
<a href="https://github.com/blendogames/flotilla">Flotilla</a> is an example of a game
which went with this approach. It allows developers to incorporate your source
code into their own games with little restriction, either by creating a direct
derivative, or by taking little samples of your code and incorporating it into
their own project. This comes with no obligation to release their own changes or
works in a similar fashion: they can just take it, with very few strings
attached. Such an approach makes it very easy to incorporate into new commercial
games.</p>
<p>This is the most selfless way to release your code. I would recommend this if
you don&rsquo;t care about what happens to your code later, and you just want to make
it open source and move on. Though this will definitely enable the largest
number of future projects to make use of your work, the copyleft approach is
better for ensuring that the largest possible number of future games are <em>also</em>
open source.</p>
<h2 id="open-assets">Open assets</h2>
<p>If you&rsquo;re feeling especially generous, you could release the assets, too. Good
licenses for this includes the <a href="https://creativecommons.org/">Creative Commons</a>
licenses. All of them permit free redistribution of your assets, so future
players won&rsquo;t have to buy your game to get them. This could be important if the
distribution platform you used is defunct, or if you&rsquo;re not around to buy it
from — consider this well before deciding that you&rsquo;d rather keep your
share of the dwindling asset sales as your game ages.</p>
<p>Using Creative Commons also allows you to tune the degree to which your assets
may be re-used. You can choose different CC licenses to control the
commercialization of your assets and use in derivative works. To allow free
redistribution and nothing else, the CC-NC-ND license (noncommercial, no
derivatives) will do the trick. The CC-BY-SA license is the copyleft of creative
commons: it will allow free redistribution, commercialization, and derivative
works, <em>if</em> the derivatives are also shared with the same rights. The permissive
approach is CC-0, which is equivalent to releasing your assets into the public
domain.</p>
<p>Permitting derivatives and re-commercialization of your assets can save a lot of
time for new game developers, especially indie devs with a small budget. It&rsquo;s
also cool for making derivative <em>games</em>, similar to modding, where creative
players can remix your assets to make a new game or expansion pack.</p>
<h2 id="what-if-i-dont-completely-own-my-game">What if I don&rsquo;t completely own my game?</h2>
<p>You can&rsquo;t give away the rights to anything you don&rsquo;t own. If you rely on
proprietary libraries, or a third-party level editor, or you don&rsquo;t own the
rights to the music or sprites, you cannot make them open source.</p>
<p>In this situation, I recommend open sourcing everything that you&rsquo;re able to.
This might mean that you open source an ultimately broken game — it simply
might not work, or not even compile, without these resources. This is
unfortunate, but by releasing everything you can, you leave your community in a
good position to fill in the gaps themselves, perhaps by refactoring your code
to work around them, or by replacing the proprietary bits with free
alternatives. This also allows the parts of your game which are open to be
reused in future games.</p>
<h2 id="but-cheaters-could-use-it">But cheaters could use it!</h2>
<p>This is true. And it&rsquo;s worth noting that if your game has a mandatory online
component based on your own servers, then making it open source doesn&rsquo;t make
nearly as much sense, especially if you ultimately decide to shut those servers
off.</p>
<p>There is a trade-off to be made here. In truth, it&rsquo;s very difficult to prevent
cheating in your game. If you&rsquo;ve made a popular competitive multiplayer game,
you and I both know that there are still cheaters using it despite your best
efforts. Keeping it proprietary is not going to stave off cheaters. Social
solutions are better — like a system to report cheaters, or to let friends
play on private servers.</p>
<p>Making your game open source might help less skilled script kiddie figure out
how to cheat more easily in your game. I can&rsquo;t decide for you if the trade-off
is worth it for your game, but I can tell you that the benefits of making it
open are vast, and the efficacy of keeping it closed to prevent cheating is
questionable.</p>
<h2 id="but-my-code-is-embarrassing">But my code is embarrassing!</h2>
<p>So is everyone else&rsquo;s. 🙂 We all know that games are running up against tight
deadlines and clean code is not going to be the #1 priority. I assure you that
your community will be too busy having fun to judge you for the quality of your
code. The idea that it just needs to be &ldquo;cleaned up&rdquo; first is the death of many
projects which would otherwise have been made open source. If you feel this way,
you will probably never be satisfied, and thus you&rsquo;ll never open it. I assure
you: your game is ready to make open source, no matter what state it&rsquo;s in!</p>
<p>Bonus: Ethan Lee tipped me off to some truly awful code which was left in
VVVVVV, which you can freely browse on the <a href="https://github.com/TerryCavanagh/vvvvvv/tree/2.2">2.2 tag</a>. It&rsquo;s truly
awful, but you probably didn&rsquo;t know that — you only remember VVVVVV as a
critically acclaimed game.</p>
<h2 id="so-what-do-i-need-to-do">So what do I need to do?</h2>
<p>Let&rsquo;s lay out the specific steps. You need to answer the following questions
first:</p>
<ul>
<li>Do I actually own the entire game? What parts am I allowed to open source?</li>
<li>Will I make the code source-available, copyleft, or permissively licensed?</li>
<li>And the assets? Proprietary? Creative Commons? If the latter, which version?</li>
</ul>
<p>If you&rsquo;re not sure what&rsquo;s best, I would recommend using the GPL for your code,
and CC-BY-SA for the assets. This allows for derivative works, so long as
they&rsquo;re also made open with a similar license. This enables the community to
build on your work, porting it to new platforms, building a thriving modding
community, and freely sharing your assets, ensuring an enduring legacy for your
game. If you&rsquo;d like to decide the details for yourself, review the comments
above once again and pick out the licenses you&rsquo;d like to use for each before
moving on.</p>
<p>If you need help with any of these steps, or have any questions, please <a href="mailto:sir@cmpwn.com">send me
an email</a>, and I will help you to the best of my ability.</p>
<p><strong>Publishing the source code</strong></p>
<p>Prepare an archive of your source code, and add the license file. If you went
with the source-available approach, simply write &ldquo;Copyright ©
&lt;<em>you</em>&gt; &lt;<em>current year</em>&gt;. All rights reserved.&rdquo; into a text file
named LICENSE. If you chose something else, copy the license text into a
LICENSE file.</p>
<p>If you want this over with quickly, just stick the code and license into a zip
file or a tarball and drop it on your website. A better approach, if you have
the patience, would be to publish it as a git repository. If you already use
version control, you may want to consider carefully if you want to publish your
full version control history — the answer might be &ldquo;yes&rdquo;, but if you&rsquo;re
unsure, the answer is probably &ldquo;no&rdquo;. Just make a copy of the code, delete the
.git directory, and import it into a new repository if you need to.</p>
<p>Double check that you aren&rsquo;t checking in any artifacts — assets,
executables, libraries, etc — and then push it to the hosting service of
your choice. GitHub is a popular choice, but I would selfishly recommend
<a href="https://sourcehut.org">sourcehut</a> as well. If you have time, write a little
README file which gives an introduction to the project as well.</p>
<p><strong>Publishing the assets</strong></p>
<p>If you choose to leave the assets proprietary, then there are no further steps.
Players can figure out how to extract the assets from their purchased game.</p>
<p>If you choose to make them open, prepare an archive of your assets. Include a
copy of the license you choose — e.g. which Creative Commons license you
used — and drop it into a zip file or a tarball or something similar.
Stick this on your website, and if you&rsquo;re feeling generous, prepare some
instructions for how to incorporate the asset bundle into the game once a player
compiles your code.</p>
<p><strong>Tell the world!</strong></p>
<p>Let everyone know that you&rsquo;ve made your game open source! Write a little blog
post, link to the source and assets, and enjoy a little bit more of the
limelight while the press and the community thanks you for your contribution.</p>
<p>One final request on this note: if you choose the source-available approach,
please refer to it as such in your public statements. Source available is <em>not</em>
the same thing as &ldquo;open source&rdquo;, and the distinction is important.</p>
<p>And now it&rsquo;s my turn to thank you: I&rsquo;m so happy that you&rsquo;ve released your game
as an open source project! The community is much richer for your contribution to
it, and I hope that your game will live on for many years to come, both in self
through ports and mods, and in spirit through its contributions to future games.
You&rsquo;ve done a wonderful thing. Thank you!</p>
<p>If you found this guide helpful in publishing your game, please <a href="mailto:sir@cmpwn.com">email
me</a> so I can play it!</p>


guid: https://drewdevault.com/2021/03/29/The-worlds-dumbest-IRC-bot.html
title: The world's stupidest IRC bot
url: https://drewdevault.com/2021/03/29/The-worlds-dumbest-IRC-bot.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I&rsquo;m an <a href="https://en.wikipedia.org/wiki/Internet_Relay_Chat">IRC</a> power user,
having been hanging out in 200+ channels on 10+ networks 24/7 for the past 10
years or so. Because IRC is
<a href="https://tools.ietf.org/html/rfc2812">standardized</a> and simple, a common
pastime for IRC enthusiasts is the creation of bots. In one of the social
channels I hang out in, we&rsquo;ve spent the past 6 years gradually building the
world&rsquo;s stupidest IRC bot: wormy.</p>
<p>For a start, wormy is highly schizophrenic. Though it presents itself as a
single bot, it is in fact a
<a href="https://en.wikipedia.org/wiki/BNC_(software)">bouncer</a> which combines the
connections of 7 independent bots. At one point, this number was higher —
as many as 11 — but some bots were consolidated.</p>
<pre><code>&lt;@sircmpwn&gt; .bots
&lt;wormy&gt; Serving text/html since 2017, yours truly [&quot;ps&quot;] For a list of commands, try `.help`
&lt;wormy&gt; minus' parcel tracking bot r10.b563abc (built on 2020-06-06T12:02:13Z, https://git.sr.ht/~minus/parcel-tracking-bot)
&lt;wormy&gt; minus' dice bot r16.498a0b8 (built on 2020-02-04T20:16:14Z, https://git.sr.ht/~minus/dice-irc-bot)
&lt;wormy&gt; Featuring arbitrary code execution by design and buffer overflows by mistake, jsbot checking in
&lt;wormy&gt; Radiobot coming to you live from The Internet, taking listener requests at 1-800-GUD-SONGS
&lt;wormy&gt; urlbot: live streaming moe directly to your eyeballs
&lt;wormy&gt; o/ SirCmpwn made me so he wouldn't forget shit so much
</code></pre><p>These bots provide a variety of features for channel members, such as checking
tracking numbers for parcels out for delivery, requesting songs for our private
internet radio, reading out the mimetypes and titles of URLs mentioned in the
channel, or feeding queries into Wolfram Alpha.</p>
<pre><code>&lt;wormy&gt; Now playing: 8369492 小さき者への贖罪の為のソナタ  by ALI PROJECT from 禁書 (4m42s FLAC)
&lt;wormy&gt; Now playing: 1045361 アキノサクラ by Wakana from magic moment (5m0s FLAC) #live ♥ minus
&lt;wormy&gt; Now playing: d0b1cb3 Forevermore by F from Cafe de Touhou 3 (4m9s FLAC) ♥ hummer12007
&lt;wormy&gt; Now playing: 0911e90 Moeru San Shimai by Iwasaki Taku from Tengen Toppa Gurren Lagann Original Soundtrack - CD01 (3m3s FLAC)
&lt;wormy&gt; Now playing: ac1a17e rebellion anthem by Yousei teikoku from rebellion anthem (5m15s MP3) ♥ minus
&lt;wormy&gt; Now playing: a5ab39a Desirable Dream by GET IN THE RING from Aki-秋- (4m38s FLAC) ♥ minus
</code></pre><p>Things really took off with the introduction of a truly stupid bot last year:
<a href="https://git.sr.ht/~sircmpwn/jsbot">jsbot</a>. This bot adds a <code>.js</code> command which
executes arbitrary JavaScript (using Fabrice Bellard&rsquo;s
<a href="https://bellard.org/quickjs/">quickjs</a>) expressions, and sending their
stringified result to the channel.</p>
<pre><code>&lt;@sircmpwn&gt; .js Array(16).join(&quot;wat&quot; - 1) + &quot; Batman!&quot;
&lt;wormy&gt; =&gt; NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN Batman!
</code></pre><p>We soon realized, however, that what we had effectively created was a persistent
JavaScript environment which was connected to IRC. This has made it possible to
write even more IRC bots in the least practical manner imaginable: by writing
JavaScript statements, one line at a time, into IRC messages, and hoping it
works.</p>
<p>This has not been an entirely smart move.</p>
<p><img src="https://l.sr.ht/OCyx.png" alt="I say &ldquo;I hate C++ templates&rdquo;, and the bot responds by writing &ldquo;yeah, fuck C++templates!&rdquo; with &ldquo;C++ templates&rdquo; displayed in rainbow colors. MartijnBraamfollows up by asking &ldquo;number of c++ programmers launched into the sun&rdquo;, whichwormy claims is 367880 people as of 2009."></p>
<p>One &ldquo;feature&rdquo;, inspired by <a href="https://www.youtube.com/watch?v=30jNsCVLpAE">Bryan
Cantrill</a>, records every time the
word &ldquo;fuck&rdquo; is used in the channel. Then, whenever anyone says &ldquo;wtf&rdquo;, the bot
helpfully offers up an example of the usage of the word &ldquo;fuck&rdquo; by printing one
of the recorded messages. Here&rsquo;s how it was made:</p>
<pre><code>&lt;sircmpwn&gt; .js let wtf = [];
&lt;wormy&gt;  =&gt; undefined
&lt;sircmpwn&gt; .js on(/fuck/, msg =&gt; wtf.push(msg.text))
&lt;wormy&gt;  =&gt; 25
&lt;sircmpwn&gt; .js on(/^what the fuck$/, msg =&gt; msg.reply(wtf[Math.floor(Math.random() * wtf.length)]))
&lt;wormy&gt;  =&gt; 26
</code></pre><p>Here&rsquo;s one which records whenever someone says &ldquo;foo++&rdquo; or &ldquo;foo--&rdquo; and keeps
track of scores:</p>
<pre><code>.js on(/^([a-zA-Z0-9_]+)(\+\+|--)$/, (msg, thing, op) =&gt; { if (typeof scores[thing] === &quot;undefined&quot;) scores[thing] = 0; scores[thing] += op === &quot;++&quot; ? 1 : -1; msg.reply(`${thing}: ${scores[thing]}`) });
.js on(/\.score (.*)/, (msg, item) =&gt; msg.reply(scores[item]));
.js let worst = () =&gt; Object.entries(scores).sort((a, b) =&gt; a[1] - b[1]).slice(0, 5).map(s =&gt; `${s[0]}: ${s[1]}`).join(&quot;, &quot;);
.js let best = () =&gt; Object.entries(scores).sort((a, b) =&gt; b[1] - a[1]).slice(0, 5).map(s =&gt; `${s[0]}: ${s[1]}`).join(&quot;, &quot;);
.js on(/^.worst$/, msg =&gt; msg.reply(worst()));
.js on(/^.best$/, msg =&gt; msg.reply(best()));
</code></pre><p>Other &ldquo;features&rdquo; written in horrible one-liners include SI unit conversions,
rewriting undesirable URLs (e.g. m.wikipedia.org =&gt; en.wikipedia.org), answering
&ldquo;wormy you piece of shit&rdquo; with &ldquo;¯\_(ツ)_/¯&rdquo;, and giving the obvious response to
&ldquo;make me a sandwich&rdquo;.</p>
<p>Eventually it occurred to us that we had two dozen stupid IRC bots storing not
only their state, but their code, in a single long-lived process on some server.
For a while, the answer to this was adding &ldquo;don&rsquo;t reboot this server kthx&rdquo; to
the MotD, but eventually we did some magic nonsense to make certain variables
persistent:</p>
<div class="highlight"><pre class="chroma"><code class="language-js" data-lang="js"><span class="kd">let</span> <span class="nx">persistent</span> <span class="o">=</span> <span class="p">{};</span>
<span class="kd">function</span> <span class="nx">writePersistent</span><span class="p">()</span> <span class="p">{</span>
  <span class="kd">let</span> <span class="nx">fd</span> <span class="o">=</span> <span class="nx">std</span><span class="p">.</span><span class="nx">open</span><span class="p">(</span><span class="s2">&#34;persist.json&#34;</span><span class="p">,</span> <span class="s2">&#34;w&#34;</span><span class="p">);</span>
  <span class="nx">fd</span><span class="p">.</span><span class="nx">puts</span><span class="p">(</span><span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">(</span><span class="nx">persistent</span><span class="p">));</span>
  <span class="nx">fd</span><span class="p">.</span><span class="nx">close</span><span class="p">();</span>
<span class="p">}</span>

<span class="kd">let</span> <span class="nx">persist_handler</span> <span class="o">=</span> <span class="p">{</span>
  <span class="nx">set</span><span class="o">:</span> <span class="p">(</span><span class="nx">obj</span><span class="p">,</span> <span class="nx">prop</span><span class="p">,</span> <span class="nx">val</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="p">{</span>
    <span class="nx">obj</span><span class="p">[</span><span class="nx">prop</span><span class="p">]</span> <span class="o">=</span> <span class="nx">val</span><span class="p">;</span>
    <span class="nx">writePersistent</span><span class="p">();</span>
  <span class="p">},</span>
<span class="p">};</span>

<span class="kd">let</span> <span class="nx">p</span> <span class="o">=</span> <span class="nx">std</span><span class="p">.</span><span class="nx">loadFile</span><span class="p">(</span><span class="s2">&#34;persist.json&#34;</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">p</span> <span class="o">!==</span> <span class="kc">null</span><span class="p">)</span> <span class="p">{</span>
  <span class="nx">persistent</span> <span class="o">=</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">p</span><span class="p">);</span>
  <span class="nb">Object</span><span class="p">.</span><span class="nx">keys</span><span class="p">(</span><span class="nx">persistent</span><span class="p">).</span><span class="nx">map</span><span class="p">(</span><span class="nx">key</span> <span class="p">=&gt;</span> <span class="p">{</span>
    <span class="kd">let</span> <span class="nx">proxy</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Proxy</span><span class="p">(</span><span class="nx">persistent</span><span class="p">[</span><span class="nx">key</span><span class="p">],</span> <span class="nx">persist_handler</span><span class="p">);</span>
    <span class="nx">persistent</span><span class="p">[</span><span class="nx">key</span><span class="p">]</span> <span class="o">=</span> <span class="nx">proxy</span><span class="p">;</span>
    <span class="nx">exports</span><span class="p">[</span><span class="nx">key</span><span class="p">]</span> <span class="o">=</span> <span class="nx">proxy</span><span class="p">;</span>
  <span class="p">});</span>
<span class="p">}</span>

<span class="nx">exports</span><span class="p">.</span><span class="nx">persist</span> <span class="o">=</span> <span class="p">(</span><span class="nx">name</span><span class="p">,</span> <span class="nx">obj</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="p">{</span>
  <span class="kd">let</span> <span class="nx">proxy</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Proxy</span><span class="p">(</span><span class="nx">obj</span><span class="p">,</span> <span class="nx">persist_handler</span><span class="p">);</span>
  <span class="nx">persistent</span><span class="p">[</span><span class="nx">name</span><span class="p">]</span> <span class="o">=</span> <span class="nx">proxy</span><span class="p">;</span>
  <span class="nx">writePersistent</span><span class="p">();</span>
  <span class="k">return</span> <span class="nx">proxy</span><span class="p">;</span>
<span class="p">};</span>
</code></pre></div><p>Anyway, there&rsquo;s no moral to this story. We just have a silly IRC bot and I
thought I&rsquo;d share that with you. If you want a stupid IRC bot for your own
channel, <a href="https://git.sr.ht/~sircmpwn/jsbot">jsbot</a> is available on sourcehut. I
highly disrecommend it and disavow any responsibility for the consequences.</p>


guid: https://drewdevault.com/2021/04/02/Go-is-a-great-language.html
title: Go is a great programming language
url: https://drewdevault.com/2021/04/02/Go-is-a-great-language.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>No software is perfect, and thus even for software I find very pleasant, I can
usually identify some problems in it — often using my blog to do so. Even
my all-time favorite software project, Plan 9, has some painful flaws! For some
projects, it may be my fondness for them that drives me to criticise them even
more, in the hope that they&rsquo;ll live up to the level of respect I feel for them.</p>
<p>One such project is the Go programming language. I have had many criticisms,
often shared on this blog and elsewhere, but for the most part, my praises have
been aired mainly in private. I&rsquo;d like to share some of those praises today,
because despite my criticisms of it, Go remains one of the best programming
languages I&rsquo;ve ever used, and I have a great deal of respect for it.</p>
<p>Perhaps the matter I most appreciate Go for is its long-term commitment to
simplicity, stability, and robustness. I prize these traits more strongly than
any other object of software design. The Go team works with an ethos of
careful restraint, with each feature given deliberate consideration towards
identifying the simplest and most complete solution, and they carefully
constrain the scope of their implementations to closely fit those solutions.
The areas where Go has failed in this regard are frightfully scarce.</p>
<p>The benefits of their discipline are numerous. The most impressive
accomplishment that I attribute to this approach is the quality of the Go
ecosystem at large. In the first place, it is a great accomplishment to produce
a language and standard library with the excellence in design and implementation
that Go offers, but it&rsquo;s a truly profound achievement to have produced a design
which the community <em>at large</em> utilizes to make similarly excellent designs as a
basic consequence of the language&rsquo;s simple elegance.  Very few other languages
enjoy a similar level of consistency and quality in the ecosystem.</p>
<p>Go is also notable for essentially inventing its own niche, and then helping
that niche grow around it into an entirely new class of software design. I
consider Go not to be a systems programming language — a title much better
earned by languages like C and Rust. Rather, Go is the best-in-class for a new
breed of software: an Internet programming language.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> The wealth of network
protocols implemented efficiently, concisely, and correctly in its standard
library, combined with its clever mixed cooperative/pre-emptive multitasking
model, make it very easy to write scalable internet-facing software. A few other
languages — Elixir comes to mind — also occupy this niche, but they
haven&rsquo;t enjoyed the runaway success that Go has.</p>
<p>The Go team has also earned my respect for their professionalism. The close
degree to which Go is tied to Google comes with its own set of trade-offs, but
the centralization of project leadership caused by this relationship is
beneficial for the project. Some members of the Go community have noticed the
apparent disadvantages of this structure, as Go is infamous for being slow to
respond to the wants of its community. This insulation, I would argue, is in
fact advantageous for the conservative language design that Go embraces, and
may actually be essential to its value-add as a project. If Go listened to the
community as much as they want, it would become a kitchen sink, and cease to be
interesting to me.</p>
<p>Rather than being closely tied to its community&rsquo;s wants, Go generally does a
much better job of being closely tied to its community&rsquo;s <em>needs</em>. If you have
correctly identified a problem in Go, when you bring it to their attention, you
will be taken seriously. Many projects struggle to separate their egos from the
software, and when mistakes are found, they take it personally. Go does an
excellent job of treating it like an engineer — a matter-of-fact analysis
of the problem, deliberation on the solution, and shipping of a fix.<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> Go has
a reputation for plain old good engineering.</p>
<p>In short, I admire Go very much, despite my frequent criticisms. I recognize Go
as one of the best programming languages ever made. Go has attained an elusive
status in the programming canon as a robust engineering tool that can be
expected to work, and work well, in its applications for decades to come. Its
because of this respect that I hold Go to such a high standard, and I hope that
it continues to impress me going forward.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>It took me a while to understand this. It was a mistake for Go to be marketed as a systems language. Any systems programmer would rightfully tell you that a language with a garbage collector and magic cooperative/pre-emptive threads is a non-starter for systems programming. But, what Go was really designed for, and is mainly used for, is not exactly systems programming. Internet-facing code has straddled the line between systems programming and high-level programming for a while: high-performance systems software would often be written in, say, C — which is definitely a systems programming language — but the vastness of the Internet&rsquo;s problem space also affords for a large number of programs for which a higher-level programming languages are a better fit, such as Java, C#, etc — and these are definitely not systems programming languages. Go is probably the first language to specifically target this space in-between with this degree of success, and it kind of makes a new domain for itself in so doing: it is the first widely successful &ldquo;Internet programming language&rdquo;. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Sometimes, this has not been the case, and this was the cause of some of my harshest criticisms of Go. Many of Go&rsquo;s advantages stem from, and even <em>require</em>, this dispassionate, matter-of-fact engineering ethos that I appreciate from Go. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2021/04/07/The-next-chat-app.html
title: What should the next chat app look like?
url: https://drewdevault.com/2021/04/07/The-next-chat-app.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>As you&rsquo;re surely aware, Signal has officially jumped the shark with the
introduction of cryptocurrency to their chat app. Back in 2018, I <a href="https://drewdevault.com/2018/08/08/Signal.html">wrote about
my concerns with Signal</a>, and those concerns were unfortunately validated by
this week&rsquo;s announcement. Moxie&rsquo;s insistence on centralized ownership,
governance, and servers for Signal puts him in a position of power which is
easily, and inevitably, abused. In that 2018 article, and in <a href="https://drewdevault.com/2020/09/20/The-potential-of-federation.html">articles since</a>,
I have spoken about the important of federation to address these problems. In
addition to federation, what else does a chat app need?</p>
<p>Well, first, the next chat app should be <strong>a protocol</strong>, not just an app. A lush
ecosystem of client and server implementations, along with bots and other
integrations, adds a tremendous amount of value and longevity to a system. A
chat app which has only one implementation and a private protocol can only ever
meet the needs that its developers (1) foresee, (2) care about, and (3) have the
capacity to address; thus, such a protocol cannot be ubiquitous. I would also
recommend that this protocol is not needlessly stapled to the beached whale that
is the web: <em>maybe</em> JSON can come, but if it&rsquo;s served with HTTP polling to
appease our Android overlords I will be very cross with you. JSON also offers
convenient extensibility, and a protocol designer who limits extensibility is a
wise one.</p>
<p>Crucially, that protocol <em>must</em> be <strong>federated</strong>. This is Signal&rsquo;s largest
failure. We simply cannot trust a single entity, even you, dear reader, to have
such a large degree of influence over the ecosystem.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> I do not trust you not
to add some crypto Ponzi scheme of your own 5 years from now. A federated system
allows multiple independent server operators to stand up their own servers which
can communicate with each other and exchange messages on behalf of their
respective users, which distributes ownership, responsibility, and governance
within the community at large, making the system less vulnerable to all kinds of
issues. You need to be prepared to relinquish control to the community. Signal
wasn&rsquo;t, and has had problems ranging from 502 Server Gone errors to 404
Ethics Not Found errors, both of which are solved by federation.</p>
<p>The next chat app also needs <strong>end-to-end encryption</strong>. This should be fairly
obvious, but it&rsquo;s worth re-iterating because this will occupy a <em>majority</em> of
the design work that goes into the app. There are complex semantics involved in
encrypting user-to-user chats, group chats (which could add or remove users at
any time), perfect forward secrecy, or multiple devices under one account; many
of these issues have implications for the user experience. This is complicated
further by the concerns of a federated design, and if you want to support voice
or video chat (please don&rsquo;t), that&rsquo;ll complicate things even more. You&rsquo;ll spend
the bulk of your time solving these problems. I would advise, however, that you
let users dial down the privacy (after explaining to them the trade-offs) in
exchange for convenience. For instance, to replace IRC you would need to support
channels which anyone can join at any time and which might make chat logs
available to the public.</p>
<p>A new chat app also needs <strong>anonymity</strong>. None of this nonsense where users have
to install your app and give you their phone number to register. In fact, you
should know next to nothing about each user, given that the most secure data is
the data you don&rsquo;t have. This is made more difficult when you consider that
you&rsquo;ll also strive to provide an authentic identity for users to establish
between themselves — but not with you. Users should also be able to
establish a pseudonymous identity, or wear multiple identities. You need to
provide (1) a strong guarantee of consistent identity from session to
session, (2) without sharing that guarantee with your servers, and (3) offer
the ability to able to change to a new identity at will. The full implications
of anonymity are a complex issue which is out of scope for this article, but for
now it suffices to say that you should at least refrain from asking for the
user&rsquo;s phone number.</p>
<p>Finally, it needs to be <strong>robust, reliable, and performant</strong>. Focus on the
basics: delivering messages quickly and reliably. The first thing you need to
produce is a reliable messenger which works in a variety of situations, on a
variety of platforms, in various network conditions, and so on, with the
underlying concerns of federation, end-to-end encryption, protocol
standardization, group and individual chats, multi-device support, and so on, in
place and working. You can try to deliver this in a moderately attractive
interface, but sinking a lot of time into fancy animations, stickers, GIF
lookups, typing notifications and read receipts — all of this is a
distraction until you get the real work done. You can have all of these things,
but if you don&rsquo;t have a reliable system underlying them, the result is
worthless.</p>
<p>I would also recommend leaving a lot of those features at the door, anyway.
Typing notifications and read receipts are pretty toxic, if you examine them
critically. A lot of chat apps have a problem with cargo-culting bad ideas from
each other. Try to resist that. Anyway, you have a lot of work to do, so I&rsquo;ll
leave you to it. <a href="mailto:sir@cmpwn.com">Let me know</a> what you&rsquo;re working on when
you&rsquo;ve got something to show for it.</p>
<p>And don&rsquo;t put a fucking cryptocurrency in it.</p>
<details>
  <summary>Regarding Matrix, IRC, etc</summary>
  <p>
    Let's quickly address the present state of the ecosystem. Matrix rates well
    in most of these respects, much better than others. However, their software
    is way too complicated. They are federated, but the software is far from
    reliable or robust, so the ecosystem tends to be centralized because
    Matrix.org are the only ones who have the knowledge and bandwidth to keep it
    up and running. The performance sucks, client and server both, and their UX
    for E2EE is confusing and difficult to use.
  <p>
    It's a good attempt, but too complex and brittle. Also, their bridge
    is a major nuisance to IRC, which biases me against them. Please don't
    integrate your next chat app with IRC; just leave us alone, thanks.
  <p>
    Speaking of IRC, it is still my main chat program, and has been for 15+
    years. The lack of E2EE, which is unacceptable for any new protocol, is not
    important enough to get me to switch to anything else until it presents a
    compelling alternative to IRC.
</details>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Even if that ecosystem is &ldquo;moving&rdquo;. Ugh. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2021/04/12/DCO.html
title: The Developer Certificate of Origin is a great alternative to a CLA
url: https://drewdevault.com/2021/04/12/DCO.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Today Amazon released their fork of ElasticSearch, <a href="https://github.com/opensearch-project/OpenSearch">OpenSearch</a>, and I want
to take a moment to draw your attention to one good decision in particular: its
use of the <a href="https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin">Developer Certificate of Origin</a> (or &ldquo;DCO&rdquo;).</p>
<hr>
<p>Previously:</p>
<ul>
<li><a href="https://drewdevault.com/2021/01/19/Elasticsearch-does-not-belong-to-Elastic.html">ElasticSearch does not belong to Elastic</a></li>
<li><a href="https://drewdevault.com/2021/01/20/FOSS-is-to-surrender-your-monopoly.html">Open source means surrendering your monopoly over commercial exploitation</a></li>
<li><a href="https://drewdevault.com/2018/10/05/Dont-sign-a-CLA.html">Don&rsquo;t sign a CLA</a></li>
</ul>
<hr>
<p>Elastic betrayed its community when they changed to a proprietary license.  We
could have seen it coming because of a particular trait of their contribution
process: the use of a Contributor License Agreement, or CLA. In principle, a CLA
aims to address legitimate concerns of ownership and copyright, but in practice,
they are a promise that one day the stewards of the codebase will take your work
and relicense it under a nonfree license. And, ultimately, this is exactly what
Elastic did, and exactly what most other projects which ask you to sign a CLA
are <em>planning</em> to do. If you ask me, that&rsquo;s a crappy deal, and I refrain from
contributing to those projects as a result.</p>
<p>However, there are some legitimate questions of ownership which a project owner
might rightfully wish to address before accepting a contribution. As is often
the case, we can look to git itself for an answer to this problem. Git was
designed for the Linux kernel, and patch ownership is a problem they faced and
solved a long time ago. Their answer is the <a href="https://developercertificate.org/">Developer Certificate of
Origin</a>, or DCO, and tools for working with
it are already built into git.</p>
<p>git provides the -s flag for git commit, which adds the following text to your
commit message:</p>
<pre><code>Signed-off-by: Drew DeVault &lt;sir@cmpwn.com&gt;
</code></pre><p>The specific meaning varies from project to project, but it is usually used to
indicate that you have read and agreed to the DCO, which reads as follows:</p>
<blockquote>
<p>By making a contribution to this project, I certify that:</p>
<ol>
<li>The contribution was created in whole or in part by me and I have the right
to submit it under the open source license indicated in the file; or</li>
<li>The contribution is based upon previous work that, to the best of my
knowledge, is covered under an appropriate open source license and I have
the right under that license to submit that work with modifications,
whether created in whole or in part by me, under the same open source
license (unless I am permitted to submit under a different license), as
indicated in the file; or</li>
<li>The contribution was provided directly to me by some other person who
certified (1), (2) or (3) and I have not modified it.</li>
<li>I understand and agree that this project and the contribution are public
and that a record of the contribution (including all personal information I
submit with it, including my sign-off) is maintained indefinitely and may
be redistributed consistent with this project or the open source license(s)
involved.</li>
</ol>
</blockquote>
<p>This neatly answers all concerns of copyright. You license your contribution
under the original license (Apache 2.0 in the case of OpenSearch), and attest
that you have sufficient ownership over your changes to do so. You retain your
copyright and you don&rsquo;t leave the door open for the maintainers to relicense
your work under some other terms in the future. This offers the maintainers the
same rights that they extended to the community themselves.</p>
<p>This is the strategy that Amazon chose for OpenSearch, and it&rsquo;s a good thing
they did, because it strongly signals to the community that it will not fall to
the same fate that ElasticSearch has. By doing this, they have imposed on
themselves a great deal of difficulty to any future attempt to change their
copyright obligations. I applaud Amazon for this move, and I&rsquo;m optimistic about
the future of OpenSearch under their stewardship.</p>
<p>If you have a project of your own that is concerned about the copyright of
third-party contributions, then please consider adopting the DCO instead of a
CLA. And, as a contributor, if someone asks you to sign a CLA, consider
withholding your contribution: a CLA is a promise to the contributors that
someday their work will be taken from them and monetized to the exclusive
benefit of the project&rsquo;s lords. This affects my personal contributions, too
— for example, I avoid contributing to Golang as a result of their CLA
requirement. Your work is important, and the projects you offer it to should
respect that.</p>


guid: https://drewdevault.com/2021/04/15/Status-update-April-2021.html
title: Status update, April 2021
url: https://drewdevault.com/2021/04/15/Status-update-April-2021.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Another month goes by! I&rsquo;m afraid that I have very little to share this month.
You can check out the <a href="https://sourcehut.org/blog/2021-04-15-whats-cooking-april-2021/">sourcehut &ldquo;what&rsquo;s cooking&rdquo; post</a> for sourcehut news,
but outside of that I have focused almost entirely on the programming language
project this month, for which the details are kept private.</p>
<p>The post <a href="https://drewdevault.com/2021/03/19/A-new-systems-language.html">calling for contributors</a> led to a lot of answers and we&rsquo;ve brought
several new people on board — thanks for answering the call! I&rsquo;d like to
narrow the range of problems we still need help with. If you&rsquo;re interested in
(and experienced in) the following problems, we need your help:</p>
<ul>
<li>Cryptography</li>
<li>Date/time support</li>
<li>Networking (DNS is up next)</li>
</ul>
<p><a href="mailto:sir@cmpwn.com">Shoot me an email</a> if you want to help. We don&rsquo;t have the
bandwidth to mentor inexperienced programmers right now, so please only reach
out if you have an established background in systems programming.</p>
<p>Here&rsquo;s a teaser of one of the stdlib APIs written by our new contributors,
unix::passwd:</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="c1">// A Unix-like group file entry.
</span><span class="c1"></span><span class="k">export</span> <span class="k">type</span> <span class="n">grent</span> <span class="o">=</span> <span class="k">struct</span> <span class="p">{</span>
	<span class="c1">// Name of the group
</span><span class="c1"></span>	<span class="n">name</span><span class="o">:</span> <span class="kt">str</span><span class="p">,</span>
	<span class="c1">// Optional encrypted password
</span><span class="c1"></span>	<span class="n">password</span><span class="o">:</span> <span class="kt">str</span><span class="p">,</span>
	<span class="c1">// Numerical group ID
</span><span class="c1"></span>	<span class="n">gid</span><span class="o">:</span> <span class="kt">uint</span><span class="p">,</span>
	<span class="c1">// List of usernames that are members of this group, comma separated
</span><span class="c1"></span>	<span class="n">userlist</span><span class="o">:</span> <span class="kt">str</span><span class="p">,</span>
<span class="p">};</span>

<span class="c1">// Reads a Unix-like group entry from a stream. The caller must free the result
</span><span class="c1">// using [grent_finish].
</span><span class="c1"></span><span class="k">export</span> <span class="k">fn</span> <span class="n">nextgr</span><span class="p">(</span><span class="n">stream</span><span class="o">:</span> <span class="o">*</span><span class="n">io</span><span class="o">::</span><span class="n">stream</span><span class="p">)</span> <span class="p">(</span><span class="n">grent</span> <span class="o">|</span> <span class="n">io</span><span class="o">::</span><span class="n">EOF</span> <span class="o">|</span> <span class="n">io</span><span class="o">::</span><span class="n">error</span> <span class="o">|</span> <span class="n">invalid</span><span class="p">);</span>

<span class="c1">// Frees resources associated with [grent].
</span><span class="c1"></span><span class="k">export</span> <span class="k">fn</span> <span class="n">grent_finish</span><span class="p">(</span><span class="n">ent</span><span class="o">:</span> <span class="n">grent</span><span class="p">)</span> <span class="kt">void</span><span class="p">;</span>

<span class="c1">// Looks up a group by name in a Unix-like group file. It expects a such file at
</span><span class="c1">// /etc/group. Aborts if that file doesn&#39;t exist or is not properly formatted.
</span><span class="c1">//
</span><span class="c1">// See [nextgr] for low-level parsing API.
</span><span class="c1"></span><span class="k">export</span> <span class="k">fn</span> <span class="n">getgroup</span><span class="p">(</span><span class="n">name</span><span class="o">:</span> <span class="kt">str</span><span class="p">)</span> <span class="p">(</span><span class="n">grent</span> <span class="o">|</span> <span class="kt">void</span><span class="p">);</span>
</code></pre></div><p>That&rsquo;s all for now. These updates might be light on details for a while as we
work on this project. See you next time!</p>


guid: https://drewdevault.com/2021/04/22/Our-self-hosted-parser-design.html
title: Parsers all the way down: writing a self-hosting parser
url: https://drewdevault.com/2021/04/22/Our-self-hosted-parser-design.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>One of the things we&rsquo;re working on in <a href="https://drewdevault.com/2021/03/19/A-new-systems-language.html">my new programming language</a> is a
self-hosting compiler. Having a self-hosted compiler is a critical step in the
development of (some) programming languages: it signals that the language is
mature enough to be comfortably used to implement itself. While this isn&rsquo;t right
for some languages (e.g. shell scripts), for a systems programming language like
ours, this is a crucial step in our bootstrapping plan. Our self-hosted parser
design was completed this week, and today I&rsquo;ll share some details about how it
works and how it came to be.</p>
<p>This is the third parser which has been implemented for this language. We wrote
a sacrificial compiler prototype upfront to help inform the language design, and
that first compiler used <a href="https://en.wikipedia.org/wiki/Yacc">yacc</a> for its parser. Using yacc was helpful at
first because it makes it reasonably simple to iterate on the parser when the
language is still undergoing frequent and far-reaching design changes. Another
nice side-effect starting with a yacc parser is that it makes it quite easy to
produce a formal grammar when you settle on the design. Here&rsquo;s a peek at some of
our original parser code:</p>
<pre><code>struct_type
	: T_STRUCT '{' struct_fields '}' {
		$$.flags = 0;
		$$.storage = TYPE_STRUCT;
		allocfrom((void **)&amp;$$.fields, &amp;$3, sizeof($3));
	}
	| T_UNION '{' struct_fields '}' {
		$$.flags = 0;
		$$.storage = TYPE_UNION;
		allocfrom((void **)&amp;$$.fields, &amp;$3, sizeof($3));
	}
	;

struct_fields
	: struct_field
	| struct_field ',' { $$ = $1; }
	| struct_field ',' struct_fields {
		$$ = $1;
		allocfrom((void **)&amp;$$.next, &amp;$3, sizeof($3));
	}
	;

struct_field
	: T_IDENT ':' type {
		$$.name = $1;
		allocfrom((void**)&amp;$$.type, &amp;$3, sizeof($3));
		$$.next = NULL;
	}
	;
</code></pre><p>This approach has you writing code which is already almost a formal grammar in
its own right. If we strip out the C code, we get the following:</p>
<pre><code>struct_type
	: T_STRUCT '{' struct_fields '}'
	| T_UNION '{' struct_fields '}'
	;

struct_fields
	: struct_field
	| struct_field ','
	| struct_field ',' struct_fields
	;

struct_field
	: T_IDENT ':' type
	;
</code></pre><p>This gives us a reasonably clean path to writing a formal grammar (and
specification) for the language, which is what we did next.</p>
<p><img src="https://l.sr.ht/CFo0.png" alt="A screenshot of a PDF file which shows a formal grammar similar to the samplegiven above."></p>
<p>All of these samples describe a struct type. The following example shows what
this grammar looks like in real code — starting from the word &ldquo;struct&rdquo; and
including up to the &ldquo;}&rdquo; at the end.</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="k">type</span> <span class="n">coordinates</span> <span class="o">=</span> <span class="k">struct</span> <span class="p">{</span>
    <span class="n">x</span><span class="o">:</span> <span class="kt">int</span><span class="p">,</span>
    <span class="n">y</span><span class="o">:</span> <span class="kt">int</span><span class="p">,</span>
    <span class="n">z</span><span class="o">:</span> <span class="kt">int</span><span class="p">,</span>
<span class="p">};</span>
</code></pre></div><p>In order to feed our parser tokens to work with, we also need a lexer, or a
<em>lexical analyzer</em>. This turns a series of characters like &ldquo;struct&rdquo; into a
single token, like the T_STRUCT we used in the yacc code. Like the original
compiler used yacc as a parser generator, we also used <a href="https://en.wikipedia.org/wiki/Lex_(software)">lex</a> as a lexer
generator. It&rsquo;s simply a list of regexes and the names of the tokens that match
those regexes, plus a little bit of extra code to do things like turning &ldquo;1234&rdquo;
into an int with a value of 1234. Our lexer also kept track of line and column
numbers as it consumed characters from input files.</p>
<pre><code>&quot;struct&quot;	{ _lineno(); return T_STRUCT; }
&quot;union&quot;		{ _lineno(); return T_UNION; }
&quot;{&quot;		{ _lineno(); return '{'; }
&quot;}&quot;		{ _lineno(); return '}'; }

[a-zA-Z][a-zA-Z0-9_]* {
	_lineno();
	yylval.sval = strdup(yytext);
	return T_IDENTIFIER;
}
</code></pre><p>After we settled on the design with our prototype compiler, which was able to
compile some simple test programs to give us a feel for our language design, we
set it aside and wrote the specification, and, alongside it, a second compiler.
This new compiler was written in C — the language was not ready to
self-host yet — and uses a hand-written <a href="https://en.wikipedia.org/wiki/Recursive_descent_parser">recursive descent</a> parser.</p>
<p>To simplify the parser, we deliberately designed a context-free LL(1) grammar,
which means it (a) can parse an input unambiguously without needing additional
context, and (b) only requires one token of look-ahead. This makes our parser
design a lot simpler, which was a deliberate goal of the language design. Our
hand-rolled lexer is <em>slightly</em> more complicated: it requires two characters of
lookahead to distinguish between the &ldquo;.&rdquo;, &ldquo;..&rdquo;, and &ldquo;&hellip;&rdquo; tokens.</p>
<p>I&rsquo;ll skip going in depth on the design of the second parser, because the hosted
parser is more interesting, and a pretty similar design anyway. Let&rsquo;s start by
taking a look at our hosted lexer. Our lexer is initialized with an input source
(e.g. a file) from which it can read a stream of characters. Then, each time we
need a token, we&rsquo;ll ask it to read the next one out. It will read as many
characters as it needs to unambiguously identify the next token, then hand it up
to the caller.</p>
<p>Our specification provides some information to guide the lexer design:</p>
<blockquote>
<p>A token is the smallest unit of meaning in the <span style="background:
black">****</span> grammar. The lexical analysis phase processes a UTF-8
source file to produce a stream of tokens by matching the terminals with the
input text.</p>
<p>Tokens may be separated by <em>white-space</em> characters, which are defined as the
Unicode code-points <code>U+0009</code> (horizontal tabulation), <code>U+000A</code> (line feed), and
<code>U+0020</code> (space). Any number of whitespace characters may be inserted between
tokens, either to disambiguate from subsequent tokens, or for aesthetic
purposes. This whitespace is discarded during the lexical analysis phase.</p>
<p><em>Within a single token, white-space is meaningful. For example, the
string-literal token is defined by two quotation marks <strong>&quot;</strong> enclosing any
number of literal characters. The enclosed characters are considered part of
the string-literal token and any whitespace therein is not discarded.</em></p>
<p>The lexical analysis process consumes Unicode characters from the source file
input until it is exhausted, performing the following steps in order: it shall
consume and discard white-space characters until a non-white-space character
is found, then consume the longest sequence of characters which could
constitute a token, and emit it to the token stream.</p>
</blockquote>
<p>There are a few different kinds of tokens our lexer is going to need to handle:
operators, like &ldquo;+&rdquo; and &ldquo;-&quot;; keywords, like &ldquo;struct&rdquo; and &ldquo;return&rdquo;; user-defined
identifiers, like variable names; and constants, like string and numeric
literals.</p>
<p>In short, given the following source code:</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="k">fn</span> <span class="n">add2</span><span class="p">(</span><span class="n">x</span><span class="o">:</span> <span class="kt">int</span><span class="p">,</span> <span class="n">y</span><span class="o">:</span> <span class="kt">int</span><span class="p">)</span> <span class="kt">int</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="p">;</span>
</code></pre></div><p>We need to return the following sequence of tokens:</p>
<pre><code>fn      (keyword)
add2    (identifier)
(       (operator)
x
:
int
,
y
int
)
int
=
x
+
y
;
</code></pre><p>This way, our parser doesn&rsquo;t have to deal with whitespace, or distinguishing
&ldquo;int&rdquo; (keyword) from &ldquo;integer&rdquo; (identifier), or handling invalid tokens like
&ldquo;$&rdquo;. To actually implement this behavior, we&rsquo;ll start with an initialization
function which populates a state structure.</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="c1">// Initializes a new lexer for the given input stream. The path is borrowed.
</span><span class="c1"></span><span class="k">export</span> <span class="k">fn</span> <span class="n">init</span><span class="p">(</span><span class="n">in</span><span class="o">:</span> <span class="o">*</span><span class="n">io</span><span class="o">::</span><span class="n">stream</span><span class="p">,</span> <span class="n">path</span><span class="o">:</span> <span class="kt">str</span><span class="p">,</span> <span class="n">flags</span><span class="o">:</span> <span class="n">flags</span><span class="p">...)</span> <span class="n">lexer</span> <span class="o">=</span> <span class="p">{</span>
	<span class="k">return</span> <span class="n">lexer</span> <span class="p">{</span>
		<span class="n">in</span> <span class="o">=</span> <span class="n">in</span><span class="p">,</span>
		<span class="n">path</span> <span class="o">=</span> <span class="n">path</span><span class="p">,</span>
		<span class="n">loc</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
		<span class="n">un</span> <span class="o">=</span> <span class="kt">void</span><span class="p">,</span>
		<span class="n">rb</span> <span class="o">=</span> <span class="p">[</span><span class="kt">void</span><span class="p">...],</span>
	<span class="p">};</span>
<span class="p">};</span>

<span class="k">export</span> <span class="k">type</span> <span class="n">lexer</span> <span class="o">=</span> <span class="k">struct</span> <span class="p">{</span>
	<span class="n">in</span><span class="o">:</span> <span class="o">*</span><span class="n">io</span><span class="o">::</span><span class="n">stream</span><span class="p">,</span>
	<span class="n">path</span><span class="o">:</span> <span class="kt">str</span><span class="p">,</span>
	<span class="n">loc</span><span class="o">:</span> <span class="p">(</span><span class="kt">uint</span><span class="p">,</span> <span class="kt">uint</span><span class="p">),</span>
	<span class="n">rb</span><span class="o">:</span> <span class="p">[</span><span class="mi">2</span><span class="p">](</span><span class="n">rune</span> <span class="o">|</span> <span class="n">io</span><span class="o">::</span><span class="n">EOF</span> <span class="o">|</span> <span class="kt">void</span><span class="p">),</span>
<span class="p">};</span>
</code></pre></div><p>This state structure holds, respectively:</p>
<ul>
<li>The input I/O stream</li>
<li>The path to the current input file</li>
<li>The current (line, column) number</li>
<li>A buffer of un-read characters from the input, for lookahead</li>
</ul>
<p>The main entry point for doing the actual lexing will look like this:</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="c1">// Returns the next token from the lexer.
</span><span class="c1"></span><span class="k">export</span> <span class="k">fn</span> <span class="n">lex</span><span class="p">(</span><span class="n">lex</span><span class="o">:</span> <span class="o">*</span><span class="n">lexer</span><span class="p">)</span> <span class="p">(</span><span class="n">token</span> <span class="o">|</span> <span class="n">error</span><span class="p">);</span>

<span class="c1">// A single lexical token, the value it represents, and its location in a file.
</span><span class="c1"></span><span class="k">export</span> <span class="k">type</span> <span class="n">token</span> <span class="o">=</span> <span class="p">(</span><span class="n">ltok</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">location</span><span class="p">);</span>

<span class="c1">// A token value, used for tokens such as &#39;1337&#39; (an integer).
</span><span class="c1"></span><span class="k">export</span> <span class="k">type</span> <span class="n">value</span> <span class="o">=</span> <span class="p">(</span><span class="kt">str</span> <span class="o">|</span> <span class="n">rune</span> <span class="o">|</span> <span class="kt">i64</span> <span class="o">|</span> <span class="kt">u64</span> <span class="o">|</span> <span class="kt">f64</span> <span class="o">|</span> <span class="kt">void</span><span class="p">);</span>

<span class="c1">// A location in a source file.
</span><span class="c1"></span><span class="k">export</span> <span class="k">type</span> <span class="n">location</span> <span class="o">=</span> <span class="k">struct</span> <span class="p">{</span>
	<span class="n">path</span><span class="o">:</span> <span class="kt">str</span><span class="p">,</span>
	<span class="n">line</span><span class="o">:</span> <span class="kt">uint</span><span class="p">,</span>
	<span class="n">col</span><span class="o">:</span> <span class="kt">uint</span>
<span class="p">};</span>

<span class="c1">// A lexical token class.
</span><span class="c1"></span><span class="k">export</span> <span class="k">type</span> <span class="n">ltok</span> <span class="o">=</span> <span class="n">enum</span> <span class="kt">uint</span> <span class="p">{</span>
	<span class="n">UNDERSCORE</span><span class="p">,</span>
	<span class="n">ABORT</span><span class="p">,</span>
	<span class="n">ALLOC</span><span class="p">,</span>
	<span class="n">APPEND</span><span class="p">,</span>
	<span class="n">AS</span><span class="p">,</span>
	<span class="c1">// ... continued ...
</span><span class="c1"></span>	<span class="n">EOF</span><span class="p">,</span>
<span class="p">};</span>
</code></pre></div><p>The idea is that when the caller needs another token, they will call <code>lex</code>, and
receive either a token or an error. The purpose of our lex function is to read
out the next character and decide what kind of tokens it might be the start of,
and dispatch to more specific lexing functions to handle each case.</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="k">export</span> <span class="k">fn</span> <span class="n">lex</span><span class="p">(</span><span class="n">lex</span><span class="o">:</span> <span class="o">*</span><span class="n">lexer</span><span class="p">)</span> <span class="p">(</span><span class="n">token</span> <span class="o">|</span> <span class="n">error</span><span class="p">)</span> <span class="o">=</span> <span class="p">{</span>
	<span class="k">let</span> <span class="n">loc</span> <span class="o">=</span> <span class="n">location</span> <span class="p">{</span> <span class="p">...</span> <span class="p">};</span>
	<span class="k">let</span> <span class="n">rn</span><span class="o">:</span> <span class="n">rune</span> <span class="o">=</span> <span class="k">match</span> <span class="p">(</span><span class="n">nextw</span><span class="p">(</span><span class="n">lex</span><span class="p">)</span><span class="o">?</span><span class="p">)</span> <span class="p">{</span>
		<span class="n">_</span><span class="o">:</span> <span class="n">io</span><span class="o">::</span><span class="n">EOF</span> <span class="o">=&gt;</span> <span class="k">return</span> <span class="p">(</span><span class="n">ltok</span><span class="o">::</span><span class="n">EOF</span><span class="p">,</span> <span class="kt">void</span><span class="p">,</span> <span class="n">mkloc</span><span class="p">(</span><span class="n">lex</span><span class="p">)),</span>
		<span class="n">rl</span><span class="o">:</span> <span class="p">(</span><span class="n">rune</span><span class="p">,</span> <span class="n">location</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
			<span class="n">loc</span> <span class="o">=</span> <span class="n">rl</span><span class="mf">.1</span><span class="p">;</span>
			<span class="n">rl</span><span class="mf">.0</span><span class="p">;</span>
		<span class="p">},</span>
	<span class="p">};</span>

	<span class="k">if</span> <span class="p">(</span><span class="n">is_name</span><span class="p">(</span><span class="n">rn</span><span class="p">,</span> <span class="nb">false</span><span class="p">))</span> <span class="p">{</span>
		<span class="n">unget</span><span class="p">(</span><span class="n">lex</span><span class="p">,</span> <span class="n">rn</span><span class="p">);</span>
		<span class="k">return</span> <span class="n">lex_name</span><span class="p">(</span><span class="n">lex</span><span class="p">,</span> <span class="n">loc</span><span class="p">,</span> <span class="nb">true</span><span class="p">);</span>
	<span class="p">};</span>
	<span class="k">if</span> <span class="p">(</span><span class="n">ascii</span><span class="o">::</span><span class="n">isdigit</span><span class="p">(</span><span class="n">rn</span><span class="p">))</span> <span class="p">{</span>
		<span class="n">unget</span><span class="p">(</span><span class="n">lex</span><span class="p">,</span> <span class="n">rn</span><span class="p">);</span>
		<span class="k">return</span> <span class="n">lex_literal</span><span class="p">(</span><span class="n">lex</span><span class="p">,</span> <span class="n">loc</span><span class="p">);</span>
	<span class="p">};</span>

	<span class="k">let</span> <span class="n">tok</span><span class="o">:</span> <span class="n">ltok</span> <span class="o">=</span> <span class="k">switch</span> <span class="p">(</span><span class="n">rn</span><span class="p">)</span> <span class="p">{</span>
		<span class="o">*</span> <span class="o">=&gt;</span> <span class="k">return</span> <span class="n">syntaxerr</span><span class="p">(</span><span class="n">loc</span><span class="p">,</span> <span class="s">&#34;invalid character&#34;</span><span class="p">),</span>
		<span class="sc">&#39;&#34;&#39;</span><span class="p">,</span> <span class="sc">&#39;\&#39;&#39;</span> <span class="o">=&gt;</span> <span class="p">{</span>
			<span class="n">unget</span><span class="p">(</span><span class="n">lex</span><span class="p">,</span> <span class="n">rn</span><span class="p">);</span>
			<span class="k">return</span> <span class="n">lex_rn_str</span><span class="p">(</span><span class="n">lex</span><span class="p">,</span> <span class="n">loc</span><span class="p">);</span>
		<span class="p">},</span>
		<span class="sc">&#39;.&#39;</span><span class="p">,</span> <span class="sc">&#39;&lt;&#39;</span><span class="p">,</span> <span class="sc">&#39;&gt;&#39;</span> <span class="o">=&gt;</span> <span class="k">return</span> <span class="n">lex3</span><span class="p">(</span><span class="n">lex</span><span class="p">,</span> <span class="n">loc</span><span class="p">,</span> <span class="n">rn</span><span class="p">),</span>
		<span class="sc">&#39;^&#39;</span><span class="p">,</span> <span class="sc">&#39;*&#39;</span><span class="p">,</span> <span class="sc">&#39;%&#39;</span><span class="p">,</span> <span class="sc">&#39;/&#39;</span><span class="p">,</span> <span class="sc">&#39;+&#39;</span><span class="p">,</span> <span class="sc">&#39;-&#39;</span><span class="p">,</span> <span class="sc">&#39;:&#39;</span><span class="p">,</span> <span class="sc">&#39;!&#39;</span><span class="p">,</span> <span class="sc">&#39;&amp;&#39;</span><span class="p">,</span> <span class="sc">&#39;|&#39;</span><span class="p">,</span> <span class="sc">&#39;=&#39;</span> <span class="o">=&gt;</span> <span class="p">{</span>
			<span class="k">return</span> <span class="n">lex2</span><span class="p">(</span><span class="n">lex</span><span class="p">,</span> <span class="n">loc</span><span class="p">,</span> <span class="n">rn</span><span class="p">);</span>
		<span class="p">},</span>
		<span class="sc">&#39;~&#39;</span> <span class="o">=&gt;</span> <span class="n">ltok</span><span class="o">::</span><span class="n">BNOT</span><span class="p">,</span>
		<span class="sc">&#39;,&#39;</span> <span class="o">=&gt;</span> <span class="n">ltok</span><span class="o">::</span><span class="n">COMMA</span><span class="p">,</span>
		<span class="sc">&#39;{&#39;</span> <span class="o">=&gt;</span> <span class="n">ltok</span><span class="o">::</span><span class="n">LBRACE</span><span class="p">,</span>
		<span class="sc">&#39;[&#39;</span> <span class="o">=&gt;</span> <span class="n">ltok</span><span class="o">::</span><span class="n">LBRACKET</span><span class="p">,</span>
		<span class="sc">&#39;(&#39;</span> <span class="o">=&gt;</span> <span class="n">ltok</span><span class="o">::</span><span class="n">LPAREN</span><span class="p">,</span>
		<span class="sc">&#39;}&#39;</span> <span class="o">=&gt;</span> <span class="n">ltok</span><span class="o">::</span><span class="n">RBRACE</span><span class="p">,</span>
		<span class="sc">&#39;]&#39;</span> <span class="o">=&gt;</span> <span class="n">ltok</span><span class="o">::</span><span class="n">RBRACKET</span><span class="p">,</span>
		<span class="sc">&#39;)&#39;</span> <span class="o">=&gt;</span> <span class="n">ltok</span><span class="o">::</span><span class="n">RPAREN</span><span class="p">,</span>
		<span class="sc">&#39;;&#39;</span> <span class="o">=&gt;</span> <span class="n">ltok</span><span class="o">::</span><span class="n">SEMICOLON</span><span class="p">,</span>
		<span class="sc">&#39;?&#39;</span> <span class="o">=&gt;</span> <span class="n">ltok</span><span class="o">::</span><span class="n">QUESTION</span><span class="p">,</span>
	<span class="p">};</span>
	<span class="k">return</span> <span class="p">(</span><span class="n">tok</span><span class="p">,</span> <span class="kt">void</span><span class="p">,</span> <span class="n">loc</span><span class="p">);</span>
<span class="p">};</span>
</code></pre></div><p>Aside from the EOF case, and simple single-character operators like &ldquo;;&rdquo;, both of
which this function handles itself, its role is to dispatch work to various
sub-lexers.</p>
<details>
	<summary>Expand me to read the helper functions</summary>
<pre>
fn nextw(lex: *lexer) ((rune, location) | io::EOF | io::error) = {
	for (true) {
		let loc = mkloc(lex);
		match (next(lex)) {
			e: (io::error | io::EOF) => return e,
			r: rune => if (!ascii::isspace(r)) {
				return (r, loc);
			} else {
				free(lex.comment);
				lex.comment = "";
			},
		};
	};
	abort();
};
</pre>
<pre>
fn unget(lex: *lexer, r: (rune | io::EOF)) void = {
	if (!(lex.rb[0] is void)) {
		assert(lex.rb[1] is void, "ungot too many runes");
		lex.rb[1] = lex.rb[0];
	};
	lex.rb[0] = r;
};
</pre>
<pre>
fn is_name(r: rune, num: bool) bool =
	ascii::isalpha(r) || r == '_' || r == '@' || (num && ascii::isdigit(r));
</pre>
</details>
<p>The sub-lexers handle more specific cases. The lex_name function handles things
which look like identifiers, including keywords; the lex_literal function
handles things which look like literals (e.g. &ldquo;1234&rdquo;); lex_rn_str handles rune
and string literals (e.g. &ldquo;hello world&rdquo; and &lsquo;\n&rsquo;); and lex2 and lex3
respectively handle two- and three-character operators like &ldquo;&amp;&amp;&rdquo; and &ldquo;&gt;&gt;=&rdquo;.</p>
<p>lex_name is the most complicated of these. Because the only thing which
distinguishes a keyword from an identifier is that the former matches a specific
list of strings, we start by reading a &ldquo;name&rdquo; into a buffer, then binary
searching against a list of known keywords to see if it matches something there.
To facilitate this, &ldquo;bmap&rdquo; is a pre-sorted array of keyword names.</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="k">const</span> <span class="n">bmap</span><span class="o">:</span> <span class="p">[</span><span class="n">_</span><span class="p">]</span><span class="kt">str</span> <span class="o">=</span> <span class="p">[</span>
	<span class="c1">// Keep me alpha-sorted and consistent with the ltok enum.
</span><span class="c1"></span>	<span class="s">&#34;_&#34;</span><span class="p">,</span>
	<span class="s">&#34;abort&#34;</span><span class="p">,</span>
	<span class="s">&#34;alloc&#34;</span><span class="p">,</span>
	<span class="s">&#34;append&#34;</span><span class="p">,</span>
	<span class="s">&#34;as&#34;</span><span class="p">,</span>
	<span class="s">&#34;assert&#34;</span><span class="p">,</span>
	<span class="s">&#34;bool&#34;</span><span class="p">,</span>
	<span class="c1">// ...
</span><span class="c1"></span><span class="p">];</span>

<span class="k">fn</span> <span class="n">lex_name</span><span class="p">(</span><span class="n">lex</span><span class="o">:</span> <span class="o">*</span><span class="n">lexer</span><span class="p">,</span> <span class="n">loc</span><span class="o">:</span> <span class="n">location</span><span class="p">,</span> <span class="n">keyword</span><span class="o">:</span> <span class="kt">bool</span><span class="p">)</span> <span class="p">(</span><span class="n">token</span> <span class="o">|</span> <span class="n">error</span><span class="p">)</span> <span class="o">=</span> <span class="p">{</span>
	<span class="k">let</span> <span class="n">buf</span> <span class="o">=</span> <span class="n">strio</span><span class="o">::</span><span class="n">dynamic</span><span class="p">();</span>
	<span class="k">match</span> <span class="p">(</span><span class="n">next</span><span class="p">(</span><span class="n">lex</span><span class="p">))</span> <span class="p">{</span>
		<span class="n">r</span><span class="o">:</span> <span class="n">rune</span> <span class="o">=&gt;</span> <span class="p">{</span>
			<span class="k">assert</span><span class="p">(</span><span class="n">is_name</span><span class="p">(</span><span class="n">r</span><span class="p">,</span> <span class="nb">false</span><span class="p">));</span>
			<span class="n">strio</span><span class="o">::</span><span class="n">appendrune</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="n">r</span><span class="p">);</span>
		<span class="p">},</span>
		<span class="n">_</span><span class="o">:</span> <span class="p">(</span><span class="n">io</span><span class="o">::</span><span class="n">EOF</span> <span class="o">|</span> <span class="n">io</span><span class="o">::</span><span class="n">error</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="k">abort</span><span class="p">(),</span> <span class="c1">// Invariant
</span><span class="c1"></span>	<span class="p">};</span>

	<span class="k">for</span> <span class="p">(</span><span class="nb">true</span><span class="p">)</span> <span class="k">match</span> <span class="p">(</span><span class="n">next</span><span class="p">(</span><span class="n">lex</span><span class="p">)</span><span class="o">?</span><span class="p">)</span> <span class="p">{</span>
		<span class="n">_</span><span class="o">:</span> <span class="n">io</span><span class="o">::</span><span class="n">EOF</span> <span class="o">=&gt;</span> <span class="n">break</span><span class="p">,</span>
		<span class="n">r</span><span class="o">:</span> <span class="n">rune</span> <span class="o">=&gt;</span> <span class="p">{</span>
			<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">is_name</span><span class="p">(</span><span class="n">r</span><span class="p">,</span> <span class="nb">true</span><span class="p">))</span> <span class="p">{</span>
				<span class="n">unget</span><span class="p">(</span><span class="n">lex</span><span class="p">,</span> <span class="n">r</span><span class="p">);</span>
				<span class="n">break</span><span class="p">;</span>
			<span class="p">};</span>
			<span class="n">strio</span><span class="o">::</span><span class="n">appendrune</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="n">r</span><span class="p">);</span>
		<span class="p">},</span>
	<span class="p">};</span>

	<span class="k">let</span> <span class="n">name</span> <span class="o">=</span> <span class="n">strio</span><span class="o">::</span><span class="n">finish</span><span class="p">(</span><span class="n">buf</span><span class="p">);</span>
	<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">keyword</span><span class="p">)</span> <span class="p">{</span>
		<span class="k">return</span> <span class="p">(</span><span class="n">ltok</span><span class="o">::</span><span class="n">NAME</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">loc</span><span class="p">);</span>
	<span class="p">};</span>

	<span class="k">return</span> <span class="k">match</span> <span class="p">(</span><span class="n">sort</span><span class="o">::</span><span class="n">search</span><span class="p">(</span><span class="n">bmap</span><span class="p">[..</span><span class="n">ltok</span><span class="o">::</span><span class="n">LAST_KEYWORD</span><span class="o">+</span><span class="mi">1</span><span class="p">],</span>
			<span class="k">size</span><span class="p">(</span><span class="kt">str</span><span class="p">),</span> <span class="o">&amp;</span><span class="n">name</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">namecmp</span><span class="p">))</span> <span class="p">{</span>
		<span class="kt">null</span> <span class="o">=&gt;</span> <span class="p">(</span><span class="n">ltok</span><span class="o">::</span><span class="n">NAME</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">loc</span><span class="p">),</span>
		<span class="n">v</span><span class="o">:</span> <span class="o">*</span><span class="kt">void</span> <span class="o">=&gt;</span> <span class="p">{</span>
			<span class="k">defer</span> <span class="k">free</span><span class="p">(</span><span class="n">name</span><span class="p">);</span>
			<span class="k">let</span> <span class="n">tok</span> <span class="o">=</span> <span class="n">v</span><span class="o">:</span> <span class="kt">uintptr</span> <span class="o">-</span> <span class="o">&amp;</span><span class="n">bmap</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">:</span> <span class="kt">uintptr</span><span class="p">;</span>
			<span class="n">tok</span> <span class="o">/=</span> <span class="k">size</span><span class="p">(</span><span class="kt">str</span><span class="p">)</span><span class="o">:</span> <span class="kt">uintptr</span><span class="p">;</span>
			<span class="p">(</span><span class="n">tok</span><span class="o">:</span> <span class="n">ltok</span><span class="p">,</span> <span class="kt">void</span><span class="p">,</span> <span class="n">loc</span><span class="p">);</span>
		<span class="p">},</span>
	<span class="p">};</span>
<span class="p">};</span>
</code></pre></div><p>The rest of the code is more of the same, but I&rsquo;ve <a href="https://paste.sr.ht/~sircmpwn/25871787b0d41db2b0af573ba1c93e1b6438b942">put it up here</a> if
you want to read it.</p>
<p>Let&rsquo;s move on to parsing: we need to turn this one dimensional stream of tokens
into an structured form: the <em>Abstract Syntax Tree</em>. Consider the following
sample code:</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="k">let</span> <span class="n">x</span><span class="o">:</span> <span class="kt">int</span> <span class="o">=</span> <span class="n">add2</span><span class="p">(</span><span class="mi">40</span><span class="p">,</span> <span class="mi">2</span><span class="p">);</span>
</code></pre></div><p>Our token stream looks like this:</p>
<pre><code>let x : int = add2 ( 40 , 2 ) ;
</code></pre><p>But what we need is something more structured, like this:</p>
<pre><code>binding
	name=&quot;x&quot;
	type=&quot;int&quot;
	initializer=call-expression
	=&gt;	func=&quot;add2&quot;
		parameters
			constant value=&quot;40&quot;
			constant value=&quot;2&quot;
</code></pre><p>We know at each step what kinds of tokens are valid in each situation. After we
see &ldquo;let&rdquo;, we know that we&rsquo;re parsing a binding, so we look for a name (&ldquo;x&rdquo;)
and a colon token, a type for the variable, an equals sign, and an expression
which initializes it. To parse the initializer, we see an identifier, &ldquo;add2&rdquo;,
then an open parenthesis, so we know we&rsquo;re in a call expression, and we can
start parsing arguments.</p>
<p>To make our parser code expressive, and to handle errors neatly, we&rsquo;re going to
implement a few helper function that lets us describe these states in terms of
what the parser wants from the lexer. We have a few functions to accomplish
this:</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="c1">// Requires the next token to have a matching ltok. Returns that token, or an error.
</span><span class="c1"></span><span class="k">fn</span> <span class="n">want</span><span class="p">(</span><span class="n">lexer</span><span class="o">:</span> <span class="o">*</span><span class="n">lex</span><span class="o">::</span><span class="n">lexer</span><span class="p">,</span> <span class="n">want</span><span class="o">:</span> <span class="n">lex</span><span class="o">::</span><span class="n">ltok</span><span class="p">...)</span> <span class="p">(</span><span class="n">lex</span><span class="o">::</span><span class="n">token</span> <span class="o">|</span> <span class="n">error</span><span class="p">)</span> <span class="o">=</span> <span class="p">{</span>
	<span class="k">let</span> <span class="n">tok</span> <span class="o">=</span> <span class="n">lex</span><span class="o">::</span><span class="n">lex</span><span class="p">(</span><span class="n">lexer</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
	<span class="k">if</span> <span class="p">(</span><span class="k">len</span><span class="p">(</span><span class="n">want</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
		<span class="k">return</span> <span class="n">tok</span><span class="p">;</span>
	<span class="p">};</span>
	<span class="k">for</span> <span class="p">(</span><span class="k">let</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="n">z</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="k">len</span><span class="p">(</span><span class="n">want</span><span class="p">);</span> <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
		<span class="k">if</span> <span class="p">(</span><span class="n">tok</span><span class="mf">.0</span> <span class="o">==</span> <span class="n">want</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="p">{</span>
			<span class="k">return</span> <span class="n">tok</span><span class="p">;</span>
		<span class="p">};</span>
	<span class="p">};</span>

	<span class="k">let</span> <span class="n">buf</span> <span class="o">=</span> <span class="n">strio</span><span class="o">::</span><span class="n">dynamic</span><span class="p">();</span>
	<span class="k">defer</span> <span class="n">io</span><span class="o">::</span><span class="n">close</span><span class="p">(</span><span class="n">buf</span><span class="p">);</span>
	<span class="k">for</span> <span class="p">(</span><span class="k">let</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="n">z</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="k">len</span><span class="p">(</span><span class="n">want</span><span class="p">);</span> <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
		<span class="n">fmt</span><span class="o">::</span><span class="n">fprintf</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="s">&#34;&#39;{}&#39;&#34;</span><span class="p">,</span> <span class="n">lex</span><span class="o">::</span><span class="n">tokstr</span><span class="p">((</span><span class="n">want</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="kt">void</span><span class="p">,</span> <span class="n">mkloc</span><span class="p">(</span><span class="n">lexer</span><span class="p">))));</span>
		<span class="k">if</span> <span class="p">(</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">&lt;</span> <span class="k">len</span><span class="p">(</span><span class="n">want</span><span class="p">))</span> <span class="p">{</span>
			<span class="n">fmt</span><span class="o">::</span><span class="n">fprint</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="s">&#34;, &#34;</span><span class="p">);</span>
		<span class="p">};</span>
	<span class="p">};</span>
	<span class="k">return</span> <span class="n">syntaxerr</span><span class="p">(</span><span class="n">mkloc</span><span class="p">(</span><span class="n">lexer</span><span class="p">),</span> <span class="s">&#34;Unexpected &#39;{}&#39;, was expecting {}&#34;</span><span class="p">,</span>
		<span class="n">lex</span><span class="o">::</span><span class="n">tokstr</span><span class="p">(</span><span class="n">tok</span><span class="p">),</span> <span class="n">strio</span><span class="o">::</span><span class="n">string</span><span class="p">(</span><span class="n">buf</span><span class="p">));</span>
<span class="p">};</span>

<span class="c1">// Looks for a matching ltok from the lexer, and if not present, unlexes the
</span><span class="c1">// token and returns void. If found, the token is consumed from the lexer and is
</span><span class="c1">// returned.
</span><span class="c1"></span><span class="k">fn</span> <span class="n">try</span><span class="p">(</span>
	<span class="n">lexer</span><span class="o">:</span> <span class="o">*</span><span class="n">lex</span><span class="o">::</span><span class="n">lexer</span><span class="p">,</span>
	<span class="n">want</span><span class="o">:</span> <span class="n">lex</span><span class="o">::</span><span class="n">ltok</span><span class="p">...</span>
<span class="p">)</span> <span class="p">(</span><span class="n">lex</span><span class="o">::</span><span class="n">token</span> <span class="o">|</span> <span class="n">error</span> <span class="o">|</span> <span class="kt">void</span><span class="p">)</span> <span class="o">=</span> <span class="p">{</span>
	<span class="k">let</span> <span class="n">tok</span> <span class="o">=</span> <span class="n">lex</span><span class="o">::</span><span class="n">lex</span><span class="p">(</span><span class="n">lexer</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
	<span class="k">assert</span><span class="p">(</span><span class="k">len</span><span class="p">(</span><span class="n">want</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">);</span>
	<span class="k">for</span> <span class="p">(</span><span class="k">let</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="n">z</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="k">len</span><span class="p">(</span><span class="n">want</span><span class="p">);</span> <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
		<span class="k">if</span> <span class="p">(</span><span class="n">tok</span><span class="mf">.0</span> <span class="o">==</span> <span class="n">want</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="p">{</span>
			<span class="k">return</span> <span class="n">tok</span><span class="p">;</span>
		<span class="p">};</span>
	<span class="p">};</span>
	<span class="n">lex</span><span class="o">::</span><span class="n">unlex</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="n">tok</span><span class="p">);</span>
<span class="p">};</span>

<span class="c1">// Looks for a matching ltok from the lexer, unlexes the token, and returns
</span><span class="c1">// it; or void if it was not a ltok.
</span><span class="c1"></span><span class="k">fn</span> <span class="n">peek</span><span class="p">(</span>
	<span class="n">lexer</span><span class="o">:</span> <span class="o">*</span><span class="n">lex</span><span class="o">::</span><span class="n">lexer</span><span class="p">,</span>
	<span class="n">want</span><span class="o">:</span> <span class="n">lex</span><span class="o">::</span><span class="n">ltok</span><span class="p">...</span>
<span class="p">)</span> <span class="p">(</span><span class="n">lex</span><span class="o">::</span><span class="n">token</span> <span class="o">|</span> <span class="n">error</span> <span class="o">|</span> <span class="kt">void</span><span class="p">)</span> <span class="o">=</span> <span class="p">{</span>
	<span class="k">let</span> <span class="n">tok</span> <span class="o">=</span> <span class="n">lex</span><span class="o">::</span><span class="n">lex</span><span class="p">(</span><span class="n">lexer</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
	<span class="n">lex</span><span class="o">::</span><span class="n">unlex</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="n">tok</span><span class="p">);</span>
	<span class="k">if</span> <span class="p">(</span><span class="k">len</span><span class="p">(</span><span class="n">want</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
		<span class="k">return</span> <span class="n">tok</span><span class="p">;</span>
	<span class="p">};</span>
	<span class="k">for</span> <span class="p">(</span><span class="k">let</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="n">z</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="k">len</span><span class="p">(</span><span class="n">want</span><span class="p">);</span> <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
		<span class="k">if</span> <span class="p">(</span><span class="n">tok</span><span class="mf">.0</span> <span class="o">==</span> <span class="n">want</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="p">{</span>
			<span class="k">return</span> <span class="n">tok</span><span class="p">;</span>
		<span class="p">};</span>
	<span class="p">};</span>
<span class="p">};</span>
</code></pre></div><p>Let&rsquo;s say we&rsquo;re looking for a binding like our sample code to show up next. The
grammar from the spec is as follows:</p>
<p><img src="https://l.sr.ht/W2AY.png" alt=""></p>
<p>And here&rsquo;s the code that parses that:</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="k">fn</span> <span class="n">binding</span><span class="p">(</span><span class="n">lexer</span><span class="o">:</span> <span class="o">*</span><span class="n">lex</span><span class="o">::</span><span class="n">lexer</span><span class="p">)</span> <span class="p">(</span><span class="n">ast</span><span class="o">::</span><span class="n">expr</span> <span class="o">|</span> <span class="n">error</span><span class="p">)</span> <span class="o">=</span> <span class="p">{</span>
	<span class="k">const</span> <span class="n">is_static</span><span class="o">:</span> <span class="kt">bool</span> <span class="o">=</span> <span class="n">try</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">STATIC</span><span class="p">)</span><span class="o">?</span> <span class="n">is</span> <span class="n">lex</span><span class="o">::</span><span class="n">token</span><span class="p">;</span>
	<span class="k">const</span> <span class="n">is_const</span> <span class="o">=</span> <span class="k">switch</span> <span class="p">(</span><span class="n">want</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">LET</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">CONST</span><span class="p">)</span><span class="o">?</span><span class="mf">.0</span><span class="p">)</span> <span class="p">{</span>
		<span class="n">ltok</span><span class="o">::</span><span class="n">LET</span> <span class="o">=&gt;</span> <span class="nb">false</span><span class="p">,</span>
		<span class="n">ltok</span><span class="o">::</span><span class="n">CONST</span> <span class="o">=&gt;</span> <span class="nb">true</span><span class="p">,</span>
	<span class="p">};</span>

	<span class="k">let</span> <span class="n">bindings</span><span class="o">:</span> <span class="p">[]</span><span class="n">ast</span><span class="o">::</span><span class="n">binding</span> <span class="o">=</span> <span class="p">[];</span>
	<span class="k">for</span> <span class="p">(</span><span class="nb">true</span><span class="p">)</span> <span class="p">{</span>
		<span class="k">const</span> <span class="n">name</span> <span class="o">=</span> <span class="n">want</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">NAME</span><span class="p">)</span><span class="o">?</span><span class="mf">.1</span> <span class="n">as</span> <span class="kt">str</span><span class="p">;</span>
		<span class="k">const</span> <span class="n">btype</span><span class="o">:</span> <span class="kt">nullable</span> <span class="o">*</span><span class="n">ast</span><span class="o">::</span><span class="n">_type</span> <span class="o">=</span>
			<span class="k">if</span> <span class="p">(</span><span class="n">try</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">COLON</span><span class="p">)</span><span class="o">?</span> <span class="n">is</span> <span class="n">lex</span><span class="o">::</span><span class="n">token</span><span class="p">)</span> <span class="p">{</span>
				<span class="k">alloc</span><span class="p">(</span><span class="n">_type</span><span class="p">(</span><span class="n">lexer</span><span class="p">)</span><span class="o">?</span><span class="p">);</span>
			<span class="p">}</span> <span class="k">else</span> <span class="kt">null</span><span class="p">;</span>
		<span class="n">want</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">EQUAL</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
		<span class="k">const</span> <span class="n">init</span> <span class="o">=</span> <span class="k">alloc</span><span class="p">(</span><span class="n">expression</span><span class="p">(</span><span class="n">lexer</span><span class="p">)</span><span class="o">?</span><span class="p">);</span>
		<span class="k">append</span><span class="p">(</span><span class="n">bindings</span><span class="p">,</span> <span class="n">ast</span><span class="o">::</span><span class="n">binding</span> <span class="p">{</span>
			<span class="n">name</span> <span class="o">=</span> <span class="n">name</span><span class="p">,</span>
			<span class="n">_type</span> <span class="o">=</span> <span class="n">btype</span><span class="p">,</span>
			<span class="n">init</span> <span class="o">=</span> <span class="n">init</span><span class="p">,</span>
		<span class="p">});</span>
		<span class="k">match</span> <span class="p">(</span><span class="n">try</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">COMMA</span><span class="p">)</span><span class="o">?</span><span class="p">)</span> <span class="p">{</span>
			<span class="n">_</span><span class="o">:</span> <span class="kt">void</span> <span class="o">=&gt;</span> <span class="n">break</span><span class="p">,</span>
			<span class="n">_</span><span class="o">:</span> <span class="n">lex</span><span class="o">::</span><span class="n">token</span> <span class="o">=&gt;</span> <span class="kt">void</span><span class="p">,</span>
		<span class="p">};</span>
	<span class="p">};</span>

	<span class="k">return</span> <span class="n">ast</span><span class="o">::</span><span class="n">binding_expr</span> <span class="p">{</span>
		<span class="n">is_static</span> <span class="o">=</span> <span class="n">is_static</span><span class="p">,</span>
		<span class="n">is_const</span> <span class="o">=</span> <span class="n">is_const</span><span class="p">,</span>
		<span class="n">bindings</span> <span class="o">=</span> <span class="n">bindings</span><span class="p">,</span>
	<span class="p">};</span>
<span class="p">};</span>
</code></pre></div><p>Hopefully the flow of this code is fairly apparent. The goal is to fill in the
following AST structure:</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="c1">// A single variable biding. For example:
</span><span class="c1">//
</span><span class="c1">// 	foo: int = bar
</span><span class="c1"></span><span class="k">export</span> <span class="k">type</span> <span class="n">binding</span> <span class="o">=</span> <span class="k">struct</span> <span class="p">{</span>
	<span class="n">name</span><span class="o">:</span> <span class="kt">str</span><span class="p">,</span>
	<span class="n">_type</span><span class="o">:</span> <span class="kt">nullable</span> <span class="o">*</span><span class="n">_type</span><span class="p">,</span>
	<span class="n">init</span><span class="o">:</span> <span class="o">*</span><span class="n">expr</span><span class="p">,</span>
<span class="p">};</span>

<span class="c1">// A variable binding expression. For example:
</span><span class="c1">//
</span><span class="c1">// 	let foo: int = bar, ...
</span><span class="c1"></span><span class="k">export</span> <span class="k">type</span> <span class="n">binding_expr</span> <span class="o">=</span> <span class="k">struct</span> <span class="p">{</span>
	<span class="n">is_static</span><span class="o">:</span> <span class="kt">bool</span><span class="p">,</span>
	<span class="n">is_const</span><span class="o">:</span> <span class="kt">bool</span><span class="p">,</span>
	<span class="n">bindings</span><span class="o">:</span> <span class="p">[]</span><span class="n">binding</span><span class="p">,</span>
<span class="p">};</span>
</code></pre></div><p>The rest of the code is pretty similar, though some corners of the grammar are a
bit hairier than others. One example is how we parse infix operators for binary
arithmetic expressions (such as &ldquo;2 + 2&rdquo;):</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="k">fn</span> <span class="n">binarithm</span><span class="p">(</span>
	<span class="n">lexer</span><span class="o">:</span> <span class="o">*</span><span class="n">lex</span><span class="o">::</span><span class="n">lexer</span><span class="p">,</span>
	<span class="n">lvalue</span><span class="o">:</span> <span class="p">(</span><span class="n">ast</span><span class="o">::</span><span class="n">expr</span> <span class="o">|</span> <span class="kt">void</span><span class="p">),</span>
	<span class="n">i</span><span class="o">:</span> <span class="kt">int</span><span class="p">,</span>
<span class="p">)</span> <span class="p">(</span><span class="n">ast</span><span class="o">::</span><span class="n">expr</span> <span class="o">|</span> <span class="n">error</span><span class="p">)</span> <span class="o">=</span> <span class="p">{</span>
	<span class="c1">// Precedence climbing parser
</span><span class="c1"></span>	<span class="c1">// https://en.wikipedia.org/wiki/Operator-precedence_parser
</span><span class="c1"></span>	<span class="k">let</span> <span class="n">lvalue</span> <span class="o">=</span> <span class="k">match</span> <span class="p">(</span><span class="n">lvalue</span><span class="p">)</span> <span class="p">{</span>
		<span class="n">_</span><span class="o">:</span> <span class="kt">void</span> <span class="o">=&gt;</span> <span class="n">cast</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="kt">void</span><span class="p">)</span><span class="o">?</span><span class="p">,</span>
		<span class="n">expr</span><span class="o">:</span> <span class="n">ast</span><span class="o">::</span><span class="n">expr</span> <span class="o">=&gt;</span> <span class="n">expr</span><span class="p">,</span>
	<span class="p">};</span>

	<span class="k">let</span> <span class="n">tok</span> <span class="o">=</span> <span class="n">lex</span><span class="o">::</span><span class="n">lex</span><span class="p">(</span><span class="n">lexer</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
	<span class="k">for</span> <span class="p">(</span><span class="k">let</span> <span class="n">j</span> <span class="o">=</span> <span class="n">precedence</span><span class="p">(</span><span class="n">tok</span><span class="p">);</span> <span class="n">j</span> <span class="o">&gt;=</span> <span class="n">i</span><span class="p">;</span> <span class="n">j</span> <span class="o">=</span> <span class="n">precedence</span><span class="p">(</span><span class="n">tok</span><span class="p">))</span> <span class="p">{</span>
		<span class="k">const</span> <span class="n">op</span> <span class="o">=</span> <span class="n">binop_for_tok</span><span class="p">(</span><span class="n">tok</span><span class="p">);</span>

		<span class="k">let</span> <span class="n">rvalue</span> <span class="o">=</span> <span class="n">cast</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="kt">void</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
		<span class="n">tok</span> <span class="o">=</span> <span class="n">lex</span><span class="o">::</span><span class="n">lex</span><span class="p">(</span><span class="n">lexer</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>

		<span class="k">for</span> <span class="p">(</span><span class="k">let</span> <span class="n">k</span> <span class="o">=</span> <span class="n">precedence</span><span class="p">(</span><span class="n">tok</span><span class="p">);</span> <span class="n">k</span> <span class="o">&gt;</span> <span class="n">j</span><span class="p">;</span> <span class="n">k</span> <span class="o">=</span> <span class="n">precedence</span><span class="p">(</span><span class="n">tok</span><span class="p">))</span> <span class="p">{</span>
			<span class="n">lex</span><span class="o">::</span><span class="n">unlex</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="n">tok</span><span class="p">);</span>
			<span class="n">rvalue</span> <span class="o">=</span> <span class="n">binarithm</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="n">rvalue</span><span class="p">,</span> <span class="n">k</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
			<span class="n">tok</span> <span class="o">=</span> <span class="n">lex</span><span class="o">::</span><span class="n">lex</span><span class="p">(</span><span class="n">lexer</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
		<span class="p">};</span>

		<span class="k">let</span> <span class="n">expr</span> <span class="o">=</span> <span class="n">ast</span><span class="o">::</span><span class="n">binarithm_expr</span> <span class="p">{</span>
			<span class="n">op</span> <span class="o">=</span> <span class="n">op</span><span class="p">,</span>
			<span class="n">lvalue</span> <span class="o">=</span> <span class="k">alloc</span><span class="p">(</span><span class="n">lvalue</span><span class="p">),</span>
			<span class="n">rvalue</span> <span class="o">=</span> <span class="k">alloc</span><span class="p">(</span><span class="n">rvalue</span><span class="p">),</span>
		<span class="p">};</span>
		<span class="n">lvalue</span> <span class="o">=</span> <span class="n">expr</span><span class="p">;</span>
	<span class="p">};</span>

	<span class="n">lex</span><span class="o">::</span><span class="n">unlex</span><span class="p">(</span><span class="n">lexer</span><span class="p">,</span> <span class="n">tok</span><span class="p">);</span>
	<span class="k">return</span> <span class="n">lvalue</span><span class="p">;</span>
<span class="p">};</span>

<span class="k">fn</span> <span class="n">precedence</span><span class="p">(</span><span class="n">tok</span><span class="o">:</span> <span class="n">lex</span><span class="o">::</span><span class="n">token</span><span class="p">)</span> <span class="kt">int</span> <span class="o">=</span> <span class="k">switch</span> <span class="p">(</span><span class="n">tok</span><span class="mf">.0</span><span class="p">)</span> <span class="p">{</span>
	<span class="n">ltok</span><span class="o">::</span><span class="n">LOR</span> <span class="o">=&gt;</span> <span class="mi">0</span><span class="p">,</span>
	<span class="n">ltok</span><span class="o">::</span><span class="n">LXOR</span> <span class="o">=&gt;</span> <span class="mi">1</span><span class="p">,</span>
	<span class="n">ltok</span><span class="o">::</span><span class="n">LAND</span> <span class="o">=&gt;</span> <span class="mi">2</span><span class="p">,</span>
	<span class="n">ltok</span><span class="o">::</span><span class="n">LEQUAL</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">NEQUAL</span> <span class="o">=&gt;</span> <span class="mi">3</span><span class="p">,</span>
	<span class="n">ltok</span><span class="o">::</span><span class="n">LESS</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">LESSEQ</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">GREATER</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">GREATEREQ</span> <span class="o">=&gt;</span> <span class="mi">4</span><span class="p">,</span>
	<span class="n">ltok</span><span class="o">::</span><span class="n">BOR</span> <span class="o">=&gt;</span> <span class="mi">5</span><span class="p">,</span>
	<span class="n">ltok</span><span class="o">::</span><span class="n">BXOR</span> <span class="o">=&gt;</span> <span class="mi">6</span><span class="p">,</span>
	<span class="n">ltok</span><span class="o">::</span><span class="n">BAND</span> <span class="o">=&gt;</span> <span class="mi">7</span><span class="p">,</span>
	<span class="n">ltok</span><span class="o">::</span><span class="n">LSHIFT</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">RSHIFT</span> <span class="o">=&gt;</span> <span class="mi">8</span><span class="p">,</span>
	<span class="n">ltok</span><span class="o">::</span><span class="n">PLUS</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">MINUS</span> <span class="o">=&gt;</span> <span class="mi">9</span><span class="p">,</span>
	<span class="n">ltok</span><span class="o">::</span><span class="n">TIMES</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">DIV</span><span class="p">,</span> <span class="n">ltok</span><span class="o">::</span><span class="n">MODULO</span> <span class="o">=&gt;</span> <span class="mi">10</span><span class="p">,</span>
	<span class="o">*</span> <span class="o">=&gt;</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span>
<span class="p">};</span>
</code></pre></div><p>I don&rsquo;t really grok this algorithm, to be honest, but hey, it works. Whenever I
write a precedence climbing parser, I&rsquo;ll stare at the Wikipedia page for 15
minutes, quickly write a parser, and then immediately forget how it works. Maybe
I&rsquo;ll write a blog post about it someday.</p>
<p>Anyway, ultimately, this code lives in our standard library and is used for
several things, including our (early in development) self-hosted compiler.
Here&rsquo;s an example of its usage, taken from our documentation generator:</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="k">fn</span> <span class="n">scan</span><span class="p">(</span><span class="n">path</span><span class="o">:</span> <span class="kt">str</span><span class="p">)</span> <span class="p">(</span><span class="n">ast</span><span class="o">::</span><span class="n">subunit</span> <span class="o">|</span> <span class="n">error</span><span class="p">)</span> <span class="o">=</span> <span class="p">{</span>
	<span class="k">const</span> <span class="n">input</span> <span class="o">=</span> <span class="k">match</span> <span class="p">(</span><span class="n">os</span><span class="o">::</span><span class="n">open</span><span class="p">(</span><span class="n">path</span><span class="p">))</span> <span class="p">{</span>
		<span class="n">s</span><span class="o">:</span> <span class="o">*</span><span class="n">io</span><span class="o">::</span><span class="n">stream</span> <span class="o">=&gt;</span> <span class="n">s</span><span class="p">,</span>
		<span class="n">err</span><span class="o">:</span> <span class="n">fs</span><span class="o">::</span><span class="n">error</span> <span class="o">=&gt;</span> <span class="n">fmt</span><span class="o">::</span><span class="n">fatal</span><span class="p">(</span><span class="s">&#34;Error reading {}: {}&#34;</span><span class="p">,</span>
			<span class="n">path</span><span class="p">,</span> <span class="n">fs</span><span class="o">::</span><span class="n">strerror</span><span class="p">(</span><span class="n">err</span><span class="p">)),</span>
	<span class="p">};</span>
	<span class="k">defer</span> <span class="n">io</span><span class="o">::</span><span class="n">close</span><span class="p">(</span><span class="n">input</span><span class="p">);</span>
	<span class="k">const</span> <span class="n">lexer</span> <span class="o">=</span> <span class="n">lex</span><span class="o">::</span><span class="n">init</span><span class="p">(</span><span class="n">input</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">lex</span><span class="o">::</span><span class="n">flags</span><span class="o">::</span><span class="n">COMMENTS</span><span class="p">);</span>
	<span class="k">return</span> <span class="n">parse</span><span class="o">::</span><span class="n">subunit</span><span class="p">(</span><span class="o">&amp;</span><span class="n">lexer</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="p">};</span>
</code></pre></div><p>Where the &ldquo;ast::subunit&rdquo; type is:</p>
<div class="highlight"><pre class="chroma"><code class="language-hare" data-lang="hare"><span class="c1">// A sub-unit, typically representing a single source file.
</span><span class="c1"></span><span class="k">export</span> <span class="k">type</span> <span class="n">subunit</span> <span class="o">=</span> <span class="k">struct</span> <span class="p">{</span>
	<span class="n">imports</span><span class="o">:</span> <span class="p">[]</span><span class="n">import</span><span class="p">,</span>
	<span class="n">decls</span><span class="o">:</span> <span class="p">[]</span><span class="n">decl</span><span class="p">,</span>
<span class="p">};</span>
</code></pre></div><p>Pretty straightforward! Having this as part of the standard library should make
it much easier for users to build language-aware tooling with the language
itself. We also plan on having our type checker in the stdlib as well. This is
something that I drew inspiration for from Golang — having a lot of their
toolchain components in the standard library makes it really easy to write
Go-aware tools.</p>
<p>So, there you have it: the next stage in the development of our language. I hope
you&rsquo;re looking forward to it!</p>


