# rss_item

guid: https://drewdevault.com/2019/06/15/Status-update-June-2019.html
title: Status update, June 2019
url: https://drewdevault.com/2019/06/15/Status-update-June-2019.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Summer is in full swing here in Philadelphia. Last night I got great views of
Jupiter and a nearly-full Moon, and my first Saturn observation of the year.  I
love astronomy on clear Friday nights, there&rsquo;s always plenty of people coming
through the city. And today, on a relaxing lazy Saturday, waiting for friends
for dinner later, I have the privilege of sharing another status report with
you.</p>
<p>First, I want to talk about some work I&rsquo;ve done with blogs lately. On the bottom
of this article you&rsquo;ll find a few blog posts from around the net. This is
populated with <a href="https://git.sr.ht/~sircmpwn/openring">openring</a>, a small Go tool
I made to fetch a few articles from a list of RSS feeds. A couple of other
people have added this to their own sites as well, and I hope to use this to
encourage the growth of a network of bloggers supporting each other without any
nonfree or centralized software. I&rsquo;ll write about this in its own article in
time. I&rsquo;ve also made an <a href="https://drewdevault.com/make-a-blog">open offer</a> to give $20 to anyone who
wants to make their own blog, and so far 5 new blogs have taken me up on the
offer. Maybe you&rsquo;ll be the next?</p>
<p>Other side projects have seen some nice progress this month, too.
<a href="https://git.sr.ht/~sircmpwn/wio">Wio</a> has received a few patches from Leon
Plickat improving the UX, and I understand more are on the way. I&rsquo;m also happy
to tell you that the RISC-V musl libc port I was working on is heading upstream
and slated for inclusion in the next release! Big thanks to everyone who helped
with that, and to Rich Felker for reviewing it and assembling the final patches.
I was also able to find some time this month to contribute to
<a href="https://git.sr.ht/~emersion/mrsh">mrsh</a>, adding support for job IDs, the
<code>wait</code>, <code>break</code>, and <code>continue</code> builtins, and a handful of other improvements.
I&rsquo;m really excited about mrsh, it&rsquo;s getting close to completion. My friend
Luminarys also finally released <a href="https://synapse-bt.org/">synapse 1.0</a>, a
bittorrent client that I had a <a href="https://github.com/Luminarys/synapse/commit/ac92bb424c3d7d99905f4c0988c924001b688080#diff-d981183863e690e9f0f2bd20145a7a16">hand in
designing</a>,
and <a href="https://github.com/ddevault/receptor">building</a>
<a href="https://broca.synapse-bt.org/">frontends</a> for. Congrats, Lumi! This one has
been a long time coming.</p>
<p>Alright, now for some updates on the larger, long-term projects. The initial
pre-release of aerc <a href="https://drewdevault.com/2019/06/03/Announcing-aerc-0.1.0.html">shipped</a> two weeks
ago! Even since then it&rsquo;s already attracted a flurry of patches from the
community. I&rsquo;m tremendously excited about this project, I think it has heaps of
potential and a community is quickly forming to help us live up to it. Since
0.1.0 it&rsquo;s already grown support for formatting the index list, swapped the
Python dependency for POSIX awk, grown temporary accounts and the ability to
view headers, and more. I&rsquo;ve already started planning 0.2.0 - check out <a href="https://todo.sr.ht/~sircmpwn/aerc2?search=label:%22blocker%22%20status%3Aopen">the
list of
blockers</a>
for a sneak peek.</p>
<p>The Godot+Wayland workstream has picked up again, and I&rsquo;ve secured some VR
hardware (an HTC Vive) and started working on <a href="https://github.com/swaywm/wlroots/issues/1723">planning the changes
necessary</a> for first-class VR
support on wlroots. In the future I also would like to contribute with the
OpenXR and OpenHMD efforts for bringing a full-stack free software solution for
VR. I also did a proof-of-concept 3D Wayland compositor that I intend to
translate to VR once I have the system up and running on Wayland:</p>
<p><video src="https://yukari.sr.ht/godot3d.webm" muted autoplay controls></video></p>
<p>In other respects, sway &amp; wlroots have been somewhat quiet. We&rsquo;ve been focusing
on small bug fixes and quality-of-life improvements, while some beefier changes
are stewing on the horizon. wlroots has seen some slow and steady progress on
refining its DRM implementation, improvements to which are going to lead to even
further improved performance and capability of the downstream compositors -
notably, direct scan-out has just been merged with the help of Scott Anderson
and Simon Ser.</p>
<p>In SourceHut news, the most exciting is perhaps that todo.sr.ht has grown an API
and webhooks! That makes it the last major sr.ht service to gain these features,
which unblocks a lot of other stuff in the pipeline. The biggest workstream
unblocked by this is dispatch.sr.ht, which has an design proposal for an
overhaul under discussion on the development list. This&rsquo;ll open the door for
features like building patches sent to mailing lists, linking tickets to
commits, and much more. I&rsquo;ve also deployed another compute server to pick up the
load as git.sr.ht grows to demand more resources, which frees up the box it used
to be on with more space for smaller services to get comfortable. I was also
happy to bring Ludovic Chabant, the driving force behind hg.sr.ht, with me to
attend a Mercurial conference in Paris, where I learned heaps about the
internals (and externals, to be honest) of Mercurial. Cool things are in store
here, too! Big thanks to the Mercurial maintainers for being so accommodating of
my ignorance, and for putting on a friendly and productive conference.</p>
<p>In the next month, I&rsquo;m moving aerc to the backburner and turning my focus back
to SourceHut &amp; wlroots VR. I&rsquo;m getting a consistent stream of great patches for
aerc to review, so I&rsquo;m happy to leave it in the community&rsquo;s hands for a while.
For SourceHut, the upcoming dispatch workstream is going to be a huge boon to
the community there. On its coattails will come more powerful data import &amp;
export tools, giving the users more ownership and autonomy over their data, and
perhaps following this will be some nice improvements to git.sr.ht. I&rsquo;m also
going to try and find time to invest more in Alpine Linux on RISC-V this month.</p>
<p>From the bottom of my heart, thank you again for lending your support. I&rsquo;ve
never been busier, happier, and more productive than I have been since working
on FOSS full-time. Let&rsquo;s keep building awesome software together.</p>
<small class="text-muted">
This work was possible thanks to users who support me financially. Please
consider <a href="https://drewdevault.com/donate">donating to my work</a> or <a
href="https://sourcehut.org">buying a sourcehut.org subscription</a>. Thank you!
</small>


guid: https://drewdevault.com/2019/07/01/Absence-of-features-in-IRC.html
title: Absence of certain features in IRC considered a feature
url: https://drewdevault.com/2019/07/01/Absence-of-features-in-IRC.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>The other day a friend of mine (an oper on Freenode) wanted to talk about IRC
compared to its peers, such as Matrix, Slack, Discord, etc. The ensuing
discussion deserves summarization here. In short: I&rsquo;m glad that IRC doesn&rsquo;t have
the features that are &ldquo;showstoppers&rdquo; for people choosing other platforms, and
I&rsquo;m worried that attempts to bring these showstopping &ldquo;features&rdquo; to IRC will
worsen the platform for the people who use it now.</p>
<p>On IRC, features like embedded images, a nice UX for messages longer than a few
lines (e.g. pasted code), threaded messages, etc; are absent. Some sort of
&ldquo;graceful degradation&rdquo; to support mixed channels with clients which support
these features and clients which don&rsquo;t may be possible, but it still <em>degrades</em>
the experience for many people. By instead making everyone work within the
limitations of IRC, we establish a shared baseline, and expressing yourself
within these limitations is not only possible but makes a better experience for
everyone.</p>
<p>Remember that [not everyone is like you][old hardware]. I regularly chat with
people on ancient hardware that slows to a crawl when a web browser is
running<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>, or people working from a niche operating system for which porting a
graphical client is a herculean task, or people with accessibility concerns for
whom the &ldquo;one line of text per statement&rdquo; fits nicely into their TTS<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> system
and screenreading Slack is a nightmare.</p>
<p>Let&rsquo;s consider what happens when these features are added but non-uniformly
available. Let&rsquo;s use rich text as an example and examine the fallback
implementation. Which of these is better?</p>
<p><span>(A) &lt;user&gt; check out [this website](<a href="https://example.org"><a href="https://example.org">https://example.org</a></a>)</span></p>
<p><span>(B) &lt;user&gt; check out this website: <a href="https://example.org"><a href="https://example.org">https://example.org</a></a></span></p>
<p>Example B is what people naturally do when rich text is unavailable, and most
clients will recognize it as a link and make it clickable anyway. But many
clients cannot and will not display example A as a link, which makes it harder
to read. Example A also makes phishing <em>much</em> easier.</p>
<p>Here&rsquo;s another example: how about a nice UI for long messages, such as pasted
code snippets? Let&rsquo;s examine how three different clients would implement this:
(1) a GUI client, (2) a TUI<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> client, and (3) a client which refuses to
implement it or is unmaintained<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>.</p>
<p>The first case is the happy path, we probably get a little scrollbox that the
user can interact with their mouse. Let&rsquo;s say <a href="https://weechat.org/">Weechat</a>
takes up option 2, but how do they do that? Some terminal emulators have mouse
support, so they could have a similar box, but since Weechat is primarily
keyboard-driven (and some terminal emulators do not support mice!), a
keyboard-based alternative will be necessary. Now we have to have some kind of
command or keybinding for scrolling through the message, and picking which of
the last few long messages we want to scroll through. This will have to be
separate from scrolling through the backlog normally, of course. The third
option is the worst: they just see a hundred lines pasted into their backlog,
which is already highly scorned behavior on most IRC channels. Only the GUI
users come away from this happy, and on IRC they&rsquo;re in the minority.</p>
<p>Some IRC clients (Matrix) have this feature today, but most Matrix users don&rsquo;t
realize what a nuisance they&rsquo;re being on the chat. Here&rsquo;s what they see:</p>
<p><img src="https://sr.ht/VOeY.png" alt=""></p>
<p>And here&rsquo;s what I see:</p>
<p><img src="https://sr.ht/HZ7Z.png" alt=""></p>
<p>Conservative improvements built on top of existing IRC norms, such as <a href="https://thelounge.chat/">The
Lounge</a>, are much better. Most people post images on
IRC as URLs, which clients can do a quick HEAD request against and embed if the
mimetype is appropriate:</p>
<p><img src="https://sr.ht/9RsR.png" alt=""></p>
<p>For most of these features, I think that people who have and think they need
them are in fact unhappier for having them. What are some of the most common
complaints from Slack users et al? &ldquo;It&rsquo;s distracting.&rdquo; &ldquo;It&rsquo;s hard to keep up
with what people said while I was away.&rdquo; &ldquo;Threads get too long and hard to
understand.&rdquo; Does any of this sound familiar? Most of these problems are caused
by or exacerbated by features which are missing from IRC.  It&rsquo;s distracting
because your colleagues are posting gifs all day. It&rsquo;s hard to keep up with
because the infinite backlog encourages a culture of catching up rather than
setting the expectation that conversations are ephemeral<sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup>. Long conversations
shouldn&rsquo;t be organized into threads, but moved into email or another medium more
suitable for that purpose.</p>
<p>None of this even considers what <em>is</em> good about IRC. It&rsquo;s a series of
decentralized networks built on the shoulders of volunteers. It&rsquo;s venerable and
well-supported with hundreds of client and server implementations. You can
connect to IRC manually using telnet and have a pretty good user experience!
Accordingly, <a href="https://drewdevault.com/2018/03/10/How-to-write-an-IRC-bot.html">a working IRC bot can be written in about 2
minutes</a>. No one is trying to monetize you on IRC.
It&rsquo;s free, in both meanings, and nothing which has come since has presented a
compelling alternative. I&rsquo;ve used IRC all day, every day for over ten years, and
that&rsquo;s not even half of IRC&rsquo;s lifetime. It&rsquo;s outlived everything else by years
and years, and it&rsquo;s not going anywhere soon.</p>
<p>In summary, I like IRC the way it is. It has problems which we ought to address,
but many people focus on the wrong problems. The culture that it fosters is good
and worth preserving, even at the expense of the features users of other
platforms demand - or those users themselves.</p>
<hr>
<p>P.S. A friend pointed out that the migration of non-hackers away from IRC is
like a reverse <a href="https://en.wikipedia.org/wiki/Eternal_September">Eternal September</a>, which sounds <em>great</em> 😉</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Often, I <em>am</em> this person. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Text to speech.
[old hardware]: <a href="https://drewdevault.com/2019/01/23/Why-I-use-old-hardware.html">https://drewdevault.com/2019/01/23/Why-I-use-old-hardware.html</a> <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Text user interface <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>IRC is over 30 years old and has barely changed since - so using unmaintained or barely-maintained clients is not entirely uncommon nor wrong. <a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>Many people have bouncers which allow them to catch up the last few lines, and keep logs which they can reference later if necessary. This is nice to have but adds enough friction to keep the expectation that discussions are ephemeral, which has a positive effect on IRC culture. <a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/07/08/Announcing-annotations-for-sourcehut.html
title: Announcing code annotations for SourceHut
url: https://drewdevault.com/2019/07/08/Announcing-annotations-for-sourcehut.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Today I&rsquo;m happy to announce that code annotations are now available for
<a href="https://sourcehut.org">SourceHut</a>! <img style="display: inline; height: 1.2rem"
src="https://drewdevault.com/img/party.png" /> These allow you to decorate your code with arbitrary
links and markdown. The end result looks something like this:</p>
<p><strong>NOTICE</strong>: Annotations were ultimately removed from sourcehut.</p>
<p><img src="https://sr.ht/w767.png" alt=""></p>
<small class="text-muted">
  <a href="https://sourcehut.org">SourceHut</a> is the "hacker's forge", a
  100% open-source platform for hosting Git &amp; Mercurial repos, bug trackers,
  mailing lists, continuous integration, and more. No JavaScript required!
</small>
<p>The annotations shown here are sourced from a JSON file which you can generate
and upload during your CI process. It looks something like this:</p>
<div class="highlight"><pre class="chroma"><code class="language-json" data-lang="json"><span class="p">{</span>
  <span class="nt">&#34;98bc0394a2f15171fb113acb5a9286a7454f22e7&#34;</span><span class="p">:</span> <span class="p">[</span>
    <span class="p">{</span>
      <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;markdown&#34;</span><span class="p">,</span>
      <span class="nt">&#34;lineno&#34;</span><span class="p">:</span> <span class="mi">33</span><span class="p">,</span>
      <span class="nt">&#34;title&#34;</span><span class="p">:</span> <span class="s2">&#34;1 reference&#34;</span><span class="p">,</span>
      <span class="nt">&#34;content&#34;</span><span class="p">:</span> <span class="s2">&#34;- [../main.c:123](https://example.org)&#34;</span>
    <span class="p">},</span>
    <span class="p">{</span>
      <span class="nt">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;link&#34;</span><span class="p">,</span>
      <span class="nt">&#34;lineno&#34;</span><span class="p">:</span> <span class="mi">38</span><span class="p">,</span>
      <span class="nt">&#34;colno&#34;</span><span class="p">:</span> <span class="mi">7</span><span class="p">,</span>
      <span class="nt">&#34;len&#34;</span><span class="p">:</span> <span class="mi">15</span><span class="p">,</span>
      <span class="nt">&#34;to&#34;</span><span class="p">:</span> <span class="s2">&#34;#L6&#34;</span>
    <span class="p">},</span>
    <span class="err">...</span>
</code></pre></div><p>You can probably infer from this that annotations are very powerful. Not only
can you annotate your code&rsquo;s semantic elements to your heart&rsquo;s content, but you
can also do exotic things we haven&rsquo;t thought of yet, for every programming
language you can find a parser for.</p>
<p>I&rsquo;ll be going into some detail on the thought process that went into this
feature&rsquo;s design and implementation in a moment, but if you&rsquo;re just excited and
want to try it out, here are a few interesting annotated repos to browse:</p>
<ul>
<li><a href="https://git.sr.ht/~sircmpwn/scdoc/tree/master/src/main.c">~sircmpwn/scdoc</a>: man page generator (C)</li>
<li><a href="https://git.sr.ht/~sircmpwn/aerc/tree/master/widgets/msgviewer.go">~sircmpwn/aerc</a>: TUI email client (Go)</li>
<li><a href="https://git.sr.ht/~mcf/cproc/tree/master/scan.c">~mcf/cproc</a>: C compiler (C)</li>
</ul>
<p>And here are the docs for generating your own: <a href="https://man.sr.ht/git.sr.ht/annotations.md">annotations on
git.sr.ht</a>. Currently annotators are
available for C and Go, and I intend to write another for Python. For the rest,
I&rsquo;ll be relying on the community to put together annotators for their favorite
programming languages, and to help me expand on the ones I&rsquo;ve built.</p>
<h2 id="design">Design</h2>
<p>A lot of design thought went into this feature, but I knew one thing from the
outset: I wanted to make a generic system that users could use to annotate their
source code in any manner they chose. My friend Andrew Kelley (of
<a href="https://ziglang.org/">Zig</a> fame) once expressed to me his frustration with
GitHub&rsquo;s refusal to implement syntax highlighting for &ldquo;small&rdquo; languages, citing
a shortage of manpower. It&rsquo;s for this reason that it&rsquo;s important to me that
SourceHut&rsquo;s open-source platform allows users large and small to volunteer to
build the perfect integration for their needs - I don&rsquo;t scale alone<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</p>
<p>To get a head start for the most common use-cases - scanning source files and
linking references and definitions together - the best approach was unclear. I
spent a lot of time studying <a href="http://ctags.sourceforge.net/">ctags</a>, for
example, which supports a huge set of programming languages, but unfortunately
only finds definitions. I thought about combining this with another approach for
finding references, but the only generic library with lots of parsers I&rsquo;m aware
of is <a href="http://pygments.org/">Pygments</a>, and I didn&rsquo;t necessarily want to bring
Python into every user&rsquo;s CI process if they weren&rsquo;t already using it. That
approach would also make it more difficult to customize the annotations for each
language. Other options I considered were
<a href="http://cscope.sourceforge.net/">cscope</a> and
<a href="https://www.gnu.org/software/global/">gtags</a>, but the former doesn&rsquo;t have many
programming languages supported (making the tradeoff questionable), and the
latter just uses Pygments anyway.</p>
<p>So I decided: I&rsquo;m going to write my own annotators for each language. Or at
least the languages I use the most:</p>
<ul>
<li>C, because I like it but also because
<a href="https://git.sr.ht/~sircmpwn/scdoc">scdoc</a> is the demo repo shown on the
<a href="https://sourcehut.org">SourceHut marketing page</a>.</li>
<li>Python, because SourceHut is largely written in Python and using it to browse
itself would be cool.</li>
<li>Go, because parts of SourceHut are written in it but also because I use it a
lot for <a href="https://git.sr.ht/~sircmpwn/aerc">my own projects</a>. I also knew that
Go had at least <em>some</em> first-class support for working with its AST - and boy
was I in for a surprise.</li>
</ul>
<p>With these initial languages decided, let&rsquo;s turn to the implementations.</p>
<h2 id="annotating-c-code">Annotating C code</h2>
<p>I began with the C annotator, because I knew it would be the most difficult.
There does not exist any widely available standalone C parsing library to
provide C programs with access to an AST. There&rsquo;s LLVM, but I have a deeply held
belief that programming language compiler and introspection tooling should be
implemented in the language itself. So, I set about to write a C parser from
scratch.</p>
<p>Or, almost from scratch. There exist two standard POSIX tools for writing
compilers with: <a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/lex.html">lex</a> and <a href="http://pubs.opengroup.org/onlinepubs/9699919799/utilities/yacc.html">yacc</a>, which are respectively a lexer
generator and a compiler compiler. Additionally, there are <a href="http://www.quut.com/c/ANSI-C-grammar-y.html">pre-fab lex and
yacc files</a> which <em>mostly</em>
implement the C11 standard grammar. However, C is <a href="https://eli.thegreenplace.net/2007/11/24/the-context-sensitivity-of-cs-grammar/">not a context-free
language</a>, so additional work was necessary to track typedefs and use
them to change future tokens emitted by the scanner. A little more work was also
necessary for keeping track of line and column numbers in the lexer. Overall,
however, this was relatively easy, and in less than a day&rsquo;s work I had a fully
functional C11 parser.</p>
<p>However, my celebration was short-lived as I started to feed my parser C
programs from the wild. The GNU C Compiler, GCC, implements many C extensions,
and their use, while inadvisable, is extremely common. Not least of the
offenders is glibc, and thus running my parser on any system with glibc headers
installed would likely immediately run into syntax errors.  GCC&rsquo;s extensions are
not documented in the form of an addendum to the C specification, but rather as
end-user documentation and a 15 million lines-of-code compiler for you to
reverse engineer. It took me almost a week of frustration to get a parser which
worked passably on a large subset of the C programs found in the wild, and I
imagine I&rsquo;ll be dealing with GNU problems for years to come. Please don&rsquo;t use C
extensions, folks.</p>
<p>In any case, the result now works fairly well for a lot of programs, and I have
plans on expanding it to integrate more nicely with build systems like meson.
Check out the code here: <a href="https://git.sr.ht/~sircmpwn/annotatec">annotatec</a>. The
features of the C annotator include:</p>
<ul>
<li>Annotating function definitions with a list of files/linenos which call them</li>
<li>Linking function calls to the definition of that function</li>
</ul>
<p>In the future I intend to add support for linking to external symbols as well -
for example, linking to the POSIX spec for functions specified by POSIX, or to
the Linux man pages for Linux calls. It would also be pretty cool to support
linking between related projects, so that wlroots calls in sway can be linked to
their declarations in the wlroots repo.</p>
<h2 id="annotating-go-code">Annotating Go code</h2>
<p>The Go annotator was far easier. I started over my morning cup of coffee today
and I was finished with the basics by lunch. Go has a bunch of support in the
standard library for parsing and analyzing Go programs - I was very impressed:</p>
<ul>
<li><a href="https://golang.org/pkg/go/ast/">go/ast</a></li>
<li><a href="https://golang.org/pkg/go/scanner/">go/scanner</a></li>
<li><a href="https://golang.org/pkg/go/token/">go/token</a></li>
<li><a href="https://golang.org/pkg/go/types/">go/types</a></li>
</ul>
<p>To support Go 1.12&rsquo;s go modules, the experimental (but good enough)
<a href="https://godoc.org/golang.org/x/tools/go/packages">packages</a> module is available
as well. All of this is nicely summarized by a lovely document in the <a href="https://github.com/golang/example/tree/master/gotypes">golang
examples repository</a>. The
type checker is also available as a library, something which is less common even
among languages with parsers-as-libraries, and allows for many features which
would be very difficult without it. Nice work, Go!</p>
<p>The <a href="https://git.sr.ht/~sircmpwn/annotatego">resulting annotator</a> clocks in at
just over 250 lines of code - compare that to the C annotator&rsquo;s ~1,300 lines of
C, lex, and yacc source code. The Go annotator is more featureful, too, it can:</p>
<ul>
<li>Link function calls to their definitions, and in reverse</li>
<li>Link method calls to their definitions, and in reverse</li>
<li>Link variables to their definitions, even in other files</li>
<li>Link to godoc for symbols defined in external packages</li>
</ul>
<p>I expect a lot more to be possible in the future. It might get noisy if you turn
everything on, so each annotation type is gated behind a command line flag.</p>
<h2 id="displaying-annotations">Displaying annotations</h2>
<p>Displaying these annotations required a bit more effort than I would have liked,
but the end result is fairly clean and reusable. Since SourceHut uses Pygments
for syntax highlighting, I ended up writing a <a href="http://pygments.org/docs/formatterdevelopment/">custom
Formatter</a> based on the existing
Pygments HtmlFormatter. The result is the [AnnotationFormatter]<a href="https://git.sr.ht/~sircmpwn/git.sr.ht/tree/master/gitsrht/annotations.py">git.sr.ht
formatter</a>, which splices annotations into the highlighted code. One downside of
this approach is that it works at the token level - a more sophisticated
implementation will be necessary for annotations that span more than a single
token. Annotations are fairly expensive to render, so the rendered HTML is
stowed in Redis.</p>
<h2 id="the-future">The future?</h2>
<p>I intend to write a Python annotator soon, and I&rsquo;ll be relying on the community
to build more. If you&rsquo;re looking for a fun weekend hack and a chance to learn
more about your favorite programming language, this&rsquo;d be a great project. The
format for annotations on SourceHut is also pretty generalizable, so I encourage
other code forges to reuse it so that our annotators are useful on every code
hosting platform.</p>
<p>builds.sr.ht will also soon grow first-class support for making these annotators
available to your build process, as well as for making an OAuth token available
(ideally with a limited set of permissions) to your build environment. Rigging
up an annotator is a bit involved today (<a href="https://man.sr.ht/git.sr.ht/annotations.md">though the docs
help</a>), and streamlining that
process will be pretty helpful. Additionally, this feature is only available for
git.sr.ht, though it should generalize to hg.sr.ht fairly easily and I hope
we&rsquo;ll see it available there soon.</p>
<p>I&rsquo;m also looking forward to seeing more novel use-cases for annotation. Can we
indicate code coverage by coloring a gutter alongside each line of code? Can we
link references to ticket numbers in the comments to your bug tracker? If you
have any cool ideas, I&rsquo;m all ears. Here&rsquo;s that list of cool annotated repos to
browse again, if you made it this far and want to check them out:</p>
<ul>
<li><a href="https://git.sr.ht/~sircmpwn/scdoc/tree/master/src/main.c">~sircmpwn/scdoc</a>: man page generator (C)</li>
<li><a href="https://git.sr.ht/~sircmpwn/aerc/tree/master/widgets/msgviewer.go">~sircmpwn/aerc</a>: TUI email client (Go)</li>
<li><a href="https://git.sr.ht/~mcf/cproc/tree/master/scan.c">~mcf/cproc</a>: C compiler (C)</li>
</ul>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>For the syntax highlighting problem, by the way, this is accomplished by using Pygments. Improvements to Pygments reach not only SourceHut, but a large community of projects, making the software ecosystem better for everyone. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/07/15/Status-update-July-2019.html
title: Status update, July 2019
url: https://drewdevault.com/2019/07/15/Status-update-July-2019.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Today I received the keys to my new apartment, which by way of not being
directly in the middle of the city<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> saves me a decent chunk of money - and
allows me to proudly announce that I have officially broken even on doing free
software full time! I owe a great deal of thanks to all of you who have <a href="https://drewdevault.com/donate">donated
to support my work</a> or purchased a paid
<a href="https://sourcehut.org">SourceHut</a> account. I&rsquo;ve dreamed of sustainably working
on free software for a long, long time, and I&rsquo;m very grateful for all of your
support in helping realize that dream. Now let me share with you what your money
has bought over the past month!</p>
<p>First, my <a href="https://drewdevault.com/make-a-blog">make a blog</a> offer has closed
for the time being, and the world is now 13 blogs richer for it. Be sure to
check them out! I have also started a mailing list for tech writers: the <a href="https://lists.sr.ht/~sircmpwn/free-writers-club">free
writers club</a>, which I
encourage anyone using free software to blog about technology to join for
editorial advice, software recommendations, and periodic reminders to keep
writing. The offer to get paid for your own new blog will reopen in the future,
keep an eye out!</p>
<p>As far as projects are concerned, lots of good stuff this month. aerc has been
making excellent progress. We just pulled in the first batch of patches adding
maildir support, and will soon have sendmail and mbox support as well. We&rsquo;ve
also begun on mouse support, and you can now click to switch between tabs. The
initial patches for tab completion have also been added. Additional changes
include an :unsubscribe command to unsubscribe from marketing emails and mailing
lists, basic search functionality, OAuth IMAP authentication, changing config
options at runtime, and DNS lookups to complete your settings in the new account
wizard more quickly. Building more upon these features, and a handler for mailto
links, are the main blockers for aerc 0.2.0.</p>
<p>In Wayland news, VR work continues. I&rsquo;ve taken on the goal of implementing DRM
leasing for Wayland, which will allow VR applications to take exclusive control
over the headset&rsquo;s graphical resources from Wayland compositor. A similar
technology exists for X11, and I&rsquo;ve written a Wayland protocol for the same
purpose on Wayland. I&rsquo;ve also written a Vulkan extension to utilize this
protocol in Vulkan&rsquo;s WSI layer. I&rsquo;ve written implementations of these for
wlroots, sway, mesa, and the radv (AMD) Vulkan driver. The result: a working VR
demo on Sway (audio warning):</p>
<p><video src="https://yukari.sr.ht/xrgears.webm" controls></video></p>
<p>There&rsquo;s still some details to sort out on the standardization of these
extensions, which are under discussion now. In the coming weeks I hope to have
an implementation for Xwayland (which will get working games based on Steam&rsquo;s
OpenVR runtime), and get a proof-of-concept of a VR-driven Wayland compositor
based on the demo shown in the previous status update. Exciting stuff!</p>
<p>I&rsquo;ve also had time to write a few more chapters for my Wayland book, which I&rsquo;ll
be speeding up my work on. I&rsquo;ll soon be leaving for an extended trip to Japan,
and on these grueling flights I&rsquo;ll have plenty of time to work on it. In
additional Wayland news, we&rsquo;ve been chugging along with small bugfixes and
improvements to wlroots and sway, and implementing more plumbing work to round
out our implementation of everything. Our work continues to evolve into the most
robust Wayland implementation available today, and I can only see it getting
stronger.</p>
<p>On SourceHut, I have plenty of developments to share, but will leave the details
for the <a href="https://lists.sr.ht/~sircmpwn/sr.ht-announce">sr.ht-announce mailing
list</a>. The most exciting news is
that <a href="https://alpinelinux.org">Alpine Linux</a>, my favorite Linux distribution,
has completed their mailing list infrastructure migration to <a href="https://lists.alpinelinux.org">their own
lists.sr.ht instance</a>! I&rsquo;ve also been hard at
work expanding lists.sr.ht&rsquo;s capabilities to this end. The other big piece of
news was announced on my blog last week: <a href="https://drewdevault.com/2019/07/08/Announcing-annotations-for-sourcehut.html">code
annotations</a>.
All of our services have also been upgraded to Alpine 3.10, and the Alpine
mirror reorganized a bit to make future upgrades smooth.  There&rsquo;s all sorts of
other goodies to share, but I&rsquo;ll leave the rest for the sr.ht-announce post
later today.</p>
<p>All sorts of other little things have gotten done, like sending patches upstream
for kmscube fixes, minor improvements to scdoc, writing a new build system for
mrsh, improvements to openring&hellip; but I&rsquo;m running out of patience and I imagine
you are, too. Again I&rsquo;m eternally grateful for your support: thank you. I&rsquo;ll see
you again for the next status update, same time next month!</p>
<small class="text-muted">
This work was possible thanks to users who support me financially. Please
consider <a href="https://drewdevault.com/donate">donating to my work</a> or <a
href="https://sourcehut.org">buying a sourcehut.org subscription</a>. Thank you!
</small>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>I can see city hall out the window of my old apartment <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/07/29/FOSS-contributor-tracks.html
title: FOSS contributor tracks
url: https://drewdevault.com/2019/07/29/FOSS-contributor-tracks.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Just like many companies have different advancement tracks for their employees
(for example, a management track and an engineering track), similar concepts
exist in free software projects. One of the roles of a maintainer is to help
contributors develop into the roles which best suit them. I&rsquo;d like to explain
what this means to me in my role as a maintainer of several projects, though I
should mention upfront that I&rsquo;m just some guy and, while I can explain what has
and hasn&rsquo;t worked for me, I can&rsquo;t claim to have all of the answers. People are
hard.</p>
<p>There are lots of different tasks which need doing on a project. A few which
come up fairly often include:</p>
<ul>
<li>End-user support</li>
<li>Graphic design</li>
<li>Marketing</li>
<li>Release planning</li>
<li>Reviewing code</li>
<li>Translations</li>
<li>Triaging tickets</li>
<li>Writing code</li>
<li>Writing documentation</li>
</ul>
<p>Within these tasks there&rsquo;s room still for more specialization - different
modules have different maintainers, each contributor&rsquo;s skills may be applicable
to different parts of the codebase, some people may like blogging about the
project where others like representing the project at conferences, and so on. To
me, one of my most important jobs is to figure out these relationships between
tasks and people.</p>
<p>There are several factors that go into this. Keeping an eye on code reviews,
social channels, etc, gives you a good pulse on what people are good at now.
Talking with them directly and discussing possible future work is a good way to
understand what they want to work on. I also often consider what they could be
good at but don&rsquo;t have exposure to yet, and encourage them to take on more of
these tasks. The most common case where I try to get people to branch out is
code review - once they&rsquo;ve contributed to a module they&rsquo;re put on the shortlist
for reviewers for future changes to nearby code. Don&rsquo;t be afraid to take risks -
a few bugs is a small price to pay for an experienced contributor.</p>
<p>This also touches on another key part of this work - fostering collaboration.
For example, if someone is taking on a cross-cutting task, I&rsquo;ll give them the
names of experts on all of the affected modules so they can ask questions and
seek buy-in on their approach. Many developers aren&rsquo;t interested in end-user
support, so getting people who are interested in this to bubble up technical
issues when they&rsquo;re found is helpful as well.</p>
<p>The final step is to gradually work your way out of the machine. Just like you
onboard someone with feature development or code review, you can onboard people
with maintainer tasks. If someone asks you to connect them to experts on some
part of the code, defer to a senior contributor - who has likely asked you the
same question at some point. Ask a contributor to go over the shortlog and
prepare a draft for the next release notes. Pull a trusted contributor aside and
ask them what they think needs to be improved in the project - then ask them to
make those improvements, and equip them with any tools they need to accomplish
it.</p>
<p>One role I tend to reserve for myself is conflict prevention and moderation. I
keep a light watch on collaboration channels and periodically sync with major
contributors, keeping a pulse for the flow of information through the project.
When arguments start brewing or things start getting emotional, I try to notice
early and smooth things over before they get heated. At an impasse, I&rsquo;ll make a
final judgement call on a feature, design decision, or whatever else. By making
the decision, I aim to make it neither party&rsquo;s fault that someone didn&rsquo;t get
their way. Instead, I point any blame at myself, and rely on the mutual trust
between myself and the contributors to see the decision through amicably. When
this works correctly, it can help preserve a good relationship between each
party.</p>
<p>If you&rsquo;re lucky, the end result is a project which can grow arbitrarily large,
with contributors bringing a variety of skills to support each other at every
level and enjoy the work they&rsquo;re doing. The bus factor is low and everyone
maintains a healthy and productive relationship with the project - yourself
included.</p>


guid: https://drewdevault.com/2019/08/09/DRM-leasing-and-VR-for-Wayland.html
title: DRM leasing: VR for Wayland
url: https://drewdevault.com/2019/08/09/DRM-leasing-and-VR-for-Wayland.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>As those who read my <a href="https://drewdevault.com/2019/07/15/Status-update-July-2019.html">status updates</a>
have been aware, recently I&rsquo;ve been working on bringing VR to Wayland (and vice
versa). The deepest and most technical part of this work is <em>DRM leasing</em>
(Direct Rendering Manager, <em>not</em> Digital Restrictions Management), and I think
it&rsquo;d be good to write in detail about what&rsquo;s involved in this part of the
effort. This work has been sponsored by <a href="https://status.im/">Status.im</a>, as part
of an effort to build a comprehensive Wayland-driven VR workspace. When we got
started, most of the plumbing was missing for VR headsets to be useful on
Wayland, so this has been my focus for a while. The result of this work is
summed up in this crappy handheld video:</p>
<video src="https://yukari.sr.ht/steamvr.webm" controls>
  Your web browser does not support the webm video codec. Please consider using
  web browsers that support free and open standards.
</video>
<p>Keith Packard, a long time Linux graphics developer, <a href="https://keithp.com/blogs/DRM-lease/">wrote several blog posts
documenting his work implementing this feature for
X11</a>. My journey was somewhat similar,
though thanks to his work I was able to save a lot of time. The rub of this idea
is that the Wayland compositor, the DRM (Direct Rendering Manager) master, can
&ldquo;lease&rdquo; some of its resources to a client so they can drive your display
directly. DRM is the kernel subsystem we use for enumerating and setting modes,
allocating pixel buffers, and presenting them in sync with the display&rsquo;s refresh
rate. For a number of reasons, minimizing latency being an important one, VR
applications prefer to do these tasks directly rather than be routed through the
display server like most applications are. The main tasks for implementing this
for Wayland were:</p>
<ol>
<li>Draft a <a href="https://lists.freedesktop.org/archives/wayland-devel/2019-July/040768.html">protocol extension</a> for issuing DRM leases</li>
<li>Write implementations for <a href="https://github.com/swaywm/wlroots/pull/1730">wlroots</a> and <a href="https://github.com/swaywm/sway/pull/4289">sway</a></li>
<li>Get a <a href="https://git.sr.ht/~sircmpwn/kmscube">simple test client</a> working</li>
<li>Draft a Vulkan extension for leasing via Wayland</li>
<li>Write an implementation for <a href="https://gitlab.freedesktop.org/mesa/mesa/merge_requests/1509">Mesa&rsquo;s Vulkan WSI implementation</a></li>
<li>Get a more complex <a href="https://git.sr.ht/~sircmpwn/xrgears">Vulkan test client</a> working</li>
<li>Add support to <a href="https://gitlab.freedesktop.org/xorg/xserver/merge_requests/248">Xwayland</a></li>
</ol>
<p>Let&rsquo;s break down exactly what was necessary for each of these steps.</p>
<h2 id="wayland-protocol-extension">Wayland protocol extension</h2>
<p>Writing a protocol extension was the first order of business. There was an
<a href="https://lists.freedesktop.org/archives/wayland-devel/2018-January/036652.html">earlier attempt</a> which petered off in January. I started
with this, by cleaning it up based on my prior experience writing protocols,
normalizing much of the terminology and style, and cleaning up the state
management. After some initial rounds of review, there were some questions to
answer. The most important ones were:</p>
<ul>
<li>How do we identify the display? Should we send the EDID, which may be
bigger than the maximum size of a Wayland message?</li>
<li>Are there security concerns? Could malicious clients read from framebuffers
they weren&rsquo;t given a lease for?</li>
</ul>
<p>The EDID I ended up sending in a side channel (file descriptor to shared
memory), and the latter was proven to be a non-issue by writing a malicious
client and demonstrating that the kernel rejects its attempts to do evil.</p>
<div class="highlight"><pre class="chroma"><code class="language-xml" data-lang="xml"><span class="nt">&lt;event</span> <span class="na">name=</span><span class="s">&#34;edid&#34;</span><span class="nt">&gt;</span>
  <span class="nt">&lt;description</span> <span class="na">summary=</span><span class="s">&#34;edid&#34;</span><span class="nt">&gt;</span>
    The compositor may send this event once the connector is created to
    provide a file descriptor which may be memory-mapped to read the
    connector&#39;s EDID, to assist in selecting the correct connectors
    for lease. The fd must be mapped with MAP_PRIVATE by the recipient.

    Note that not all displays have an EDID, and this event will not be
    sent in such cases.
  <span class="nt">&lt;/description&gt;</span>
  <span class="nt">&lt;arg</span> <span class="na">name=</span><span class="s">&#34;edid&#34;</span> <span class="na">type=</span><span class="s">&#34;fd&#34;</span> <span class="na">summary=</span><span class="s">&#34;EDID file descriptor&#34;</span> <span class="nt">/&gt;</span>
  <span class="nt">&lt;arg</span> <span class="na">name=</span><span class="s">&#34;size&#34;</span> <span class="na">type=</span><span class="s">&#34;uint&#34;</span> <span class="na">summary=</span><span class="s">&#34;EDID size, in bytes&#34;</span><span class="nt">/&gt;</span>
<span class="nt">&lt;/event&gt;</span> 
</code></pre></div><p>A few more changes would happen to this protocol in the following weeks, but
this was good enough to move on to&hellip;</p>
<h2 id="wlroots--sway-implementation">wlroots &amp; sway implementation</h2>
<p>After a chat with Scott Anderson (the maintainer of DRM support in wlroots) and
thanks to his timely refactoring efforts, the stage was well set for introducing
this feature to wlroots. I had a good idea of how it would take shape. <a href="https://github.com/swaywm/wlroots/pull/1730/files#diff-77b17feac8a8af251811a20e5b9bbdd1">Half of
the work</a> - the state machine which maintains the server-side
view of the protocol - is well trodden ground and was fairly easy to put
together. Despite being a well-understood problem in the wlroots codebase, these
state machines are always a bit tedious to implement correctly, and I was still
to flushing out bugs well into the remainder of this workstream.</p>
<p>The other half of this work was in <a href="https://github.com/swaywm/wlroots/pull/1730/files#diff-8b05a774317ee8e87d51422170f82d4b">the DRM subsystem</a>. We
decided that we&rsquo;d have leased connectors appear &ldquo;destroyed&rdquo; to the compositor,
and thus the compositor would have an opportunity to clean it up and stop using
them, similar to the behavior of when an output is hotplugged. Further changes
were necessary to have the DRM internals elegantly carry around some state for
the leased connector and avoid using the connector itself, as well as dealing
with the termination of the lease (either by the client or by the compositor).
With all of this in place, it&rsquo;s a <a href="https://github.com/swaywm/wlroots/pull/1730/files#diff-8b05a774317ee8e87d51422170f82d4bR1601">simple matter</a> to enumerate
the DRM object IDs for all of the resources we intend to lease and issue the
lease itself.</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">int</span> <span class="n">nobjects</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">nconns</span><span class="p">;</span> <span class="o">++</span><span class="n">i</span><span class="p">)</span> <span class="p">{</span>
	<span class="k">struct</span> <span class="n">wlr_drm_connector</span> <span class="o">*</span><span class="n">conn</span> <span class="o">=</span> <span class="n">conns</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
	<span class="n">assert</span><span class="p">(</span><span class="n">conn</span><span class="o">-&gt;</span><span class="n">state</span> <span class="o">!=</span> <span class="n">WLR_DRM_CONN_LEASED</span><span class="p">);</span>
	<span class="n">nobjects</span> <span class="o">+=</span> <span class="mi">0</span>
		<span class="o">+</span> <span class="mi">1</span> <span class="cm">/* connector */</span>
		<span class="o">+</span> <span class="mi">1</span> <span class="cm">/* crtc */</span>
		<span class="o">+</span> <span class="mi">1</span> <span class="cm">/* primary plane */</span>
		<span class="o">+</span> <span class="p">(</span><span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">cursor</span> <span class="o">!=</span> <span class="nb">NULL</span> <span class="o">?</span> <span class="mi">1</span> <span class="o">:</span> <span class="mi">0</span><span class="p">)</span> <span class="cm">/* cursor plane */</span>
		<span class="o">+</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">num_overlays</span><span class="p">;</span> <span class="cm">/* overlay planes */</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">nobjects</span> <span class="o">&lt;=</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
	<span class="n">wlr_log</span><span class="p">(</span><span class="n">WLR_ERROR</span><span class="p">,</span> <span class="s">&#34;Attempted DRM lease with &lt;= 0 objects&#34;</span><span class="p">);</span>
	<span class="k">return</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">wlr_log</span><span class="p">(</span><span class="n">WLR_DEBUG</span><span class="p">,</span> <span class="s">&#34;Issuing DRM lease with the %d objects:&#34;</span><span class="p">,</span> <span class="n">nobjects</span><span class="p">);</span>
<span class="n">uint32_t</span> <span class="n">objects</span><span class="p">[</span><span class="n">nobjects</span> <span class="o">+</span> <span class="mi">1</span><span class="p">];</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">j</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">nconns</span><span class="p">;</span> <span class="o">++</span><span class="n">i</span><span class="p">)</span> <span class="p">{</span>
	<span class="k">struct</span> <span class="n">wlr_drm_connector</span> <span class="o">*</span><span class="n">conn</span> <span class="o">=</span> <span class="n">conns</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
	<span class="n">objects</span><span class="p">[</span><span class="n">j</span><span class="o">++</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">;</span>
	<span class="n">objects</span><span class="p">[</span><span class="n">j</span><span class="o">++</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">;</span>
	<span class="n">objects</span><span class="p">[</span><span class="n">j</span><span class="o">++</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">primary</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">;</span>
	<span class="n">wlr_log</span><span class="p">(</span><span class="n">WLR_DEBUG</span><span class="p">,</span> <span class="s">&#34;connector: %d crtc: %d primary plane: %d&#34;</span><span class="p">,</span>
			<span class="n">conn</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">,</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">,</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">primary</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">);</span>
	<span class="k">if</span> <span class="p">(</span><span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">cursor</span><span class="p">)</span> <span class="p">{</span>
		<span class="n">wlr_log</span><span class="p">(</span><span class="n">WLR_DEBUG</span><span class="p">,</span> <span class="s">&#34;cursor plane: %d&#34;</span><span class="p">,</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">cursor</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">);</span>
		<span class="n">objects</span><span class="p">[</span><span class="n">j</span><span class="o">++</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">cursor</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">;</span>
	<span class="p">}</span>
	<span class="k">if</span> <span class="p">(</span><span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">num_overlays</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
		<span class="n">wlr_log</span><span class="p">(</span><span class="n">WLR_DEBUG</span><span class="p">,</span> <span class="s">&#34;+%zd overlay planes:&#34;</span><span class="p">,</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">num_overlays</span><span class="p">);</span>
	<span class="p">}</span>
	<span class="k">for</span> <span class="p">(</span><span class="n">size_t</span> <span class="n">k</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">k</span> <span class="o">&lt;</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">num_overlays</span><span class="p">;</span> <span class="o">++</span><span class="n">k</span><span class="p">)</span> <span class="p">{</span>
		<span class="n">objects</span><span class="p">[</span><span class="n">j</span><span class="o">++</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">overlays</span><span class="p">[</span><span class="n">k</span><span class="p">];</span>
		<span class="n">wlr_log</span><span class="p">(</span><span class="n">WLR_DEBUG</span><span class="p">,</span> <span class="s">&#34;</span><span class="se">\t</span><span class="s">overlay plane: %d&#34;</span><span class="p">,</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">overlays</span><span class="p">[</span><span class="n">k</span><span class="p">]);</span>
	<span class="p">}</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="n">lease_fd</span> <span class="o">=</span> <span class="n">drmModeCreateLease</span><span class="p">(</span><span class="n">backend</span><span class="o">-&gt;</span><span class="n">fd</span><span class="p">,</span>
		<span class="n">objects</span><span class="p">,</span> <span class="n">nobjects</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">lessee_id</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">lease_fd</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
	<span class="k">return</span> <span class="n">lease_fd</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">wlr_log</span><span class="p">(</span><span class="n">WLR_DEBUG</span><span class="p">,</span> <span class="s">&#34;Issued DRM lease %d&#34;</span><span class="p">,</span> <span class="o">*</span><span class="n">lessee_id</span><span class="p">);</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">nconns</span><span class="p">;</span> <span class="o">++</span><span class="n">i</span><span class="p">)</span> <span class="p">{</span>
	<span class="k">struct</span> <span class="n">wlr_drm_connector</span> <span class="o">*</span><span class="n">conn</span> <span class="o">=</span> <span class="n">conns</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
	<span class="n">conn</span><span class="o">-&gt;</span><span class="n">lessee_id</span> <span class="o">=</span> <span class="o">*</span><span class="n">lessee_id</span><span class="p">;</span>
	<span class="n">conn</span><span class="o">-&gt;</span><span class="n">crtc</span><span class="o">-&gt;</span><span class="n">lessee_id</span> <span class="o">=</span> <span class="o">*</span><span class="n">lessee_id</span><span class="p">;</span>
	<span class="n">conn</span><span class="o">-&gt;</span><span class="n">state</span> <span class="o">=</span> <span class="n">WLR_DRM_CONN_LEASED</span><span class="p">;</span>
	<span class="n">conn</span><span class="o">-&gt;</span><span class="n">lease_terminated_cb</span> <span class="o">=</span> <span class="n">lease_terminated_cb</span><span class="p">;</span>
	<span class="n">conn</span><span class="o">-&gt;</span><span class="n">lease_terminated_data</span> <span class="o">=</span> <span class="n">lease_terminated_data</span><span class="p">;</span>
	<span class="n">wlr_output_destroy</span><span class="p">(</span><span class="o">&amp;</span><span class="n">conn</span><span class="o">-&gt;</span><span class="n">output</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">lease_fd</span><span class="p">;</span>
</code></pre></div><p>The <a href="https://github.com/swaywm/sway/pull/4289">sway implementation</a> is very simple. I added a note in wlroots
which exposes whether or not an output is considered &ldquo;non-desktop&rdquo; (a property
which is set for most VR headsets), then sway just rigs up the lease manager and
offers all non-desktop outputs for lease.</p>
<h2 id="kmscube">kmscube</h2>
<p>Testing all of this required the use of a simple test client. During his earlier
work, Keith wrote some patches on top of
<a href="https://gitlab.freedesktop.org/mesa/kmscube/">kmscube</a>, a simple Mesa demo
which renders a spinning cube directly via DRM/KMS/GBM. A <a href="https://git.sr.ht/~sircmpwn/kmscube/commit/60d89ef1d9304427a1289174d9a311ab06e39b44">few simple
tweaks</a> was suitable to get this working through my protocol
extension, and for the first time I saw something rendered on my headset through
sway!</p>
<video src="https://yukari.sr.ht/vr.webm" controls>
  Your web browser does not support the webm video codec. Please consider using
  web browsers that support free and open standards.
</video>
<h2 id="vulkan">Vulkan</h2>
<p>Vulkan has a subsystem called WSI - Window System Integration - which handles
the linkage between Vulkan&rsquo;s rendering process and the underlying window system,
such as Wayland, X11, or win32. Keith added an extension to this system called
<a href="https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VK_EXT_acquire_xlib_display">VK_EXT_acquire_xlib_display</a>, which lives on top
of <a href="https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VK_EXT_direct_mode_display">VK_EXT_direct_mode_display</a>, a system for
driving displays directly with Vulkan. As the name implies, this system is
especially X11-specific, so I&rsquo;ve drafted my own VK extension for Wayland:
VK_EXT_acquire_wl_display. This is the crux of it:</p>
<div class="highlight"><pre class="chroma"><code class="language-xml" data-lang="xml"><span class="nt">&lt;command</span> <span class="na">successcodes=</span><span class="s">&#34;VK_SUCCESS&#34;</span> <span class="na">errorcodes=</span><span class="s">&#34;VK_ERROR_INITIALIZATION_FAILED&#34;</span><span class="nt">&gt;</span>
  <span class="nt">&lt;proto&gt;&lt;type&gt;</span>VkResult<span class="nt">&lt;/type&gt;</span> <span class="nt">&lt;name&gt;</span>vkAcquireWaylandDisplayEXT<span class="nt">&lt;/name&gt;&lt;/proto&gt;</span>
  <span class="nt">&lt;param&gt;&lt;type&gt;</span>VkPhysicalDevice<span class="nt">&lt;/type&gt;</span> <span class="nt">&lt;name&gt;</span>physicalDevice<span class="nt">&lt;/name&gt;&lt;/param&gt;</span>
  <span class="nt">&lt;param&gt;</span>struct <span class="nt">&lt;type&gt;</span>wl_display<span class="nt">&lt;/type&gt;</span>* <span class="nt">&lt;name&gt;</span>display<span class="nt">&lt;/name&gt;&lt;/param&gt;</span>
  <span class="nt">&lt;param&gt;</span>struct <span class="nt">&lt;type&gt;</span>zwp_drm_lease_manager_v1<span class="nt">&lt;/type&gt;</span>* <span class="nt">&lt;name&gt;</span>manager<span class="nt">&lt;/name&gt;&lt;/param&gt;</span>
  <span class="nt">&lt;param&gt;&lt;type&gt;</span>int<span class="nt">&lt;/type&gt;</span> <span class="nt">&lt;name&gt;</span>nConnectors<span class="nt">&lt;/name&gt;&lt;/param&gt;</span>
  <span class="nt">&lt;param&gt;&lt;type&gt;</span>VkWaylandLeaseConnectorEXT<span class="nt">&lt;/type&gt;</span>* <span class="nt">&lt;name&gt;</span>pConnectors<span class="nt">&lt;/name&gt;&lt;/param&gt;</span>
<span class="nt">&lt;/command&gt;</span>
</code></pre></div><p>I chose to leave it up to the user to enumerate the leasable connectors from the
Wayland protocol, then populate these structs with references to the connectors
they want to lease:</p>
<div class="highlight"><pre class="chroma"><code class="language-xml" data-lang="xml"><span class="nt">&lt;type</span> <span class="na">category=</span><span class="s">&#34;struct&#34;</span> <span class="na">name=</span><span class="s">&#34;VkWaylandLeaseConnectorEXT&#34;</span><span class="nt">&gt;</span>
  <span class="nt">&lt;member&gt;</span>struct <span class="nt">&lt;type&gt;</span>zwp_drm_lease_connector_v1<span class="nt">&lt;/type&gt;</span>* <span class="nt">&lt;name&gt;</span>pConnectorIn<span class="nt">&lt;/name&gt;&lt;/member&gt;</span>
  <span class="nt">&lt;member&gt;&lt;type&gt;</span>VkDisplayKHR<span class="nt">&lt;/type&gt;</span> <span class="nt">&lt;name&gt;</span>displayOut<span class="nt">&lt;/name&gt;&lt;/member&gt;</span>
<span class="nt">&lt;/type&gt;</span>
</code></pre></div><p>Again, this was the result of some iteration and design discussions with other
folks knowledgable in these topics. I owe special thanks to Daniel Stone for
sitting down with me (figuratively, on IRC) and going over ideas for how to
design the Vulkan API. Armed with this specification, I now needed a Vulkan
driver which supported it.</p>
<h2 id="implementing-the-vk-extension-in-mesa">Implementing the VK extension in Mesa</h2>
<p><a href="https://www.mesa3d.org/">Mesa</a> is the premier free software graphics suite
powering graphics on Linux and other operating systems. It includes an
implementation of OpenGL and Vulkan for several GPU vendors, and is the home of
the userspace end of AMDGPU, Intel, nouveau, and other graphics drivers. A
specification is nothing without its implementation, so I set out to
implementing this extension for Mesa. In the end, it turned out to be much
simpler than the corresponding X version. This is the complete code for the WSI
part of this feature:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">static</span> <span class="kt">void</span> <span class="nf">drm_lease_handle_lease_fd</span><span class="p">(</span>
      <span class="kt">void</span> <span class="o">*</span><span class="n">data</span><span class="p">,</span>
      <span class="k">struct</span> <span class="n">zwp_drm_lease_v1</span> <span class="o">*</span><span class="n">zwp_drm_lease_v1</span><span class="p">,</span>
      <span class="n">int32_t</span> <span class="n">leased_fd</span><span class="p">)</span>
<span class="p">{</span>
   <span class="k">struct</span> <span class="n">wsi_display</span> <span class="o">*</span><span class="n">wsi</span> <span class="o">=</span> <span class="n">data</span><span class="p">;</span>
   <span class="n">wsi</span><span class="o">-&gt;</span><span class="n">fd</span> <span class="o">=</span> <span class="n">leased_fd</span><span class="p">;</span>
<span class="p">}</span>

<span class="k">static</span> <span class="kt">void</span> <span class="nf">drm_lease_handle_finished</span><span class="p">(</span>
      <span class="kt">void</span> <span class="o">*</span><span class="n">data</span><span class="p">,</span>
      <span class="k">struct</span> <span class="n">zwp_drm_lease_v1</span> <span class="o">*</span><span class="n">zwp_drm_lease_v1</span><span class="p">)</span>
<span class="p">{</span>
   <span class="k">struct</span> <span class="n">wsi_display</span> <span class="o">*</span><span class="n">wsi</span> <span class="o">=</span> <span class="n">data</span><span class="p">;</span>
   <span class="k">if</span> <span class="p">(</span><span class="n">wsi</span><span class="o">-&gt;</span><span class="n">fd</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
      <span class="n">close</span><span class="p">(</span><span class="n">wsi</span><span class="o">-&gt;</span><span class="n">fd</span><span class="p">);</span>
      <span class="n">wsi</span><span class="o">-&gt;</span><span class="n">fd</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
   <span class="p">}</span>
<span class="p">}</span>

<span class="k">static</span> <span class="k">const</span> <span class="k">struct</span> <span class="n">zwp_drm_lease_v1_listener</span> <span class="n">drm_lease_listener</span> <span class="o">=</span> <span class="p">{</span>
   <span class="n">drm_lease_handle_lease_fd</span><span class="p">,</span>
   <span class="n">drm_lease_handle_finished</span><span class="p">,</span>
<span class="p">};</span>

<span class="cm">/* VK_EXT_acquire_wl_display */</span>
<span class="n">VkResult</span>
<span class="nf">wsi_acquire_wl_display</span><span class="p">(</span><span class="n">VkPhysicalDevice</span> <span class="n">physical_device</span><span class="p">,</span>
                       <span class="k">struct</span> <span class="n">wsi_device</span> <span class="o">*</span><span class="n">wsi_device</span><span class="p">,</span>
                       <span class="k">struct</span> <span class="n">wl_display</span> <span class="o">*</span><span class="n">display</span><span class="p">,</span>
                       <span class="k">struct</span> <span class="n">zwp_drm_lease_manager_v1</span> <span class="o">*</span><span class="n">manager</span><span class="p">,</span>
                       <span class="kt">int</span> <span class="n">nConnectors</span><span class="p">,</span>
                       <span class="n">VkWaylandLeaseConnectorEXT</span> <span class="o">*</span><span class="n">connectors</span><span class="p">)</span>
<span class="p">{</span>
   <span class="k">struct</span> <span class="n">wsi_display</span> <span class="o">*</span><span class="n">wsi</span> <span class="o">=</span>
      <span class="p">(</span><span class="k">struct</span> <span class="n">wsi_display</span> <span class="o">*</span><span class="p">)</span> <span class="n">wsi_device</span><span class="o">-&gt;</span><span class="n">wsi</span><span class="p">[</span><span class="n">VK_ICD_WSI_PLATFORM_DISPLAY</span><span class="p">];</span>

   <span class="cm">/* XXX no support for mulitple leases yet */</span>
   <span class="k">if</span> <span class="p">(</span><span class="n">wsi</span><span class="o">-&gt;</span><span class="n">fd</span> <span class="o">&gt;=</span> <span class="mi">0</span><span class="p">)</span>
      <span class="k">return</span> <span class="n">VK_ERROR_INITIALIZATION_FAILED</span><span class="p">;</span>

   <span class="cm">/* XXX no support for mulitple connectors yet */</span>
   <span class="cm">/* The solution will eventually involve adding a listener to each
</span><span class="cm">    * connector, round tripping, and matching EDIDs once the lease is
</span><span class="cm">    * granted. */</span>
   <span class="k">if</span> <span class="p">(</span><span class="n">nConnectors</span> <span class="o">&gt;</span> <span class="mi">1</span><span class="p">)</span>
      <span class="k">return</span> <span class="n">VK_ERROR_INITIALIZATION_FAILED</span><span class="p">;</span>

   <span class="k">struct</span> <span class="n">zwp_drm_lease_request_v1</span> <span class="o">*</span><span class="n">lease_request</span> <span class="o">=</span>
      <span class="n">zwp_drm_lease_manager_v1_create_lease_request</span><span class="p">(</span><span class="n">manager</span><span class="p">);</span>
   <span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">nConnectors</span><span class="p">;</span> <span class="o">++</span><span class="n">i</span><span class="p">)</span> <span class="p">{</span>
      <span class="n">zwp_drm_lease_request_v1_request_connector</span><span class="p">(</span><span class="n">lease_request</span><span class="p">,</span>
                                                 <span class="n">connectors</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">pConnectorIn</span><span class="p">);</span>
   <span class="p">}</span>

   <span class="k">struct</span> <span class="n">zwp_drm_lease_v1</span> <span class="o">*</span><span class="n">drm_lease</span> <span class="o">=</span>
      <span class="n">zwp_drm_lease_request_v1_submit</span><span class="p">(</span><span class="n">lease_request</span><span class="p">);</span>
   <span class="n">zwp_drm_lease_request_v1_destroy</span><span class="p">(</span><span class="n">lease_request</span><span class="p">);</span>
   <span class="n">zwp_drm_lease_v1_add_listener</span><span class="p">(</span><span class="n">drm_lease</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">drm_lease_listener</span><span class="p">,</span> <span class="n">wsi</span><span class="p">);</span>
   <span class="n">wl_display_roundtrip</span><span class="p">(</span><span class="n">display</span><span class="p">);</span>

   <span class="k">if</span> <span class="p">(</span><span class="n">wsi</span><span class="o">-&gt;</span><span class="n">fd</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="p">)</span>
      <span class="k">return</span> <span class="n">VK_ERROR_INITIALIZATION_FAILED</span><span class="p">;</span>

   <span class="kt">int</span> <span class="n">nconn</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
   <span class="n">drmModeResPtr</span> <span class="n">res</span> <span class="o">=</span> <span class="n">drmModeGetResources</span><span class="p">(</span><span class="n">wsi</span><span class="o">-&gt;</span><span class="n">fd</span><span class="p">);</span>
   <span class="n">drmModeObjectListPtr</span> <span class="n">lease</span> <span class="o">=</span> <span class="n">drmModeGetLease</span><span class="p">(</span><span class="n">wsi</span><span class="o">-&gt;</span><span class="n">fd</span><span class="p">);</span>
   <span class="k">for</span> <span class="p">(</span><span class="n">uint32_t</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">res</span><span class="o">-&gt;</span><span class="n">count_connectors</span><span class="p">;</span> <span class="o">++</span><span class="n">i</span><span class="p">)</span> <span class="p">{</span>
      <span class="k">for</span> <span class="p">(</span><span class="n">uint32_t</span> <span class="n">j</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">j</span> <span class="o">&lt;</span> <span class="n">lease</span><span class="o">-&gt;</span><span class="n">count</span><span class="p">;</span> <span class="o">++</span><span class="n">j</span><span class="p">)</span> <span class="p">{</span>
         <span class="k">if</span> <span class="p">(</span><span class="n">res</span><span class="o">-&gt;</span><span class="n">connectors</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">!=</span> <span class="n">lease</span><span class="o">-&gt;</span><span class="n">objects</span><span class="p">[</span><span class="n">j</span><span class="p">])</span> <span class="p">{</span>
            <span class="k">continue</span><span class="p">;</span>
         <span class="p">}</span>
         <span class="k">struct</span> <span class="n">wsi_display_connector</span> <span class="o">*</span><span class="n">connector</span> <span class="o">=</span>
            <span class="n">wsi_display_get_connector</span><span class="p">(</span><span class="n">wsi_device</span><span class="p">,</span> <span class="n">res</span><span class="o">-&gt;</span><span class="n">connectors</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span>
         <span class="cm">/* TODO: Match EDID with requested connector */</span>
         <span class="n">connectors</span><span class="p">[</span><span class="n">nconn</span><span class="p">].</span><span class="n">displayOut</span> <span class="o">=</span>
            <span class="n">wsi_display_connector_to_handle</span><span class="p">(</span><span class="n">connector</span><span class="p">);</span>
         <span class="o">++</span><span class="n">nconn</span><span class="p">;</span>
      <span class="p">}</span>
   <span class="p">}</span>
   <span class="n">drmModeFreeResources</span><span class="p">(</span><span class="n">res</span><span class="p">);</span>

   <span class="k">return</span> <span class="n">VK_SUCCESS</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div><p>Rigging it up to each driver&rsquo;s WSI shim is pretty straightforward from this
point. I only did it for radv - AMD&rsquo;s Vulkan driver (cause that&rsquo;s the hardware I
was using at the time) - but the rest should be trivial to add. Equipped with a
driver in hand, it&rsquo;s time to make a Real VR Application work on Wayland.</p>
<h2 id="xrgears">xrgears</h2>
<p><a href="https://gitlab.com/lubosz/xrgears">xrgears</a> is another simple demo application
like kmscube - but designed to render a VR scene. It leverages Vulkan and
<a href="http://www.openhmd.net/">OpenHMD</a> (Open Head Mounted Display) to display this
scene and stick the camera to your head. With the Vulkan extension implemented,
it was a fairly simple matter to <a href="https://git.sr.ht/~sircmpwn/xrgears/commit/41ef1d1dfe3e56766d1f8b72b335567eb7842d04">rig up a Wayland backend</a>. The
result:</p>
<video src="https://yukari.sr.ht/xrgears.webm" controls>
  Your web browser does not support the webm video codec. Please consider using
  web browsers that support free and open standards.
</video>
<h2 id="xwayland">Xwayland</h2>
<p>The final step was to integrate this extension with Xwayland, so that X
applications which took advantage of Keith&rsquo;s work would work via Xwayland. This
ended up being more difficult than I expected for one reason in particular:
modes. Keith&rsquo;s Vulkan extension is designed in two steps:</p>
<ol>
<li>Convert an RandR output into a VkDisplayKHR</li>
<li>Acquire a lease for a set of VkDisplayKHRs</li>
</ol>
<p>Between these steps, you can query the modes (available resolutions and refresh
rates) of the display. However, the Wayland protocol I designed does not let you
query modes until <em>after</em> you get the DRM handle, at which point you should
query them through DRM, thus reducing the number of sources of truth and
simplifying things considerably. This is arguably a design misstep in the
original Vulkan extension, but it&rsquo;s shipped in a lot of software and is beyond
fixing. So how do we deal with it?</p>
<p>One way (which was suggested at one point) would be to change the protocol to
include the relevant mode information, so that Xwayland could populate the RandR
modes from it. I found this distasteful, because it was making the protocol more
complex for the sake of a legacy system. Another option would be to make a
second protocol which includes this extra information especially for Xwayland,
but this also seemed like a compromise that compositors would rather not make.
Yet another option would be to have Xwayland request a lease with zero objects
and scan connectors itself, but zero-object leases are not possible.</p>
<p>The option I ended up going with is to have Xwayland open the DRM device itself
and scan connectors there. This is less palatable because (1) we can&rsquo;t be sure
which DRM device is correct, and (2) we can&rsquo;t be sure Xwayland will have
permission to read it. We&rsquo;re still not sure how best to solve this in the long
term. As it stands, this approach is sufficient to get it working in the common
case. The code looks something like this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">static</span> <span class="n">RRModePtr</span> <span class="o">*</span>
<span class="nf">xwl_get_rrmodes_from_connector_id</span><span class="p">(</span><span class="n">int32_t</span> <span class="n">connector_id</span><span class="p">,</span> <span class="kt">int</span> <span class="o">*</span><span class="n">nmode</span><span class="p">,</span> <span class="kt">int</span> <span class="o">*</span><span class="n">npref</span><span class="p">)</span>
<span class="p">{</span>
    <span class="n">drmDevicePtr</span> <span class="n">devices</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span>
    <span class="n">drmModeConnectorPtr</span> <span class="n">conn</span><span class="p">;</span>
    <span class="n">drmModeModeInfoPtr</span> <span class="n">kmode</span><span class="p">;</span>
    <span class="n">RRModePtr</span> <span class="o">*</span><span class="n">rrmodes</span><span class="p">;</span>
    <span class="kt">int</span> <span class="n">drm</span><span class="p">;</span>
    <span class="kt">int</span> <span class="n">pref</span><span class="p">,</span> <span class="n">i</span><span class="p">;</span>

    <span class="o">*</span><span class="n">nmode</span> <span class="o">=</span> <span class="o">*</span><span class="n">npref</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>

    <span class="cm">/* TODO: replace with zero-object lease once kernel supports them */</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">drmGetDevices2</span><span class="p">(</span><span class="n">DRM_NODE_PRIMARY</span><span class="p">,</span> <span class="n">devices</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">1</span>
            <span class="o">||</span> <span class="o">!*</span><span class="n">devices</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-&gt;</span><span class="n">nodes</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="p">{</span>
        <span class="n">ErrorF</span><span class="p">(</span><span class="s">&#34;Failed to enumerate DRM devices&#34;</span><span class="p">);</span>
        <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="n">drm</span> <span class="o">=</span> <span class="n">open</span><span class="p">(</span><span class="n">devices</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-&gt;</span><span class="n">nodes</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">O_RDONLY</span><span class="p">);</span>
    <span class="n">drmFreeDevices</span><span class="p">(</span><span class="n">devices</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span>

    <span class="n">conn</span> <span class="o">=</span> <span class="n">drmModeGetConnector</span><span class="p">(</span><span class="n">drm</span><span class="p">,</span> <span class="n">connector_id</span><span class="p">);</span>
    <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">conn</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">close</span><span class="p">(</span><span class="n">drm</span><span class="p">);</span>
        <span class="n">ErrorF</span><span class="p">(</span><span class="s">&#34;drmModeGetConnector failed&#34;</span><span class="p">);</span>
        <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="n">rrmodes</span> <span class="o">=</span> <span class="n">xallocarray</span><span class="p">(</span><span class="n">conn</span><span class="o">-&gt;</span><span class="n">count_modes</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">RRModePtr</span><span class="p">));</span>
    <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">rrmodes</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">close</span><span class="p">(</span><span class="n">drm</span><span class="p">);</span>
        <span class="n">ErrorF</span><span class="p">(</span><span class="s">&#34;Failed to allocate connector modes&#34;</span><span class="p">);</span>
        <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
    <span class="p">}</span>

    <span class="cm">/* This spaghetti brought to you courtesey of xf86RandrR12.c
</span><span class="cm">     * It adds preferred modes first, then non-preferred modes */</span>
    <span class="k">for</span> <span class="p">(</span><span class="n">pref</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="n">pref</span> <span class="o">&gt;=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">pref</span><span class="o">--</span><span class="p">)</span> <span class="p">{</span>
        <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">conn</span><span class="o">-&gt;</span><span class="n">count_modes</span><span class="p">;</span> <span class="o">++</span><span class="n">i</span><span class="p">)</span> <span class="p">{</span>
            <span class="n">kmode</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">conn</span><span class="o">-&gt;</span><span class="n">modes</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
            <span class="k">if</span> <span class="p">((</span><span class="n">pref</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="p">((</span><span class="n">kmode</span><span class="o">-&gt;</span><span class="n">type</span> <span class="o">&amp;</span> <span class="n">DRM_MODE_TYPE_PREFERRED</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">))</span> <span class="p">{</span>
                <span class="n">xRRModeInfo</span> <span class="n">modeInfo</span><span class="p">;</span>
                <span class="n">RRModePtr</span> <span class="n">rrmode</span><span class="p">;</span>

                <span class="n">modeInfo</span><span class="p">.</span><span class="n">nameLength</span> <span class="o">=</span> <span class="n">strlen</span><span class="p">(</span><span class="n">kmode</span><span class="o">-&gt;</span><span class="n">name</span><span class="p">);</span>

                <span class="n">modeInfo</span><span class="p">.</span><span class="n">width</span> <span class="o">=</span> <span class="n">kmode</span><span class="o">-&gt;</span><span class="n">hdisplay</span><span class="p">;</span>
                <span class="n">modeInfo</span><span class="p">.</span><span class="n">dotClock</span> <span class="o">=</span> <span class="n">kmode</span><span class="o">-&gt;</span><span class="n">clock</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">;</span>
                <span class="n">modeInfo</span><span class="p">.</span><span class="n">hSyncStart</span> <span class="o">=</span> <span class="n">kmode</span><span class="o">-&gt;</span><span class="n">hsync_start</span><span class="p">;</span>
                <span class="n">modeInfo</span><span class="p">.</span><span class="n">hSyncEnd</span> <span class="o">=</span> <span class="n">kmode</span><span class="o">-&gt;</span><span class="n">hsync_end</span><span class="p">;</span>
                <span class="n">modeInfo</span><span class="p">.</span><span class="n">hTotal</span> <span class="o">=</span> <span class="n">kmode</span><span class="o">-&gt;</span><span class="n">htotal</span><span class="p">;</span>
                <span class="n">modeInfo</span><span class="p">.</span><span class="n">hSkew</span> <span class="o">=</span> <span class="n">kmode</span><span class="o">-&gt;</span><span class="n">hskew</span><span class="p">;</span>

                <span class="n">modeInfo</span><span class="p">.</span><span class="n">height</span> <span class="o">=</span> <span class="n">kmode</span><span class="o">-&gt;</span><span class="n">vdisplay</span><span class="p">;</span>
                <span class="n">modeInfo</span><span class="p">.</span><span class="n">vSyncStart</span> <span class="o">=</span> <span class="n">kmode</span><span class="o">-&gt;</span><span class="n">vsync_start</span><span class="p">;</span>
                <span class="n">modeInfo</span><span class="p">.</span><span class="n">vSyncEnd</span> <span class="o">=</span> <span class="n">kmode</span><span class="o">-&gt;</span><span class="n">vsync_end</span><span class="p">;</span>
                <span class="n">modeInfo</span><span class="p">.</span><span class="n">vTotal</span> <span class="o">=</span> <span class="n">kmode</span><span class="o">-&gt;</span><span class="n">vtotal</span><span class="p">;</span>
                <span class="n">modeInfo</span><span class="p">.</span><span class="n">modeFlags</span> <span class="o">=</span> <span class="n">kmode</span><span class="o">-&gt;</span><span class="n">flags</span><span class="p">;</span>

                <span class="n">rrmode</span> <span class="o">=</span> <span class="n">RRModeGet</span><span class="p">(</span><span class="o">&amp;</span><span class="n">modeInfo</span><span class="p">,</span> <span class="n">kmode</span><span class="o">-&gt;</span><span class="n">name</span><span class="p">);</span>
                <span class="k">if</span> <span class="p">(</span><span class="n">rrmode</span><span class="p">)</span> <span class="p">{</span>
                    <span class="n">rrmodes</span><span class="p">[</span><span class="o">*</span><span class="n">nmode</span><span class="p">]</span> <span class="o">=</span> <span class="n">rrmode</span><span class="p">;</span>
                    <span class="o">*</span><span class="n">nmode</span> <span class="o">=</span> <span class="o">*</span><span class="n">nmode</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span>
                    <span class="o">*</span><span class="n">npref</span> <span class="o">=</span> <span class="o">*</span><span class="n">npref</span> <span class="o">+</span> <span class="n">pref</span><span class="p">;</span>
                <span class="p">}</span>
            <span class="p">}</span>
        <span class="p">}</span>
    <span class="p">}</span>

    <span class="n">close</span><span class="p">(</span><span class="n">drm</span><span class="p">);</span>
    <span class="k">return</span> <span class="n">rrmodes</span><span class="p">;</span>
<span class="p">}</span>

</code></pre></div><p>A simple update to the Wayland protocol was necessary to add the <code>CONNECTOR_ID</code>
atom to the RandR output, which is used by Mesa&rsquo;s Xlib WSI code for acquiring
the display, and was reused here to line up a connector offered by the Wayland
compositor with a connector found in the kernel. The <a href="https://gitlab.freedesktop.org/xorg/xserver/merge_requests/248">rest of the
changes</a> were pretty simple, and the result is that SteamVR works,
capping everything off nicely:</p>
<video src="https://yukari.sr.ht/steamvr.webm" controls>
  Your web browser does not support the webm video codec. Please consider using
  web browsers that support free and open standards.
</video>


guid: https://drewdevault.com/2019/08/15/Status-update-August-2019.html
title: Status update, August 2019
url: https://drewdevault.com/2019/08/15/Status-update-August-2019.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Outside my window, the morning sun can be seen rising over the land of the
rising sun, as I sip from a coffee purchased at the konbini down the street. I
almost forgot to order it, as the staffer behind the counter pointed out with a
smile and a joke that, having been told in Japanese, mostly went over my head.
It&rsquo;s on this quiet Osaka morning I write today&rsquo;s status update - there are lots
of existing developments to share!</p>
<p>Let&rsquo;s start with sourcehut news. I deployed a cool feature yesterday - SSH
access to builds.sr.ht. You can now SSH into a failed build to examine the
failure and investigate the root cause. You can also get a shell on-demand for
any build image, including for experimental arm64 support. I&rsquo;ll be writing a
full-length blog post going into detail about this feature later in the week.
Additionally, with contributor Ryan Chan&rsquo;s help, man.sr.ht received a huge
overhaul which moved wikis out of man.sr.ht&rsquo;s dedicated git subsystem and into
git.sr.ht repositories, allowing you to make your wiki out of a branch of your
main project repo or browse the git data on the web. I&rsquo;ll be posting more sr.ht
news to sr.ht-announce later today if you want to hear more!</p>
<p><img src="https://sr.ht/thL-.png" alt="Screenshot of a failed build on builds.sr.ht offering SSH access to the buildenvironment"></p>
<p><a href="https://git.sr.ht/~sircmpwn/aerc/refs/0.2.0">aerc 0.2.0</a> has been released,
which included nearly 200 changes from 34 contributors. I&rsquo;m grateful to the
community for this crazy amount of support - working together we&rsquo;ll make aerc
amazing in no time. Highlights include maildir and sendmail transports, search
and filtering, support for <code>mailto:</code> links, tab completion, and more. We haven&rsquo;t
slowed down since, and the next release already has support lined up for
notmuch, more tab completion support, and more features for mail composition. In
related news, Greg Kroah-Hartman of Linux kernel fame was kind enough to <a href="http://www.kroah.com/log/blog/2019/08/14/patch-workflow-with-mutt-2019/">write
up</a>
details about his email workflow to help guide the direction of aerc. I&rsquo;ll be
writing a follow-up post next week explaining how aerc aims to solve the
problems he lays out.</p>
<p>Sway and wlroots continue chugging along as well, with the release of Sway
1.2-rc1 coming earlier this week. This release adds many features from the
recent i3 4.17 release, and adds a handful of small features and bug fixes. The
corresponding wlroots release will be pretty cool, too, adding support for
direct scanout and fixing dozens of bugs. I&rsquo;d like to draw your attention as
well to a cool project from the Sway community: Jason Francis&rsquo;s
<a href="https://github.com/cyclopsian/wdisplays">wdisplays</a>, a GUI for arranging and
configuring displays on wlroots-based desktops. The changes necessary for it to
work will land in sway 1.2, and users building from git can try it out today.</p>
<p><img src="https://sr.ht/iyU4.png" alt=""></p>
<p>On the DRM leasing and VR for Wayland work I was discussing in the last update,
I&rsquo;m happy to share that I&rsquo;ve got it working with SteamVR! I&rsquo;ve written a
<a href="https://drewdevault.com/2019/08/09/DRM-leasing-and-VR-for-Wayland.html">detailed blog post</a> which
explains all of the work that went into this project, if you want to learn about
it in depth and watch some cool videos summing up the work. There&rsquo;s still a lot
of work to do in negotiating the standardization of new interfaces to support
this feature in several projects, but all of the unknowns have been discovered
and answered. We will have VR on Wayland soon. I plan on making my way to the
<a href="https://monado.dev/">Monado</a> and <a href="https://www.khronos.org/openxr">OpenXR</a> to
help realize a top-to-bottom free software VR stack designed with Wayland in
mind. I&rsquo;ll also be joining many members of the wlroots gang at
<a href="https://xdc2019.x.org/">XDC</a> in October, where I hope to meet the people
working on OpenXR.</p>
<p>I&rsquo;ve also invested more time into my Wayland book, because I&rsquo;ve realized that at
my current pace it won&rsquo;t be done any time soon. It&rsquo;s now about half complete and
I&rsquo;ve picked up the pace considerably. If you&rsquo;re interested in helping review the
drafts, please let me know!</p>
<p>That&rsquo;s all for today. Thank you for your continued support!</p>
<small class="text-muted">
This work was possible thanks to users who support me financially. Please
consider <a href="https://drewdevault.com/donate">donating to my work</a> or <a
href="https://sourcehut.org">buying a sourcehut.org subscription</a>. Thank you!
</small>


guid: https://drewdevault.com/2019/08/19/Introducing-shell-access-for-builds.html
title: Shell access for builds.sr.ht CI
url: https://drewdevault.com/2019/08/19/Introducing-shell-access-for-builds.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Have you ever found yourself staring at a failed CI build, wondering desperately
what happened? Or, have you ever needed a fresh machine on-demand to test out an
idea in? Have you been working on Linux, but need to test something on OpenBSD?
Starting this week, builds.sr.ht can help with all of these problems, because
you can now SSH into the build environment.</p>
<small class="text-muted">
  If you didn't know, <a href="https://sourcehut.org">Sourcehut</a> is the 100%
  open/libre software forge for hackers, complete with git and Mercurial
  hosting, CI, mailing lists, and more - with no JavaScript. Try it out!
</small>
<p>The next time your build fails on builds.sr.ht, you&rsquo;ll probably notice the
following message:</p>
<p><img src="https://sr.ht/thL-.png" alt="Screenshot of builds.sr.ht showing a prompt to SSH into the failed buildVM and examine it"></p>
<p>After the build fails, we process everything normally - sending emails,
webhooks, and so on - but keep the VM booted for an additional 10 minutes. If
you do log in during this window, we keep the VM alive until you log out or
until your normal build time limit has elapsed. Once you&rsquo;ve logged in, you get a
shell and can do anything you like, such as examining the build artifacts or
tweaking the source and trying again.</p>
<pre><code>$ ssh -t builds@azusa.runners.sr.ht connect 81809
Connected to build job #81809 (failed):
https://builds.sr.ht/jobs/~sircmpwn/81809
Your VM will be terminated 4 hours from now, or when you log out.

bash-5.0 $
</code></pre><p>You can also connect to any build over SSH by adding <code>shell: true</code> to your build
manifest. When you do, the VM will be kept alive after all of the tasks have
finished (even if it doesn&rsquo;t fail) so you can SSH in. You can also SSH in before
the tasks have finished, and tail the output of the build in your terminal. An
example use case might be getting a fresh Alpine environment to test build your
package on:</p>
<script
  id="asciicast-wnLYZwDuvkbIHwgTdmnqtQpXh"
  src="https://asciinema.org/a/pafXXANiWHY9MOH2yXdVHHJRd.js" async
></script>
<p><noscript><i>This article includes third-party JavaScript content from
asciinema.org, a free- and open-source platform that I trust.</i></noscript></p>
<p>This was accomplished with a simple build manifest:</p>
<pre><code>image: alpine/edge
shell: true
sources:
- https://git.alpinelinux.org/aports
tasks:
- &quot;prep-abuild&quot;: |
    abuild-keygen -an
</code></pre><p>Since build manifests run normally in advance of your shell login, you can do
things like install your preferred editor and dotfiles, pull down your SSH keys
through <a href="https://man.sr.ht/tutorials/builds.sr.ht/using-build-secrets.md">build
secrets</a>, or
anything else you desire to set up a comfortable working environment.</p>
<p>Furthermore, by leveraging the <a href="https://man.sr.ht/builds.sr.ht/api.md">builds.sr.ht
API</a>, you can write scripts which take
advantage of the shell features. Need a NetBSD shell? With a little scripting
you can get something like this working:</p>
<script
  id="asciicast-8etTNE7Ptgmu6hO3cVDlvrAal"
  src="https://asciinema.org/a/pafXXANiWHY9MOH2yXdVHHJRd.js" async
></script>
<p>With experimental multiarch support being rolled out, soon you&rsquo;ll be just a few
keystrokes away from an ARM or PowerPC shell, too.</p>
<p>I want to expand more on SSH access in the future. Stay tuned and <a href="mailto:sir@cmpwn.com">let me
know</a> if you have any cool ideas!</p>


guid: https://drewdevault.com/2019/09/02/Interactive-SSH-programs.html
title: Building interactive SSH applications
url: https://drewdevault.com/2019/09/02/Interactive-SSH-programs.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>After the announcement of [shell access for builds.sr.ht jobs]<a href="https://drewdevault.com/2019/08/19/Introducing-shell-access-for-builds.html">builds
announcement</a>, a few people sent me some questions, wondering how this sort of
thing is done. Writing interactive SSH applications is actually pretty easy, but
it does require some knowledge of the pieces involved and a little bit of
general Unix literacy.</p>
<p>On the server, there are three steps which you can meddle with using OpenSSH:
authentication, the shell session, and the command. The shell is pretty easily
manipulated. For example, if you set the user&rsquo;s login shell to
<code>/usr/bin/nethack</code>, then <a href="https://www.nethack.org/">nethack</a> will run when they log in. Editing
this is pretty straightforward, just pop open <code>/etc/passwd</code> as root and set
their shell to your desired binary. If the user SSHes into your server with a
TTY allocated (which is done by default), then you&rsquo;ll be able to run a curses
application or something interactive.</p>
<script
  id="asciicast-CQ5iaFl8kMnOGV3x0TeI7vfjV"
  src="https://asciinema.org/a/pafXXANiWHY9MOH2yXdVHHJRd.js" async
></script>
<p><noscript><i>This article includes third-party JavaScript content from
asciinema.org, a free- and open-source platform that I trust.</i></noscript></p>
<p>However, a downside to this is that, if you choose a &ldquo;shell&rdquo; which does not
behave like a shell, it will break when the user passes additional command line
arguments, such as <code>ssh user@host ls -a</code>. To address this, instead of overriding
the shell, we can override the <em>command</em> which is run. The best place to do this
is in the user&rsquo;s <code>authorized_keys</code> file. Before each line, you can add options
which apply to users who log in with that key. One of these options is the
&ldquo;command&rdquo; option. If you add this to <code>/home/user/.ssh/authorized_keys</code> instead:</p>
<pre><code>command=&quot;/usr/bin/nethack&quot; ssh-rsa ... user
</code></pre><p>Then it&rsquo;ll use the user&rsquo;s shell (which should probably be <code>/bin/sh</code>) to run
<code>nethack</code>, which will work regardless of the command supplied by the user (which
is stored into <code>SSH_ORIGINAL_COMMAND</code> in the environment, should you need it).
There are probably some other options you want to set here, as well, for
security reasons:</p>
<pre><code>restrict,pty,command=&quot;...&quot; ssh-rsa ... user
</code></pre><p>The full list of options you can set here is available in the <code>sshd(8)</code> man
page. <code>restrict</code> just turns off most stuff by default, and <code>pty</code> explicitly
re-enables TTY allocation, so that we can do things like curses. This will work
if you want to explicitly authorize specific people, one at a time, in your
<code>authorized_keys</code> file, to use your SSH-driven application.  However, there&rsquo;s
one more place where we can meddle: the <code>AuthorizedKeysCommand</code> in
<code>/etc/ssh/sshd_config</code>. Instead of having OpenSSH read from the
<code>authorized_keys</code> file in the user&rsquo;s home directory, it can execute an arbitrary
program and read the <code>authorized_keys</code> file from its stdout. For example, on
Sourcehut we use something like this:</p>
<pre><code>AuthorizedKeysCommand /usr/bin/gitsrht-dispatch &quot;%u&quot; &quot;%h&quot; &quot;%t&quot; &quot;%k&quot;
AuthorizedKeysUser root
</code></pre><p>Respectively, these format strings will supply the command with the username
attempting login, the user&rsquo;s home directory, the type of key in use (e.g.
<code>ssh-rsa</code>), and the base64-encoded key itself. More options are available - see
<code>TOKENS</code>, in the <code>sshd_config(8)</code> man page. The key supplied here can be used to
identify the user - on Sourcehut we look up their SSH key in the database. Then
you can choose whether or not to admit the user based on any logic of your
choosing, and print an appropriate <code>authorized_keys</code> to stdout. You can also
take this opportunity to forward this information along to the command that gets
executed, by appending them to the command option or by using the environment
options.</p>
<h2 id="how-this-works-on-buildssrht">How this works on builds.sr.ht</h2>
<p>We use a somewhat complex system for incoming SSH connections, which I won&rsquo;t go
into here - it&rsquo;s only necessary to support multiple SSH applications on the same
server, like git.sr.ht and builds.sr.ht. For builds.sr.ht, we accept all
connections and authenticate later on. This means our AuthorizedKeysCommand is
quite simple:</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="ch">#!/usr/bin/env python3</span>
<span class="c1"># We just let everyone in at this stage, authentication is done later on.</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="n">key_type</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="n">b64key</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span>

<span class="n">keys</span> <span class="o">=</span> <span class="p">(</span><span class="n">f</span><span class="s2">&#34;command=</span><span class="se">\&#34;</span><span class="s2">buildsrht-shell &#39;{b64key}&#39;</span><span class="se">\&#34;</span><span class="s2">,restrict,pty &#34;</span> <span class="o">+</span>
    <span class="n">f</span><span class="s2">&#34;{key_type} {b64key} somebody</span><span class="se">\n</span><span class="s2">&#34;</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</code></pre></div><p>The command, <code>buildsrht-shell</code>, does some more interesting stuff. First, the
user is told to connect with a command like <code>ssh builds@buildhost connect &lt;job ID&gt;</code>, so we use the <code>SSH_ORIGINAL_COMMAND</code> variable to grab the command line
they included:</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="n">cmd</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;SSH_ORIGINAL_COMMAND&#34;</span><span class="p">)</span> <span class="ow">or</span> <span class="s2">&#34;&#34;</span>
<span class="n">cmd</span> <span class="o">=</span> <span class="n">shlex</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">2</span><span class="p">:</span>
    <span class="n">fail</span><span class="p">(</span><span class="s2">&#34;Usage: ssh ... connect &lt;job ID&gt;&#34;</span><span class="p">)</span>
<span class="n">op</span> <span class="o">=</span> <span class="n">cmd</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">if</span> <span class="n">op</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">[</span><span class="s2">&#34;connect&#34;</span><span class="p">,</span> <span class="s2">&#34;tail&#34;</span><span class="p">]:</span>
    <span class="n">fail</span><span class="p">(</span><span class="s2">&#34;Usage: ssh ... connect &lt;job ID&gt;&#34;</span><span class="p">)</span>
<span class="n">job_id</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">cmd</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
</code></pre></div><p>Then we do some authentication, fetching the job info from the local job runner
and checking their key against meta.sr.ht (the authentication service).</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="n">b64key</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>

<span class="k">def</span> <span class="nf">get_info</span><span class="p">(</span><span class="n">job_id</span><span class="p">):</span>
    <span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">f</span><span class="s2">&#34;http://localhost:8080/job/{job_id}/info&#34;</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="o">!=</span> <span class="mi">200</span><span class="p">:</span>
        <span class="k">return</span> <span class="bp">None</span>
    <span class="k">return</span> <span class="n">r</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>

<span class="n">info</span> <span class="o">=</span> <span class="n">get_info</span><span class="p">(</span><span class="n">job_id</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">info</span><span class="p">:</span>
    <span class="n">fail</span><span class="p">(</span><span class="s2">&#34;No such job found.&#34;</span><span class="p">)</span>

<span class="n">meta_origin</span> <span class="o">=</span> <span class="n">get_origin</span><span class="p">(</span><span class="s2">&#34;meta.sr.ht&#34;</span><span class="p">)</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">f</span><span class="s2">&#34;{meta_origin}/api/ssh-key/{b64key}&#34;</span><span class="p">)</span>
<span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">200</span><span class="p">:</span>
    <span class="n">username</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">json</span><span class="p">()[</span><span class="s2">&#34;owner&#34;</span><span class="p">][</span><span class="s2">&#34;name&#34;</span><span class="p">]</span>
<span class="k">elif</span> <span class="n">r</span><span class="o">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">404</span><span class="p">:</span>
    <span class="n">fail</span><span class="p">(</span><span class="s2">&#34;We don&#39;t recognize your SSH key. Make sure you&#39;ve added it to &#34;</span> <span class="o">+</span>
        <span class="n">f</span><span class="s2">&#34;your account.</span><span class="se">\n</span><span class="s2">{get_origin(&#39;meta.sr.ht&#39;, external=True)}/keys&#34;</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
    <span class="n">fail</span><span class="p">(</span><span class="s2">&#34;Temporary authentication failure. Try again later.&#34;</span><span class="p">)</span>

<span class="k">if</span> <span class="n">username</span> <span class="o">!=</span> <span class="n">info</span><span class="p">[</span><span class="s2">&#34;username&#34;</span><span class="p">]:</span>
    <span class="n">fail</span><span class="p">(</span><span class="s2">&#34;You are not permitted to connect to this job.&#34;</span><span class="p">)</span>
</code></pre></div><p>There are two modes from here on out: connecting and tailing. The former logs
into the local build VM, and the latter prints the logs to the terminal.
Connecting looks like this:</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="n">job_id</span><span class="p">,</span> <span class="n">info</span><span class="p">):</span>
    <span class="s2">&#34;&#34;&#34;Opens a shell on the build VM&#34;&#34;&#34;</span>
    <span class="n">limit</span> <span class="o">=</span> <span class="n">naturaltime</span><span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">-</span> <span class="n">deadline</span><span class="p">)</span>
    <span class="k">print</span><span class="p">(</span><span class="n">f</span><span class="s2">&#34;Your VM will be terminated {limit}, or when you log out.&#34;</span><span class="p">)</span>
    <span class="k">print</span><span class="p">()</span>
    <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">f</span><span class="s2">&#34;http://localhost:8080/job/{job_id}/claim&#34;</span><span class="p">)</span>
    <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
    <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
    <span class="n">tty</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="s2">&#34;/dev/tty&#34;</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">O_RDWR</span><span class="p">)</span>
    <span class="n">os</span><span class="o">.</span><span class="n">dup2</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">tty</span><span class="p">)</span>
    <span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">([</span>
        <span class="s2">&#34;ssh&#34;</span><span class="p">,</span> <span class="s2">&#34;-qt&#34;</span><span class="p">,</span>
        <span class="s2">&#34;-p&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">info</span><span class="p">[</span><span class="s2">&#34;port&#34;</span><span class="p">]),</span>
        <span class="s2">&#34;-o&#34;</span><span class="p">,</span> <span class="s2">&#34;UserKnownHostsFile=/dev/null&#34;</span><span class="p">,</span>
        <span class="s2">&#34;-o&#34;</span><span class="p">,</span> <span class="s2">&#34;StrictHostKeyChecking=no&#34;</span><span class="p">,</span>
        <span class="s2">&#34;-o&#34;</span><span class="p">,</span> <span class="s2">&#34;LogLevel=quiet&#34;</span><span class="p">,</span>
        <span class="s2">&#34;build@localhost&#34;</span><span class="p">,</span> <span class="s2">&#34;bash&#34;</span>
    <span class="p">])</span>
    <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">f</span><span class="s2">&#34;http://localhost:8080/job/{job_id}/terminate&#34;</span><span class="p">)</span>
</code></pre></div><p>This is pretty self explanatory, except perhaps for the dup2 - we just open
<code>/dev/tty</code> and make <code>stdin</code> a copy of it. Some interactive applications
misbehave if stdin is not a tty, and this mimics the normal behavior of SSH.
Then we log into the build VM over SSH, which with stdin/stdout/stderr rigged up
like so will allow the user to interact with the build VM. After that completes,
we terminate the VM.</p>
<p>This is mostly plumbing work that just serves to get the user from point A to
point B. The tail functionality is more application-like:</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">tail</span><span class="p">(</span><span class="n">job_id</span><span class="p">,</span> <span class="n">info</span><span class="p">):</span>
    <span class="s2">&#34;&#34;&#34;Tails the build logs to stdout&#34;&#34;&#34;</span>
    <span class="n">logs</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">cfg</span><span class="p">(</span><span class="s2">&#34;builds.sr.ht::worker&#34;</span><span class="p">,</span> <span class="s2">&#34;buildlogs&#34;</span><span class="p">),</span> <span class="nb">str</span><span class="p">(</span><span class="n">job_id</span><span class="p">))</span>
    <span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">([</span><span class="s2">&#34;tail&#34;</span><span class="p">,</span> <span class="s2">&#34;-f&#34;</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">logs</span><span class="p">,</span> <span class="s2">&#34;log&#34;</span><span class="p">)])</span>
    <span class="n">tasks</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
    <span class="n">procs</span> <span class="o">=</span> <span class="p">[</span><span class="n">p</span><span class="p">]</span>
    <span class="c1"># holy bejeezus this is hacky</span>
    <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
        <span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">manifest</span><span class="o">.</span><span class="n">tasks</span><span class="p">:</span>
            <span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="n">name</span> <span class="ow">in</span> <span class="n">tasks</span><span class="p">:</span>
                <span class="k">continue</span>
            <span class="n">path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">logs</span><span class="p">,</span> <span class="n">task</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="s2">&#34;log&#34;</span><span class="p">)</span>
            <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
                <span class="n">procs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span>
                    <span class="n">f</span><span class="s2">&#34;tail -f {shlex.quote(path)} | &#34;</span> <span class="o">+</span>
                    <span class="s2">&#34;awk &#39;{ print </span><span class="se">\&#34;</span><span class="s2">[&#34;</span> <span class="o">+</span> <span class="n">shlex</span><span class="o">.</span><span class="n">quote</span><span class="p">(</span><span class="n">task</span><span class="o">.</span><span class="n">name</span><span class="p">)</span> <span class="o">+</span> <span class="s2">&#34;] </span><span class="se">\&#34;</span><span class="s2"> $0 }&#39;&#34;</span><span class="p">,</span>
                    <span class="n">shell</span><span class="o">=</span><span class="bp">True</span><span class="p">))</span>
                <span class="n">tasks</span><span class="o">.</span><span class="n">update</span><span class="p">({</span> <span class="n">task</span><span class="o">.</span><span class="n">name</span> <span class="p">})</span>
        <span class="n">info</span> <span class="o">=</span> <span class="n">get_info</span><span class="p">(</span><span class="n">job_id</span><span class="p">)</span>
        <span class="k">if</span> <span class="ow">not</span> <span class="n">info</span><span class="p">:</span>
            <span class="k">break</span>
        <span class="k">if</span> <span class="n">info</span><span class="p">[</span><span class="s2">&#34;task&#34;</span><span class="p">]</span> <span class="o">==</span> <span class="n">info</span><span class="p">[</span><span class="s2">&#34;tasks&#34;</span><span class="p">]:</span>
            <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">procs</span><span class="p">:</span>
                <span class="n">p</span><span class="o">.</span><span class="n">kill</span><span class="p">()</span>
            <span class="k">break</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>

<span class="k">if</span> <span class="n">op</span> <span class="o">==</span> <span class="s2">&#34;connect&#34;</span><span class="p">:</span>
    <span class="k">if</span> <span class="n">info</span><span class="p">[</span><span class="s2">&#34;task&#34;</span><span class="p">]</span> <span class="o">!=</span> <span class="n">info</span><span class="p">[</span><span class="s2">&#34;tasks&#34;</span><span class="p">]</span> <span class="ow">and</span> <span class="n">info</span><span class="p">[</span><span class="s2">&#34;status&#34;</span><span class="p">]</span> <span class="o">==</span> <span class="s2">&#34;running&#34;</span><span class="p">:</span>
        <span class="n">tail</span><span class="p">(</span><span class="n">job_id</span><span class="p">,</span> <span class="n">info</span><span class="p">)</span>
    <span class="n">connect</span><span class="p">(</span><span class="n">job_id</span><span class="p">,</span> <span class="n">info</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">op</span> <span class="o">==</span> <span class="s2">&#34;tail&#34;</span><span class="p">:</span>
    <span class="n">tail</span><span class="p">(</span><span class="n">job_id</span><span class="p">,</span> <span class="n">info</span><span class="p">)</span>
</code></pre></div><p>This&hellip; I&hellip; let&rsquo;s just pretend you never saw this. And that&rsquo;s how SSH access to
builds.sr.ht works!</p>


guid: https://drewdevault.com/2019/09/08/Enough-to-decide.html
title: How I decide between many programming languages
url: https://drewdevault.com/2019/09/08/Enough-to-decide.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I have a few old standards in my toolbelt that I find myself calling upon most
often, but I try to learn enough about many programming languages to reason
about whether or not they&rsquo;re suitable to any use-case I&rsquo;m thinking about. The
best way is to learn by doing, so getting a general impression of the utility of
many languages helps equip you with the knowledge of whether or not they&rsquo;d be
useful for a particular problem even if you don&rsquo;t know them yet.</p>
<p>Only included are languages which I feel knowledgable enough about to comment
on, there are many that aren&rsquo;t here and which I encourage you to research.</p>
<h2 id="c">C</h2>
<p>Pros: good performance, access to low-level tooling, useful in systems
programming, statically typed, standardized and venerable, the lingua franca,
universal support on all platforms.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p>
<p>Cons: string munging, extensible programming, poor availability of ergonomic
libraries in certain domains, has footguns, some programmers in the wild think
the footguns are useful.</p>
<style>
.bullshit {
    color: white;
    font-style: italic;
    font-weight: bold;
    text-shadow: -1px -1px 0 #000,
           1px -1px 0 #000, -1px 1px 0 #000, 1px 1px 0 #000;
}
</style>
<h2 id="go">Go</h2>
<p>Pros: fast, conservative, good package manager and a healthy ecosystem, standard
library is well designed, best in class for many problems, has a spec and
multiple useful implementations, easy interop with C.</p>
<p>Cons: the runtime is too complicated, no distinction between green threads and
real threads (meaning all programs deal with the problems of the latter).</p>
<h2 id="rust">Rust</h2>
<p>Pros: it&rsquo;s <em class="bullshit">SAFE</em>, useful for systems programming,
better than C++, ecosystem which is diverse but just short of the npm disease,
easy interop with C.</p>
<p>Cons: far too big, non-standardized, only one meaningful implementation.</p>
<h2 id="python">Python</h2>
<p>Pros: easy and fast to get things done, diverse package ecosystem of reasonably
well designed packages, deeply extensible, useful for server-side web software.</p>
<p>Cons: bloated, poor performance, dynamically typed, cpython internals being
available to programmers has led to an implementation monoculture.</p>
<h2 id="javascript">JavaScript</h2>
<p><em>* and all of its derivatives, which ultimately inherit its problems.</em></p>
<p>Pros: functional but with an expressive and C-like syntax, ES6 improved on many
fronts, async/await/promises are well designed, no threading.</p>
<p>Cons: dynamic types, package ecosystem is a flaming pile, many JS programmers
aren&rsquo;t very good at it and they make ecosystem-defining libraries anyway, born
in web browsers and inherited their many flaws.</p>
<h2 id="java">Java</h2>
<p><em>* and all of its derivatives, which ultimately inherit its problems.</em></p>
<p>Pros: has had enough long-term investment to be well understood and reasonably
fast.</p>
<p>Cons: hella boilerplate, missing lots of useful things, package management, XML
is everywhere, not useful for low-level programming (this applies to all
Java-family languages).</p>
<h2 id="c-1">C#</h2>
<p>Pros: less boilerplate than Java, reasonably healthy package ecosystem, good
access to low level tools for interop with C, async/await started here.</p>
<p>Cons: ecosystem is in turmoil because Microsoft cannot hold a singular vision,
they became open-source too late and screwed over Mono.</p>
<h2 id="haskell">Haskell</h2>
<p><em>* and every other functional-oriented programming language in its class, such
as elixir, erlang, most lisps, even if they resent being lumped together</em></p>
<p>Pros: it&rsquo;s <em class="bullshit">FUNCTIONAL</em>, reasonably fast, useful
when the answer to your problem is more important than the means by which you
find it, good for research-grade<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> compilers.</p>
<p>Cons: it&rsquo;s <em class="bullshit">FUNCTIONAL</em>, somewhat inscrutable, awful
package management, does not fit well into its environment, written by people
who wish the world could be described with a pure function and design software
as if it could.</p>
<h2 id="perl">Perl</h2>
<p>Pros: <a href="https://github.com/Perl/perl5/blob/blead/Configure">entertaining</a>, best in class at regexes/string munging,
useful for making hacky kludges when such solutions are appropriate.</p>
<p>Cons: inscrutable, too extensible, too much junk/jank.</p>
<h2 id="lua">Lua</h2>
<p>Pros: embeddable &amp; easily plugged into its host, fairly simple, portable.</p>
<p>Cons: 1-based indexing is objectively bad, the upstream maintainers are kind of
doing their own thing and no one really likes it.</p>
<h2 id="posix-shell-scripts">POSIX Shell scripts</h2>
<p>Pros: nothing can string together commands better, if you learn 90% of it then
you can make pretty nice and expressive programs with it for a certain class of
problem, standardized (I do not use bash).</p>
<p>Cons: most people learn only 10% of it and therefore make pretty bad and
unintuitive programs with it, not useful for most complex tasks.</p>
<hr>
<p>Disclaimer: I don&rsquo;t like the rest of these programming languages and would not
use them to solve any problem. If you don&rsquo;t want your sacred cow gored, leave
here.</p>
<h2 id="c-2">C++</h2>
<p>Pros: none</p>
<p>Cons: ill-defined, far too big, <em class="bullshit">
Object Oriented Programming</em>, loads of baggage, ecosystem that buys into
its crap, enjoyed by bad programmers.</p>
<h2 id="php">PHP</h2>
<p>Pros: none</p>
<p>Cons: every PHP programmer is bad at programming, the language is designed to
accommodate them with convenient footguns (or faceguns) at every step, and the
ecosystem is accordingly bad. No, PHP7 doesn&rsquo;t fix this. Use a real programming
language, jerk.</p>
<h2 id="ruby">Ruby</h2>
<p>Pros: It&rsquo;s both <em class="bullshit">ENTERPRISE</em> and <em
class="bullshit">HIP</em> at the same time, and therefore effective at herding a
group of junior to mid-level programmers in a certain direction, namely towards
your startup&rsquo;s exit.</p>
<p>Cons: bloated, awful performance, before Node.js took off this is what all of
those programmers used.</p>
<h2 id="scala">Scala</h2>
<p>Pros: more expressive than Java, useful for <em class="bullshit">Big
Data</em> problems.</p>
<p>Cons: Java derivative, type system requires a PhD to comprehend, too siloed from
Java, meaning it gets all of the disadvantages of being a Java ecosystem member
but few of the advantages. The type system is so needlessly complicated that it
basically cripples the language on its own merits alone.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Except one, and it can go suck an egg for all I care. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>but not production-grade. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/09/15/Status-update-September-2019.html
title: Status update, September 2019
url: https://drewdevault.com/2019/09/15/Status-update-September-2019.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Finally home again after a long series of travels! I spent almost a month in
Japan, then visited my sister&rsquo;s new home in Hawaii on the way eastwards, then
some old friends in Seattle, and finally after 5½ long weeks, it&rsquo;s home sweet
home here in Philadelphia. At least until I leave for
<a href="https://xdc2019.x.org/">XDC</a> in Montreal 2 weeks from now. Someday I&rsquo;ll have
some rest&hellip; throughout all of these wild travels, I&rsquo;ve been hard at work on my
free software projects. Let&rsquo;s get started with this month&rsquo;s status update!</p>
<p><img src="https://sr.ht/iuDE.jpg" alt=""></p>
<p style="text-align: center">
  <small>Great view from a hike on O'ahu</small>
</p>
<p>First, Wayland news. I&rsquo;m happy to share with you that the Wayland book is now
more than halfway complete, and I&rsquo;ve made the drafts available online for a
discounted price: <a href="https://wayland-book.com">The Wayland Protocol</a>. Thanks to
all of my collaborators and readers who volunteered to provide feedback! There&rsquo;s
more Wayland-related news still, as this month marked the release of <a href="https://github.com/swaywm/sway/releases/tag/1.2">sway
1.2</a> and <a href="https://github.com/swaywm/wlroots/releases/tag/0.7.0">wlroots 0.7.0</a>. I like this release
because it&rsquo;s light on new features - showing that sway is maturing into a stable
and reliable Wayland desktop. The features which were added are subtle and serve
to improve sway&rsquo;s status as a member of the broader ecosystem - sway 1.2
supports the new <a href="https://github.com/mate-desktop/mate-panel/pull/991">layer shell support in the MATE panel</a>, and the
same improvements are already helping with the development of other software.</p>
<p><a href="https://sr.ht/9Oro.png"><img src="https://sr.ht/9Oro.png" alt="Screenshot of MATE panel running on sway"></a></p>
<p style="text-align: center">
  <small>Rest assured, the weird alignment issues were fixed</small>
</p>
<p>On the topic of <a href="https://aerc-mail.org">aerc</a>, I still haven&rsquo;t gotten around to
that write-up responding to <a href="http://www.kroah.com/log/blog/2019/08/14/patch-workflow-with-mutt-2019/">Greg KH&rsquo;s post</a>&hellip; but I will. Travels have
made it difficult to sit down for a while and do some serious long-term project
planning. Regardless, the current plans have still been being executed well.
Notmuch support continues to improve thanks to Reto Brunner&rsquo;s help, completions
are improving throughout, and heaps of little features - signatures, unread
message counts, :prompt, forward-as-attachment - are now supported.</p>
<p>I also spent some time this month working on Simon Ser&rsquo;s
<a href="https://mrsh.sh">mrsh</a>. I cleaned up call frames, implemented the <code>return</code>
builtin, finished the <code>pwd</code> builtin, improved readline support, fleshed out job
control, and made many other small improvements. With mrsh nearing completion,
I&rsquo;ve started up another project: <a href="https://git.sr.ht/~sircmpwn/ctools">ctools</a>. This provides the rest of the
POSIX commands required of a standard scripting environment (it replaces
coreutils or busybox). I&rsquo;m taking this one pretty seriously from the start -
every command has full POSIX.1-2017 support with a conformance test and a man
page, in one C source file and no dependencies. If you&rsquo;re looking for a good
afternoon project (or weekend, for some utilities), how about picking up your
favorite <a href="https://pubs.opengroup.org/onlinepubs/9699919799/">POSIX</a> tool and
sending along an implementation?</p>
<p><a href="https://builds.sr.ht/~sircmpwn/job/88955"><img src="https://sr.ht/DSxS.png" alt="Screenshot of ctools test suite"></a></p>
<p>With these projects, along with ~mcf&rsquo;s <a href="https://git.sr.ht/~mcf/cproc">cproc</a>,
we&rsquo;re starting to see a simple and elegant operating system come together -
exactly the kind I wish we already had. To track our progress towards this goal,
I&rsquo;ve put up <a href="https://arewesimpleyet.org">arewesimpleyet.org</a>. A day may soon
come when computers become the again elegant and simple tools they were always
meant to be! At least if we assume &ldquo;within a few decades&rdquo; as a valid definition
of &ldquo;soon&rdquo;.</p>
<p>To cover SourceHut news briefly: we hit 10,000 users this month! And it&rsquo;s
continued to grow since, up to 10,649 users at the time of writing. On the
subject of feature development, with Denis Laxalde&rsquo;s help we&rsquo;re starting to put
together a Debian repository for installing the services on Debian hosts. On
todo.sr.ht, users without accounts can now create and comment on tickets via
email. I also redesigned <a href="https://sourcehut.org">sourcehut.org</a>, adding a blog
with a greater breadth of topics than we&rsquo;ll see on the sr.ht-announce mailing
list.</p>
<p>That&rsquo;s all for this month! I enjoyed my vacation and some much needed time away
from work&hellip; though for me a &ldquo;day off&rdquo; is a day where I write less than 1,000
lines of code. Thank you again for your support - it means the world to me. I&rsquo;ll
see you next month!</p>
<p><img src="https://sr.ht/1cuE.jpg" alt=""></p>
<p style="text-align: center">
  <small>Had the best seats at a concert in Tokyo!</small>
</p>


guid: https://drewdevault.com/2019/09/17/The-wrong-words-but-the-right-ideas.html
title: Don't sacrifice the right ideas to win the right words
url: https://drewdevault.com/2019/09/17/The-wrong-words-but-the-right-ideas.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>There is a difference between free software and open-source software. But you
have to squint to see it. Software licenses which qualify for one title but not
the other are exceptionally rare.</p>
<p>A fascination with linguistics is common among hackers, and I encourage and
participate in language hacking myself. Unfortunately, that seems to seep into
the Free Software Foundation&rsquo;s message a bit too much. Let&rsquo;s see if any of this
rings familiar:</p>
<blockquote>
<p>It&rsquo;s not actually open source, but free software. You see, &ldquo;open source&rdquo; is a
plot by the commercial software industry to subvert the &ldquo;free software&rdquo;
movement&hellip;</p>
</blockquote>
<blockquote>
<p>No, it&rsquo;s free-as-in-freedom, not free-as-in-beer. Sometimes we call it &ldquo;libre&rdquo;
software, borrowing the French or Spanish word, because in English&hellip;</p>
</blockquote>
<blockquote>
<p>What you&rsquo;re referring to as Linux, is in fact, GNU/Linux, or as I&rsquo;ve recently
taken to calling it, GNU plus Linux. Linux is not an operating system&hellip;</p>
</blockquote>
<p>What do all of these have in common? The audience already agrees with the
speaker on the ideas, but this becomes less so with every word. This kind of
pedantry lacks tact and pushes people away from the movement. No one wants to
talk to someone who corrects them like this, so people shut down and stop
listening. The speaker gains the self-satisfaction that comes with demonstrating
that you&rsquo;re smarter than someone else, but the cost is pushing that person away
from the very ideals you&rsquo;re trying to clarify. This approach doesn&rsquo;t help the
movement, it&rsquo;s just being a dick.</p>
<p>For this reason, even though I fully understand the difference between free and
open-source software, I use the terms basically interchangeably. In practice
they are effectively the same thing. Then, I preach the ideologies behind free
software even when discussing open-source software.  The ideas are what matters,
the goal is to get people thinking on your wavelength. If they hang around long
enough, they&rsquo;ll start using your words, too.  That&rsquo;s how language works.</p>
<p>The crucial distinction of the free software movement is less about &ldquo;free
software&rdquo;, after all, and more about copyleft. But, because the FSF pushes
copyleft <em>and</em> free software, and because many FSF advocates are pedantic and
abrasive, many people check out before they&rsquo;re told the distinction between free
software and copyleft. This leads to the listener <em>equivocating</em> free software
with copyleft software, which undermines the message and hurts both.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p>
<p>This lack of tact is why I find it difficult to accept the FSF as a
representative of the movement I devote myself to. If your goal is to strengthen
the resolve and unity of people who already agree with you by appealing to
tribalism, then this approach is effective - but remember that it strengthens
the opposing tribes, too. If your goal is to grow the movement and win the
hearts and minds of the people, then you need to use more tact in your language.
Turn that hacker knack for linguistic hacking towards <em>this</em> goal, of thinking
over how your phrasing and language makes different listeners feel. The
resulting literature will be much more effective.</p>
<p>Attack the systems and individuals who brought about the circumstances that
frustrate your movement, but don&rsquo;t attack their victims. It&rsquo;s not the user&rsquo;s
fault that they were raised on proprietary software. The system which installed
proprietary software on their school computers is the one to blame. Our goals
should be things like introducing Linux to the classroom, petitioning our
governments to require taxpayer-funded software to be open source, eliminating
Digital Restrictions Management<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>, pushing for right to repair, and so on. Why
is &ldquo;get everyone to say &lsquo;libre&rsquo; instead of &lsquo;open-source&rsquo;&rdquo; one of our goals
instead?</p>
<p>An aside: sometimes language <em>is</em> important. When someone has the wrong words
but the right ideas, it&rsquo;s not a big deal. When someone has the wrong <em>ideas</em> and
is appropriating the words to support them, that&rsquo;s a problem. This is why I
still come down hard on companies which gaslight users with faux-open software
licenses like the Commons Clause or the debacle with RedisLabs.</p>
<p><em>Note: this article is not about Richard Stallman. I have no comment on the
recent controversies.</em></p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>For those unaware, copyleft is any &ldquo;viral&rdquo; license, where using copyleft code requires also using a copyleft license for your derived work.  Free software is just software which meets the <a href="https://www.gnu.org/philosophy/free-sw.html">free software definition</a>, which is in practice just about all free <em>and</em> open-source software, including MIT or BSD licensed works. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>This kind of pedantry, which deliberately misrepresents the acronym (which is rightly meant to be &ldquo;Digital Rights Management&rdquo;), is more productive, since the people insulted by it are not the victims of DRM, but the perpetrators of it. Also, &ldquo;Digital Rights Management&rdquo; is itself a euphemism, or perhaps more accurately a kind of doublespeak, which invites a similar response. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/09/23/RaptorCS-Blackbird-a-horror-story.html
title: RaptorCS POWER9 Blackbird PC review
url: https://drewdevault.com/2019/09/23/RaptorCS-Blackbird-a-horror-story.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p><strong>November 2018</strong>: Ordered <a href="https://www.raptorcs.com/content/BK1B01/intro.html">Basic Blackbird
Bundle</a> w/32 GB RAM:
$1,935.64</p>
<p><strong>Update 2019-12-23</strong>: This article was originally titled &ldquo;RaptorCS POWER9
Blackbird PC: An expensive mistake&rdquo;. Please read the follow-up article,
published 2019-10-10:
<a href="https://drewdevault.com/2019/10/10/RaptorCS-redemption.html">RaptorCS&rsquo;s redemption: the POWER9 machine works</a></p>
<p><strong>June 2019</strong></p>
<p>Order ships, and arrives without RAM. It had been long enough that I didn&rsquo;t
realize the order had only been partially fulfilled, so I order some RAM from
the <a href="https://wiki.raptorcs.com/wiki/POWER9_Hardware_Compatibility_List/Memory">list of recommended chips</a> ($338.40), along with the other necessities
that I didn&rsquo;t purchase from Raptor: a case ($97.99) and a PSU ($68.49), and grab
some hard drives I have lying around. Total cost: about $2,440. Worth it to get
POWER9 builds working on builds.sr.ht!</p>
<p>I carefully put everything together, consulting the manual at each step, plug in
a display, and turn it on. Lights come on, things start whizzing, and the screen
comes to life - and promptly starts boot looping.</p>
<p><strong>June 27th</strong></p>
<p>Support ticket created. What&rsquo;s going on with my board?</p>
<p><strong>June 28th</strong></p>
<p>Support gets back to me the next day with a suggestion which is unrelated to the
problem, but no matter - I spoke with volunteers in the IRC channel a few hours
earlier and we found out that - whoops! - I hadn&rsquo;t connected the CPU power to
the motherboard. This is the end of the PEBKAC errors, but not the end of the
problems. The machine gets further ahead in the boot - almost to &ldquo;petitboot&rdquo;,
and then the display dies and the machine reveals no further secrets.</p>
<p>I sent an update to the support team.</p>
<p><strong>July 1st</strong></p>
<blockquote>
<p>We have normally only seen this type of failure when there is a RAM-related
fault, or if the PSU is underpowered enough that bringing the CPUs online at
full power causes a power fault and immediate safety power off.</p>
<p>Can you watch the internal lights while the system is booting, and see if the
power LED cluster immediately changes from green to orange as the system stops
responding over SSH?</p>
</blockquote>
<p>The IRC channel suspects this is not related to the problem. Regardless, I reply
a few hours later with two videos showing the boot up process from power-out to
display death, with the internal LEDs and the display output clearly visible.</p>
<p><strong>July 4th</strong></p>
<p>&ldquo;Any progress on this issue?&rdquo;, I ask.</p>
<p><strong>July 15th</strong></p>
<p>&ldquo;Hi guys, I&rsquo;m still experiencing this problem. If you&rsquo;re unsure of the issue I
would like to send the board back to you for diagnosis or a refund.&rdquo;</p>
<p><strong>July 25th</strong></p>
<blockquote>
<p>Sorry for the delay. Having senior support check out the videos.</p>
<p>Thanks for writing back. We should have something for you by tomorrow during
the day.</p>
</blockquote>
<p><strong>July 31st</strong></p>
<blockquote>
<p>Hi Drew.</p>
<p>The videos are being reviewed this week. Thank you for sending them.</p>
<p>Please stay tuned.</p>
</blockquote>
<p><strong>September 15th</strong></p>
<p>No reply from support. I have since bought a little more hardware for
self-diagnosis, namely the necessary pieces to connect to the two (or is it 3?)
serial ports. I manage to get a log, which points to several failures, but none
of them seem to be related to the problem at hand (they do indicate some network
failures, which would explain why I can&rsquo;t log into the BMC over SSH for further
diagnosis). And the getty is looping, so I can&rsquo;t log in on the serial console to
explore any further.</p>
<hr>
<p>That was a week ago. Radio silence since.</p>
<p>So, 10 months after I placed an order for a POWER9 machine, 3 months after I
received it (without the RAM I purchased, no less), and over $2,500 invested&hellip;
it&rsquo;s clear that buying the Blackbird was an expensive mistake. Maybe someday
I&rsquo;ll get it working. If I do, I doubt the &ldquo;support&rdquo; team will have been
involved. Currently my best bet seems to be waiting for some apparent staff
member (the only apparent staff member) who idles in the IRC channel on Freenode
and allegedly comes online from time to time.</p>
<p>I&rsquo;m not alone in these problems. Here are some (anonymized) quotes I&rsquo;ve heard
from others while trying to troubleshoot this on IRC.</p>
<p>On support:</p>
<blockquote>
<p>ugh, ddevault, yeah. [Blackbird ownership] has not been a smooth experience
for me, either.</p>
</blockquote>
<blockquote>
<p>my personal theory is that they have really bad ticket software that &lsquo;loses&rsquo;
tickets somehow</p>
</blockquote>
<p>On reliability:</p>
<blockquote>
<p>I&rsquo;ve found openbmc&rsquo;s networking to be&hellip; a bit unreliable&hellip; maybe 20% of the
time it does not responed[sic]/does not respond fast enough to networking
requests.</p>
</blockquote>
<blockquote>
<p>yeah the vga handoff failing doesn&rsquo;t surprise me (other people here have
reported it). but the BMC not getting a DHCP lease is odd. (well maybe not
that odd if you look at the crumminess of the OpenBMC software stack&hellip;)</p>
</blockquote>
<p>So, yeah, don&rsquo;t buy from Raptor Computer Systems. It&rsquo;s too large and unwieldly
to be an effective paper weight, either!</p>
<hr>
<p><strong>Erratta</strong></p>
<p><em>2019-09-24 @ 00:19 UTC</em>: Raptor has reached out and apologized for my support
experience. We are discussing these problems in more detail now. They have also
issued a refund for the unshipped RAM.</p>
<p><em>2019-09-24 @ 00:51 UTC</em>: Raptor believes the CPU to be faulty and is shipping a
replacement. They attribute the delay to having to reach out to IBM about the
problem, but don&rsquo;t have a satisfactory answer to why the support process failed.
I understand it&rsquo;s being discussed internally.</p>
<p><em>2019-09-24 @ 13:08 UTC</em>:</p>
<blockquote>
<p>After investigation, we are implementing new mechanisms to avoid support
issues like the one you experienced. We now have a self-serve RMA generation
system which would have significantly reduced your wait time, and are taking
measures to ensure that tickets are no longer able to be ignored by front line
support staff. We believe we have addressed the known failure modes at this
time, and management will be keeping a close eye on the operation of the
support system to ensure that new failure modes are handled rapidly.</p>
</blockquote>
<p>They&rsquo;ve tweeted this about their new self-service RMA system as well:</p>
<blockquote>
<p>We&rsquo;ve made it easy to submit RMA requests for defective products on our Web
site. Simply go to your account, select the &ldquo;Submit RMA Request&rdquo; link, and
fill out the form.  Your product will be warranty checked and, if valid, you
will receive an RMA number and shipping address!</p>
</blockquote>
<p>— @RaptorCompSys via <a href="https://twitter.com/RaptorCompSys/status/1176432946670186498">Twitter</a></p>
<p>I agree that this shows positive improvements and a willingness to continue
making improvements in their support experience. Thanks to Raptor for taking
these concerns seriously. I hope to have a working Blackbird system soon, and
will publish a follow-up review when the time comes.</p>
<p><em>2019-10-08 @ 22:30 UTC</em> A source quoted anonymously in this article asked me to
remove their quote, after a change of heart. They feel that the attention this
article has received has made their statement reach beyond the level of
dissatisfaction they had with Raptor at the time.</p>


guid: https://drewdevault.com/2019/10/07/HDCP-in-Weston.html
title: Why Collabora really added Digital Restrictions Management to Weston
url: https://drewdevault.com/2019/10/07/HDCP-in-Weston.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>A recent article from Collabora, <a href="https://www.collabora.com/news-and-blog/blog/2019/10/03/why-hdcp-support-in-weston-is-a-good-thing/">Why HDCP support in Weston is a good
thing</a>, proports to offer a lot of insight into why
<a href="https://en.wikipedia.org/wiki/High-bandwidth_Digital_Content_Protection">HDCP</a> - a Digital Restrictions Management (DRM) related technology - was
added to <a href="https://gitlab.freedesktop.org/wayland/weston">Weston</a> - a well known basic Wayland compositor which was once
the reference compositor for Wayland. But this article is gaslighting you.
There is one reason and one reason alone that explains why HDCP support landed
in Weston.</p>
<p>Q: Why was HDCP added to Weston?</p>
<p>A: $$$$$</p>
<p>Why does Collabora want you to <em>believe</em> that HDCP support in Weston is a good
thing? Let&rsquo;s look into this in more detail. First: <em>is</em> HDCP a bad thing?</p>
<p>DRM (Digital Restrictions Management) is the collective term for software which
attempts to restrict the rights of users attempting to access digital media.
It&rsquo;s mostly unrelated to Direct Rendering Manager, an important Linux subsystem
for graphics which is closely related to Wayland. Digital Restrictions
Management is software used by media owners to prevent you from enjoying their
content except in specific, pre-prescribed ways.</p>
<p>There is universal agreement among the software community that DRM is
ineffective. Ultimately, these systems are defeated by the simple fact that no
amount of DRM can stop you from pointing your camera at your screen and pushing
record. But in practice, we don&rsquo;t even need to resort to that - these systems
are far too weak to demand such measures. <a href="https://www.amazon.com/HSV321/dp/B07C6KCBYB">Here&rsquo;s a $100 device on Amazon which
can break HDCP</a>. DRM is shown to be impossible even in <em>theory</em>, as the
decryption keys have to live somewhere in your house in order to watch movies
there. Exfiltrating them is just a matter of putting forth the effort.  For most
users, it hardly requires any effort to bypass DRM - they can just punch &ldquo;watch
[name of movie] for free&rdquo; into Google. It&rsquo;s well-understood and rather obvious
that DRM systems completely and entirely fail at their stated goal.</p>
<p>No reasonable engineer would knowingly agree to adding a broken system like that
to their system, and trust me - the entire engineering community has been made
well-aware of these faults. Any other system with these obvious flaws would be
discarded immediately, and if the media industry hadn&rsquo;t had their hands firmly
clapped over their ears, screaming &ldquo;la la la&rdquo;, and throwing money at the
problem, it would have been. But, just adding a broken system isn&rsquo;t necessarily
going to hurt much.  The problem is that, in its failure to achieve its stated
goals, DRM brings with it some serious side-effects. DRM is closely tied to
nonfree software - the RIAA mafia wants to keep their garbage a secret, after
all. Moreover, DRM takes away the freedom to play your media when and where you
want. Why should you have to have an internet connection? Why can&rsquo;t you watch it
on your ancient iPod running Rockbox? DRM exists to restrict users from doing
what they want. More sinisterly, it exists to further the industry&rsquo;s push to
end consumer ownership of its products - preferring to steal from you monthly
subscription fees and lease the media to you. Free software maintainers are
responsible for protecting their users from this kind of abuse, and putting DRM
into our software betrays them.</p>
<p>The authors are of the opinion that HDCP support in Weston does not take away
any rights from users. It doesn&rsquo;t <em>stop</em> you from doing anything. This is true,
in the same way that killing environmental regulations doesn&rsquo;t harm the
environment. Adding HDCP support is handing a bottle of whiskey to an abusive
husband. And the resulting system - and DRM as a whole - is known to be
inherently broken and ineffective, a fact that they even acknowledge in their
article. This feature <em>enables</em> media companies to abuse <em>your</em> users. Enough
cash might help some devs to doublethink their way out of it, but it&rsquo;s true all
the same. They added these features to help abusive companies abuse their users,
in the hopes that they&rsquo;ll send back more money or more patches. They say as much
in the article, it&rsquo;s no secret.</p>
<p>Or, let&rsquo;s give them the benefit of the doubt: perhaps their bosses forced them
to add this<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. There have been other developers on this ledge, and I&rsquo;ve talked
them down. Here&rsquo;s the thing: it worked. Their organizations didn&rsquo;t pursue DRM
any further. You are not the lowly code monkey you may think you are. Engineers
have real power in the organization. You can say &ldquo;no&rdquo; and it&rsquo;s your
responsibility to say &ldquo;no&rdquo; when someone asks you to write unethical code.</p>
<p>Some of the people I&rsquo;ve spoken to about HDCP for Wayland, particularly for
Weston, are of the opinion that &ldquo;a protocol for it exists, therefore we will
implement it&rdquo;. This is reckless and stupid. We already know what happens when
you bend the knee to our DRM overlords: look at Firefox. In 2014, Mozilla
added DRM to Firefox after a year of fighting against its standardization in the
W3C (a <a href="https://en.wikipedia.org/wiki/Regulatory_capture">captured</a> organization which governs<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> web standards). They
capitulated, and it did absolutely nothing to stop them from being steamrolled
by Chrome&rsquo;s growing popularity. Their market-share freefall didn&rsquo;t even slow
down in 2014, or in any year since<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>. Collabora went down without a fight in
the first place.</p>
<p>Anyone who doesn&rsquo;t recognize that self-interested organizations with a great
deal of resources are working against <em>our</em> interests as a free software
community is an idiot. We are at war with the bad actors pushing these systems,
and they are to be <a href="https://en.wikipedia.org/wiki/No_quarter">given no quarter</a>.
Anyone who realizes this and turns a blind eye to it is a coward. Anyone who
doesn&rsquo;t stand up to their boss, sits down, implements it in our free software
ecosystem, and cashes their check the next Friday - is not only a coward, but a
traitor to their users, their peers, and to society as a whole.</p>
<p>&ldquo;HDCP support in Weston is a good thing&rdquo;? It&rsquo;s a good thing for <em>you</em>, maybe.
It&rsquo;s a good thing for media conglomerates which want our ecosystem crushed
underfoot. It&rsquo;s a bad thing for your users, and you know it, Collabora. Shame on
you for gaslighting us.</p>
<p>However&hellip; the person who <em>reverts</em> these changes is a hero, even in the face of
past mistakes. Weston, Collabora, you still have a chance to repent. Do what you
know is right and stand by those principles in the future.</p>
<hr>
<p>P.S. To make sure I&rsquo;m not writing downers all the time, rest assured that the
next article will bring good news - RaptorCS has been working hard to correct
the issues I raised in my last article.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>This is just for the sake of argument. I&rsquo;ve spoken 1-on-1 with some of the developers responsible and they stand by their statements as their personal opinions. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Or at least attempts to govern. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p><a href="https://en.wikipedia.org/wiki/File:StatCounter-browser-ww-monthly-200901-201905.png">Source: StatCounter</a>. Measuring browser market-share is hard, collect your grain of salt <a href="https://en.wikipedia.org/wiki/Usage_share_of_web_browsers">here</a>. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/10/10/RaptorCS-redemption.html
title: RaptorCS's redemption: the POWER9 machine works
url: https://drewdevault.com/2019/10/10/RaptorCS-redemption.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>This is a follow-up to my earlier article, &ldquo;<a href="https://drewdevault.com/2019/09/23/RaptorCS-Blackbird-a-horror-story.html">RaptorCS POWER9 Blackbird PC: An
expensive mistake</a>&rdquo;. Since I published that article, I&rsquo;ve been in
touch with Raptor and they&rsquo;ve been much more communicative and helpful. I now
have a working machine!</p>
<p><img src="https://sr.ht/OTyo.jpeg" alt="Picture of uname -sm showing &ldquo;Linux ppcle64&rdquo;"></p>
<p>After I published my article, Raptor reached out and apologised for my
experience. They offered a full refund, but I agreed to work on further
diagnosis now that we had opened a dialogue<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. They identified that my CPU was
defective and sent me a replacement, then we found the mainboard to be
defective, too, and the whole thing was shipped back and replaced. I installed
the new hardware into the datacenter today and it was quite pleasant to get up
and running. Raptor assures me that my nightmares with the old board are
atypical, and if the new board is representative of the usual user experience, I
would have to agree. The installation was completely painless.<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup></p>
<p>However, I refuse to give any company credit for waking up their support team
only when a scathing article about them frontpages on Hacker News. I told them I
wouldn&rsquo;t publish a positive follow-up unless they also convinced me that the
support experience had been fixed for the typical user as well. To this end,
Raptor has made a number of substantive changes. To quote their support staff:</p>
<blockquote>
<p>After investigation, we are implementing new mechanisms to avoid support
issues like the one you experienced. We now have a
<a href="https://twitter.com/RaptorCompSys/status/1176432946670186498">self-serve RMA generation system</a>
which would have significantly reduced your wait time, and are taking measures
to ensure that tickets are no longer able to be ignored by front line support
staff. We believe we have addressed the known failure modes at this time, and
management will be keeping a close eye on the operation of the support system
to ensure that new failure modes are handled rapidly.</p>
</blockquote>
<p>They&rsquo;ve tweeted this about their new self-service RMA system as well:</p>
<blockquote>
<p>We&rsquo;ve made it easy to submit RMA requests for defective products on our Web
site. Simply go to your account, select the &ldquo;Submit RMA Request&rdquo; link, and
fill out the form.  Your product will be warranty checked and, if valid, you
will receive an RMA number and shipping address!</p>
</blockquote>
<p>— @RaptorCompSys via <a href="https://twitter.com/RaptorCompSys/status/1176432946670186498">Twitter</a></p>
<p>They&rsquo;re also working on other improvements to make the end-user experience
better, including <a href="https://wiki.raptorcs.com/wiki/Main_Page">more content on the
wiki</a>, such as a <a href="https://wiki.raptorcs.com/wiki/Troubleshooting/Support_Request_Checklist">flowchart for
dealing with common
problems</a>.</p>
<p>Thanks to Raptor for taking the problem seriously, quickly fixing the problems
with my board, and for addressing the systemic problems which led to the
failure of their support system.</p>
<p>On the subject of the working machine, I am quite impressed with it so far.
Installation was a breeze, it compiles the kernel on 32 threads from spinning
rust in 4m15s, and I was able to get KVM working without much effort. I have
christened it &ldquo;flandre&rdquo;<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>, which I think is fitting. I plan on bringing it up
as a build slave for builds.sr.ht in the coming weeks/months, and offering
ppc64le builds on Sourcehut in the near future. I have another board which was
generously donated by another Raptor customer<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>, which arrived last week and
that I hope to bring up and use for testing Wayland before introducing it to the
Sourcehut fleet.</p>
<hr>
<p>P.S. For those interested in more details of the actual failures:</p>
<p>This machine is so badly broken that it would actually be hilarious if the
manufacturer had been more present in the troubleshooting process. I think the
best way to sum it up is &ldquo;FUBAR&rdquo;. Among problems I encountered were:</p>
<ul>
<li>The CPU experiences a &ldquo;ZCAL failure&rdquo; (???)</li>
<li>The BMC (responsible for bringing up the main CPU(s)) had broken ethernet,
making login over SSH impossible</li>
<li>The BMC&rsquo;s getty would boot loop, making login over serial impossible</li>
<li>The BMC&rsquo;s u-Boot would boot loop if the TX pin on the serial cable was plugged
in, making diagnosing issues from that stage impossible</li>
<li>petitboot&rsquo;s ncurses output was being piped into a shell and executed (what the fuck?)</li>
</ul>
<p>In the immortal words of James Mickens, &ldquo;I HAVE NO TOOLS BECAUSE I HAVE
DESTROYED MY TOOLS WITH MY TOOLS.&rdquo; A staff member at Raptor tells me:
&ldquo;Your box ended up on my desk [&hellip;] This is easily the most broken board I&rsquo;ve
seen, ever, and that includes prototypes. This will help educate us for a while
to come due to the unique nature of some of the faults.&rdquo;</p>
<p>Not sure what can cause such an impressive cacophony of failures, but it&rsquo;s so
catastrophic that I can easily believe that this is far from typical. The
hardware is back in Raptor&rsquo;s hands now, and I would be interested to hear about
their insights after further diagnosis.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>They did refund the RAM which was unfulfilled from my original order. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>They did give me a little heart attack, however, by sending the replacement CPU to me in the same box I had returned the faulty CPU back to them with - a box which I had labelled &ldquo;BAD CPU&rdquo;. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Sourcehut virtual machines are named after their purpose, but our physical servers are named after <a href="https://en.wikipedia.org/wiki/Touhou_Project">Touhou</a> characters. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>This happened prior to any of the problems with the first machine. <a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/10/12/how-to-fuck-up-releases.html
title: How to fuck up software releases
url: https://drewdevault.com/2019/10/12/how-to-fuck-up-releases.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I manage releases for a bunch of free &amp; open-source software. Just about every
time I ship a release, I find a novel way to fuck it up. Enough of these
fuck-ups have accumulated now that I wanted to share some of my mistakes and how
I (try to) prevent them from happening twice.</p>
<p>At first, I did everything manually. This is fine enough for stuff with simple
release processes - stuff that basically amounts to tagging a commit, pushing
it, and calling it a day. But even this gets tedious, and I&rsquo;d often make a
mistake when picking the correct version number. So, I wrote a small script:
<a href="https://git.sr.ht/~sircmpwn/dotfiles/tree/master/bin/semver">semver</a>. <code>semver patch</code> bumps the patch version, <code>semver minor</code> bumps the minor version, and
<code>semver major</code> bumps the major version, based on semantic versioning. I got into
the habit of using this script instead of making the tags manually. The next
fuckup soon presented itself: when preparing the
<a href="https://git-scm.com/docs/git-shortlog">shortlog</a>, I would often feed it the
wrong commits, and the changelog would be messed up. So, I updated the script to
run the appropriate shortlog command and pre-populate the annotated tag with it,
launching the editor to adjust the changelog as necessary.</p>
<p>Soon I wanted to apply this script to other projects, but not all of them used
semantic versioning. I updated it to work for projects which just use
<code>major.minor</code> versions as well. However, another problem arose: some projects
have the version number specified in the Makefile or meson.build. I would
frequently fuck this up in many creative ways: forgetting it entirely; updating
it but not committing it; updating it and committing it, but tagging the wrong
commit; etc. <a href="https://github.com/swaywm/wlroots">wlroots</a> in particular was
difficult because I also had to update the soversion, which had special
requirements. To address these issues, I added a custom <code>.git/_incr_version</code>
script which can add additional logic on a per-repo basis, and updated semver to
call this script if present.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p>
<p>Eventually, I went on vacation and shipped a release while I was there. The
<code>_incr_version</code> script I had put into <code>.git</code> on my home workstation wasn&rsquo;t
checked into version control and didn&rsquo;t come with me on vacation, leading to yet
another fucked up release. I moved it from <code>.git/_incr_version</code> to
<code>contrib/_incr_version</code>. I made the mistake, however, of leaving the old path in
as a fallback, which meant that I never noticed that <em>another</em> project&rsquo;s script
was still in <code>.git</code> until I went on another vacation and fucked up another
release. Add a warning which detects if the script is at the old path&hellip;</p>
<p>Some of my projects don&rsquo;t use semantic versioning at all, but still have all of
these other gotchas, so I added an option to just override the automatic version
increment with a user-specified override. For a while, this worked well. But,
inevitably, no matter how much I scripted away my mistakes I would always find a
new and novel way of screwing up. The next one came when I shipped a release
while on an Alpine Linux machine, which ships Busybox instead of GNU tools.
Turns out Busybox gzip produces output which does not match the GNU output,
which means the tarballs I signed locally differed from the ones generated by
Github. Update the signing script to save the tarball to disk (previously,
it lived in a pipe) and upload these alongside the releases&hellip;<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup></p>
<p>Surely, there are no additional ways to fuck it up at this point. I must have
every base covered, right? Wrong. Dead wrong. On the very next release I
shipped, I mistakenly did everything from a feature branch, and shipped
experimental, incomplete code in a stable release. Update the script to warn if
the master branch isn&rsquo;t checked out&hellip; Then, of course, another fuckup: I tagged
a release without pulling first, and when I pushed, git happily rejected my
branch and accepted the tag - shipping an outdated commit as the release. Update
the script to <code>git pull</code> first&hellip;</p>
<p>I am doomed to creatively outsmart my tools in releases. If you&rsquo;d like to save
yourself from some of the mistakes I&rsquo;ve made, you can <a href="https://git.sr.ht/~sircmpwn/dotfiles/tree/master/bin/semver">find my semver script
here</a>.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Each of these <code>_incr_version</code> scripts proved to have many bugs of their own, of course. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Eli Schwartz of Arch Linux also sent a patch to Busybox which made their gzip implementation consistent with GNU&rsquo;s. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/10/15/Status-update-October-2019.html
title: Status update, October 2019
url: https://drewdevault.com/2019/10/15/Status-update-October-2019.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Last month, I gave you an update at the conclusion of a long series of travels.
But, I wasn&rsquo;t done yet - this month, I spent a week in Montreal for <a href="https://xdc2019.x.org/">XDC</a>.
Simon Ser put up <a href="https://emersion.fr/blog/2019/xdc2019-wrap-up/">a great write-up</a> which goes over a lot of the
important things we discussed there. It was a wonderful conference and well
worth the trip - but I truly am sick of travelling. Now, I can enjoy some time
at home, working on free and open source software.</p>
<p>I have a video to share today, of a workflow on git.sr.ht that I&rsquo;m very excited
about: sending patchsets as emails from the web.</p>
<video src="https://sr.ht/_fUk.webm" controls muted>
  Your web browser does not support the webm video codec. Please consider using
  web browsers that support free and open standards.
</video>
<p>Sourcehut&rsquo;s development plans can be described in three broad strokes: (1) make
a bunch of services (or: primitives for a development hub); (2) rig them all up
with APIs and webhooks; and (3) teach them how to talk to each other. Over the
past year, (1) and (2) are mostly complete, and (3) is now underway. Teaching
git.sr.ht and lists.sr.ht to talk to each other is an important step, because it
will give us a web-based code review flow which is backed by emails. This meets
an original design goal of Sourcehut: to build user-friendly tools on top of
existing systems.</p>
<p>The other end of this work is on lists.sr.ht, but for now it&rsquo;s indirect: I&rsquo;ve
also been working on <a href="https://github.com/libgit2/pygit2/pulls?q=is%3Apr+author%3Addevault+is%3Aclosed">pygit2</a> fleshing out the Odb backend API, so
that I can make a pygit2 repo which is backed by the git.sr.ht API. From there,
it&rsquo;ll be easy to teach lists.sr.ht about git.sr.ht - and perhaps other git
services as well.</p>
<p>There&rsquo;s also a fourth stage of Sourcehut: giving back to the free software
community. To this end, I intend to spend Sourcehut&rsquo;s profits on sponsoring
motivated and talented free software developers to work on self-directed
projects. I&rsquo;m very excited to announce that there&rsquo;s progress here as well:
<a href="https://emersion.fr">Simon Ser</a> is now joining Sourcehut and will be doing just
that: self-directed free software projects. He&rsquo;s written more about this on <a href="https://emersion.fr/blog/2019/working-full-time-on-open-source/">his
blog</a> and I&rsquo;ll
be writing more on <a href="https://sourcehut.org">sourcehut.org</a> later.</p>
<p>Wrapping up Sourcehut news, I&rsquo;ll leave you with an out-of-context screenshot of
a mockup I made this month:</p>
<p><a href="https://sr.ht/_yhw.png"><img src="https://sr.ht/_yhw.png" alt="Screenshot of a Sourcehut DNS service showing DNS records managed by zonefiles in a git repository"></a></p>
<p>Let&rsquo;s move on to Wayland news. We&rsquo;ve started the planning for the next sway
release, and it&rsquo;s shaping up to be really cool. We expect to ship patches which
can reduce input latency to as low as 1ms, introduce the foreign toplevel
management protocol for better mate-panel support, and introduce damage tracking
to our screencopy protocol - which is being used to make a VNC server for
sway and other wlroots-based compositors; and proper drawing tablet support.
We&rsquo;re also making strong headway on a long-term project to overhaul rendering
and DRM in wlroots, with the long term goal of achieving the holy grail levels
of performance on any device.</p>
<p><a href="https://wayland-book.com">The Wayland book</a> is also in good shape. A lot of
people have purchased the drafts - over a hundred! Thank you for picking it up,
and please send your feedback along. I completed chapter 8 this month. I also
expect to receive the last few parts for my second POWER9 machine today, and I
plan on using this to test Wayland, Mesa, etc - on ppc64le. The <a href="https://drewdevault.com/2019/10/10/RaptorCS-redemption.html">first POWER9
machine</a> is now provisioned and humming along in the Sourcehut
datacenter, by the way.</p>
<p>VR work has also been chugging along again this month. I&rsquo;ve started contributing
to <a href="https://gitlab.freedesktop.org/monado/monado/merge_requests?scope=all&amp;utf8=%E2%9C%93&amp;state=merged&amp;author_username=ddevault">Monado</a>, which is basically to OpenXR as Mesa is to OpenGL. I&rsquo;ve
seen merged an overhaul to their build system, an overhaul for their dated
Wayland backend, and even some deeper work ensuring conformance with the OpenXR
specification. A lot of this work has also been in getting to know everyone and
planning the future of the project, as it&rsquo;s still in early stages.</p>
<p>To quickly summarize my other various projects:</p>
<ul>
<li><strong>ctools</strong> has seen many small improvements and bug fixes, and has grown the
dirname, rmdir, env, and sleep utilities.</li>
<li><strong>aerc</strong> has also seen small improvements and bug fixes, but has also learned
about sorting and will soon grow a threaded message list</li>
<li><strong>chopsui</strong> is stirring in its sleep, and I&rsquo;ve been giving some new attention
to its design problems in the hopes that the next iteration will be the
correct design for a new GUI toolkit.</li>
<li><a href="https://git.sr.ht/~sircmpwn/wshowkeys"><strong>wshowkeys</strong></a> is a new little tool I
built to display your keypresses on-screen during a Wayland session, useful
for live streaming or video recording</li>
<li><strong>9front</strong> has been eating some of my evenings lately, and I&rsquo;ve been making
small improvements to various tools and improving Plan 9 support among some
packages in the Go ecosystem. I have more plans for this&hellip; stay tuned.</li>
</ul>
<p>That&rsquo;s all I&rsquo;ve got for today. Thank you for your support! Oh, and one last
note: I&rsquo;ve been invited to the <a href="https://github.com/users/ddevault/sponsorship">Github sponsors
program</a>, so if you want to
donate through it Github will match your donation for a little while. Cheers!</p>


guid: https://drewdevault.com/2019/10/30/Line-printer-shell-hack.html
title: An old-school shell hack on a line printer
url: https://drewdevault.com/2019/10/30/Line-printer-shell-hack.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>It&rsquo;s been too long since I last did a good hack, for no practical reason other
than great hack value. In my case, these <a href="https://drewdevault.com/2016/03/22/Integrating-a-VT220-into-my-life.html">often amount</a> to a nostalgia
for an age of computing I wasn&rsquo;t present for. In a recent bid to capture more of
this nostalgia, I recently picked up a dot matrix line printer, specifically the
Epson LX-350 printer. This one is nice because it has a USB port, so I don&rsquo;t
have to break out my pile of serial cable hacks to get it talking to Linux 😁</p>
<p>This is the classic printer style, with infinite paper and a lovely noise during
printing. They are also fairly simple to operate - you can just write text
directly to <code>/dev/lp</code> (or <code>/dev/usb/lp9</code> in my case) and it&rsquo;ll print it out.
Slightly more sophisticated instructions can be written to them with ANSI escape
sequences, just like a terminal. They can also be rigged up to CUPS, then you
can use something like <code>man -t 5 scdoc</code> to produce printouts like this:</p>
<p><a href="https://sr.ht/gHCA.jpg"><img src="https://sr.ht/gHCA.jpg" alt=""></a></p>
<p>Plugging the printer into Linux and writing out pages isn&rsquo;t much for hack value,
however. What I really wanted to make was something resembling an old-school
TTY - teletypewriter. So I wrote some <a href="https://git.sr.ht/~sircmpwn/lpsh">glue code in
Golang</a>, and soon enough I had a shell:</p>
<iframe width="560" height="315" sandbox="allow-same-origin allow-scripts
allow-popups"
src="https://spacepub.space/videos/embed/d8943b2d-8280-497b-85ec-bc282ec2afdc"
frameborder="0" allowfullscreen style="width: 100%"></iframe>
<p>The glue code I wrote for this is fairly straightforward. In the simplest form,
it spins up a pty (pseudo-terminal), runs <code>/bin/sh</code> in it, and writes the pty
output into the line printer device. For those unaware, a pseudo-terminal is the
key piece of software infrastructure for running interactive text applications.
Applications which want to do things like print colored text, move
the cursor around and draw a TUI, and so on, will open <code>/dev/tty</code> to open the
current TTY device. For most applications used today, this is a
&ldquo;pseudo-terminal&rdquo;, or pty, which is a terminal emulated in userspace - i.e. by
your terminal emulator. However, your terminal emulator is <em>emulating</em> a
terminal - the control sequences applications send to these are
backwards-compatible with 50 years of computing history.  Interfaces like these
are the namesake of the TTY.</p>
<p>Visual terminals came onto the scene later on, and in the classic computing
tradition, the old hands complained that it was less useful - you could no
longer write notes on your backlog, tear off a page and hand it to a colleague,
or <a href="https://en.wikipedia.org/wiki/Wite-Out">white-out</a> mistakes. Early
<a href="https://en.wikipedia.org/wiki/Computer_terminal">visual terminals</a> could also
be plugged directly into a line printer, and you could configure them to echo to
the printer or print out a screenfull of text at a time. A distinct advantage of
visual terminals is not having to deal with so much bloody paper, a problem that
I&rsquo;ve become acutely familiar with in the past few days<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</p>
<p>Getting back to the glue code, I chose Golang because setting up a TTY is a bit
of a hassle in C, but in Golang it&rsquo;s pretty straightforward. There is a serial
port and in theory I could have plugged it in and spawned a getty on the
resulting serial device - but (1) it&rsquo;d be write-only, so not especially
interactive without <em>hardware</em> hacks, and (2) I didn&rsquo;t feel like digging out my
serial cables. So:</p>
<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kn">import</span> <span class="s">&#34;git.sr.ht/~sircmpwn/pty&#34;</span> <span class="c1">// fork of github.com/kr/pty
</span><span class="c1"></span>
<span class="c1">// ...
</span><span class="c1"></span><span class="nx">winsize</span> <span class="o">:=</span> <span class="nx">pty</span><span class="p">.</span><span class="nx">Winsize</span><span class="p">{</span>
  <span class="nx">Cols</span><span class="p">:</span> <span class="mi">160</span><span class="p">,</span>
  <span class="nx">Rows</span><span class="p">:</span> <span class="mi">24</span><span class="p">,</span>
<span class="p">}</span>
<span class="nx">cmd</span> <span class="o">:=</span> <span class="nx">exec</span><span class="p">.</span><span class="nf">Command</span><span class="p">(</span><span class="s">&#34;/bin/sh&#34;</span><span class="p">)</span>
<span class="nx">cmd</span><span class="p">.</span><span class="nx">Env</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">os</span><span class="p">.</span><span class="nf">Environ</span><span class="p">(),</span>
  <span class="s">&#34;TERM=lp&#34;</span><span class="p">,</span>
  <span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;COLUMNS=%d&#34;</span><span class="p">,</span> <span class="mi">180</span><span class="p">))</span>
<span class="nx">tty</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">pty</span><span class="p">.</span><span class="nf">StartWithSize</span><span class="p">(</span><span class="nx">cmd</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">winsize</span><span class="p">)</span>
</code></pre></div><p><em>P.S. We&rsquo;re going to dive through the code in detail now. If you just want more
cool videos of this in action, skip to the bottom.</em></p>
<p>I set the TERM environment variable to <code>lp</code>, for line printer, which doesn&rsquo;t
really exist but prevents most applications from trying anything too tricksy
with their escape codes. The <code>tty</code> variable here is an <code>io.ReadWriter</code> whose
output is sent to the printer and whose input is sourced from wherever, in my
case from the stdin of this process<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>.</p>
<p>For a little more quality-of-life, I looked up Epson&rsquo;s proprietary ANSI escape
sequences and found out that you can tell the printer to feed back and forth in
216th&quot; increments with the j and J escape sequences. The following code will
feed 2.5&quot; out, then back in:</p>
<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">f</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;\x1BJ\xD8\x1BJ\xD8\x1BJ\x6C&#34;</span><span class="p">))</span>
<span class="nx">f</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;\x1Bj\xD8\x1Bj\xD8\x1Bj\x6C&#34;</span><span class="p">))</span>
</code></pre></div><p>Which happens to be the perfect amount to move the last-written line up out of
the printer for the user to read, then back in to be written to some more. A
little bit of timing logic in a goroutine manages the transition between &ldquo;spool
out so the user can read the output&rdquo; and &ldquo;spool in to write some more output&rdquo;:</p>
<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">func</span> <span class="nf">lpmgr</span><span class="p">(</span><span class="nx">in</span> <span class="kd">chan</span> <span class="p">(</span><span class="kd">interface</span><span class="p">{}),</span> <span class="nx">out</span> <span class="kd">chan</span> <span class="p">([]</span><span class="kt">byte</span><span class="p">))</span> <span class="p">{</span>
	<span class="c1">// TODO: Runtime configurable option? Discover printers? dunno
</span><span class="c1"></span>	<span class="nx">f</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">os</span><span class="p">.</span><span class="nf">OpenFile</span><span class="p">(</span><span class="s">&#34;/dev/usb/lp9&#34;</span><span class="p">,</span> <span class="nx">os</span><span class="p">.</span><span class="nx">O_RDWR</span><span class="p">,</span> <span class="mo">0755</span><span class="p">)</span>
	<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
		<span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
	<span class="p">}</span>

	<span class="nx">feed</span> <span class="o">:=</span> <span class="kc">false</span>
	<span class="nx">f</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;\n\n\n\r&#34;</span><span class="p">))</span>

	<span class="nx">timeout</span> <span class="o">:=</span> <span class="mi">250</span> <span class="o">*</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Millisecond</span>
	<span class="k">for</span> <span class="p">{</span>
		<span class="k">select</span> <span class="p">{</span>
		<span class="k">case</span> <span class="o">&lt;-</span><span class="nx">in</span><span class="p">:</span>
			<span class="c1">// Increase the timeout after input
</span><span class="c1"></span>			<span class="nx">timeout</span> <span class="p">=</span> <span class="mi">1</span> <span class="o">*</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Second</span>
		<span class="k">case</span> <span class="nx">data</span> <span class="o">:=</span> <span class="o">&lt;-</span><span class="nx">out</span><span class="p">:</span>
			<span class="k">if</span> <span class="nx">feed</span> <span class="p">{</span>
				<span class="nx">f</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;\x1Bj\xD8\x1Bj\xD8\x1Bj\x6C&#34;</span><span class="p">))</span>
				<span class="nx">feed</span> <span class="p">=</span> <span class="kc">false</span>
			<span class="p">}</span>
			<span class="nx">f</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nf">lptl</span><span class="p">(</span><span class="nx">data</span><span class="p">))</span>
		<span class="k">case</span> <span class="o">&lt;-</span><span class="nx">time</span><span class="p">.</span><span class="nf">After</span><span class="p">(</span><span class="nx">timeout</span><span class="p">):</span>
			<span class="nx">timeout</span> <span class="p">=</span> <span class="mi">200</span> <span class="o">*</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Millisecond</span>
			<span class="k">if</span> <span class="p">!</span><span class="nx">feed</span> <span class="p">{</span>
				<span class="nx">feed</span> <span class="p">=</span> <span class="kc">true</span>
				<span class="nx">f</span><span class="p">.</span><span class="nf">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">&#34;\x1BJ\xD8\x1BJ\xD8\x1BJ\x6C&#34;</span><span class="p">))</span>
			<span class="p">}</span>
		<span class="p">}</span>
	<span class="p">}</span>
<span class="p">}</span>
</code></pre></div><p><code>lptl</code> is a work-in-progress thing which tweaks the outgoing data for some
quality-of-life changes, like changing backspace to ^H. Then, the main event
loop looks something like this:</p>
<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="nx">inch</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="p">(</span><span class="kd">interface</span><span class="p">{}))</span>
<span class="nx">outch</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="p">([]</span><span class="kt">byte</span><span class="p">))</span>
<span class="k">go</span> <span class="nf">lpmgr</span><span class="p">(</span><span class="nx">inch</span><span class="p">,</span> <span class="nx">outch</span><span class="p">)</span>

<span class="nx">inbuf</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="mi">4096</span><span class="p">)</span>
<span class="k">go</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span>
  <span class="k">for</span> <span class="p">{</span>
    <span class="nx">n</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">os</span><span class="p">.</span><span class="nx">Stdin</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">inbuf</span><span class="p">)</span>
    <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
      <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
    <span class="p">}</span>
    <span class="nx">tty</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nx">inbuf</span><span class="p">[:</span><span class="nx">n</span><span class="p">])</span>
    <span class="nx">inch</span> <span class="o">&lt;-</span> <span class="kc">nil</span>
  <span class="p">}</span>
<span class="p">}()</span>

<span class="nx">outbuf</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="mi">4096</span><span class="p">)</span>
<span class="k">for</span> <span class="p">{</span>
  <span class="nx">n</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">tty</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">outbuf</span><span class="p">)</span>
  <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
    <span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span>
  <span class="p">}</span>
  <span class="nx">b</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="nx">n</span><span class="p">)</span>
  <span class="nb">copy</span><span class="p">(</span><span class="nx">b</span><span class="p">,</span> <span class="nx">outbuf</span><span class="p">[:</span><span class="nx">n</span><span class="p">])</span>
  <span class="nx">outch</span> <span class="o">&lt;-</span> <span class="nx">b</span>
<span class="p">}</span>
</code></pre></div><p>The tty will echo characters written to it, so we just write to it from stdin
and increase the form feed timeout closer to the user&rsquo;s input so that it&rsquo;s not
constantly feeding in and out as you write. The resulting system is pretty
pleasant to use! I spent about hour working on improvements to it on a <a href="https://live.drewdevault.com">live
stream</a>. You can watch the system in action on the
archive here:</p>
<iframe width="560" height="370" sandbox="allow-same-origin allow-scripts"
src="https://spacepub.space/videos/embed/a8be6c87-9267-452e-8d3e-dd206880fa98"
frameborder="0" allowfullscreen style="width: 100%"></iframe>
<p>If you were a fly on the wall when Unix was written, it would have looked a lot
like this. And remember: <a href="https://www.gnu.org/fun/jokes/ed-msg.html">ed is the standard text
editor</a>.</p>
<p>?</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Don&rsquo;t worry, I recycled it all. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>In the future I want to make this use libinput or something, or eventually make a kernel module which lets you pair a USB keyboard with a line printer to make a TTY directly. Or maybe a little microcontroller which translates a USB keyboard into serial TX and forwards RX to the printer. Possibilities! <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/11/15/Status-update-November-2019.html
title: Status update, November 2019
url: https://drewdevault.com/2019/11/15/Status-update-November-2019.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Today&rsquo;s update is especially exciting, because today marks the 1 year
anniversary of Sourcehut <a href="https://drewdevault.com/2018/11/15/sr.ht-general-availability.html">opening it&rsquo;s alpha</a> to public
registration. I wrote a <a href="https://sourcehut.org/blog/2019-11-15-sourcehut-1-year-alpha/">nice long article</a> which goes into
detail about what Sourcehut accomplished in 2019, what&rsquo;s to come for 2020, and
it lays out the entire master plan for your consideration. Be sure to give that
a look if you have the time. I haven&rsquo;t slowed down on my other projects, though,
so here&rsquo;re some more updates!</p>
<p>I&rsquo;ve been pushing hard on the VR work this month, with lots of help from Simon
Ser. We&rsquo;ve put together <a href="https://git.sr.ht/~sircmpwn/wxrc">wxrc</a> - Wayland XR
Compositor - which does what it says on the tin. It&rsquo;s similar to what you&rsquo;ve
seen in my earlier updates, but it&rsquo;s a bespoke C project instead of a
Godot-based compositor, resulting in something much lighter weight and more
efficient. The other advantage is that it&rsquo;s based on OpenXR, thanks to <a href="https://gitlab.freedesktop.org/monado/monado/merge_requests?scope=all&amp;utf8=%E2%9C%93&amp;state=all&amp;author_username=ddevault">our
many</a> <a href="https://gitlab.freedesktop.org/monado/monado/merge_requests?scope=all&amp;utf8=%E2%9C%93&amp;state=all&amp;author_username=emersion">contributions</a> to Monado, an open-source
OpenXR runtime - the previous incarnations were based on SteamVR, which is
a proprietary runtime and proprietary API. We&rsquo;ve also got 3D Wayland clients
working as of this week, check out our video:</p>
<video src="https://yukari.sr.ht/wxrc-demo3.webm" muted autoplay loop>
  Your web browser does not support the webm video codec. Please consider using
  web browsers that support free and open standards.
</video>
<p>This work has generated more patches for a large variety of projects - Mesa,
Wayland, Xorg, wlroots, sway, new Vulkan and OpenXR standards, and more. This
is really cross-cutting work and we&rsquo;re making improvements across the whole
graphics ecosystem to support it.</p>
<p>Speaking of Wayland, the upcoming Sway release is looking like it&rsquo;s going to be
really good. I mentioned this last month, but we&rsquo;re still on track for getting
lots of great features in - VNC support, foreign toplevel management (taskbars),
input latency reductions, drawing tablet support, and more. I&rsquo;m pretty excited.
I wrote chapters 9 and 9.1 for the Wayland book this month as well.</p>
<p>In aerc news, thanks entirely to its contributors and not to me, lots of new
features have been making their way in. Message templates are one of them, which
you can take advantage of to customize the reply and forwarded message
templates, or make new templates of your own. aerc has learned AUTH LOGIN
support as well, and received a number of bugfixes. ctools has also seen a
number of patches coming in, including support for echo, tee, and nohup, along
with several bug fixes.</p>
<p>In totally off-the-wall news, I&rsquo;ve <a href="https://drewdevault.com/japanese.html">started a page</a> cataloguing my
tools and recommendations for Japanese language learners.</p>
<p>That&rsquo;s all I&rsquo;ve got for you today, I hope it&rsquo;s enough! Thank you for your
continued love and support, I&rsquo;m really proud to be able to work on these
projects for you.</p>


guid: https://drewdevault.com/2019/11/20/China.html
title: China
url: https://drewdevault.com/2019/11/20/China.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>This article will be difficult to read and was difficult to write. I hope that
you can stomach the uncomfortable nature of this topic and read my thoughts in
earnest.  I usually focus on technology-related content, but at the end of the
day, this is my personal blog and I feel that it would betray my personal
principles to remain silent. I&rsquo;ve made an effort to provide citations for all of
my assertions.</p>
<p><em>Note: if you are interested in conducting an independent review of the
factuality of the claims expressed in this article, please <a href="mailto:sir@cmpwn.com">contact
me</a>.</em></p>
<p>The keyboard I&rsquo;m typing these words into bears &ldquo;Made in China&rdquo; on the bottom.
The same is true of the monitor I&rsquo;m using to edit the article. It&rsquo;s not true of
all of my electronics — the graphics processing unit which is driving the
monitor was made in Taiwan<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> and my phone was made in
Vietnam.<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> Regardless, there&rsquo;s no doubt that my life would be,
to some degree, worse off if not for trade with China. Despite this, I am
prepared to accept the consequences of severing economic relations with China.</p>
<p>How bad would being cut-off from China&rsquo;s economy be? We&rsquo;re a net importer from
China, and by over 4 times the volume.<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> Let&rsquo;s assume, in
the worst case, trade ties were completely severed. The United States would be
unable to buy $155B worth of electronics, which we already have domestic
manufacturing capabilities for<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup> and which have a productive
life of several years. We could definitely stand to get used to repairing and
reusing these instead of throwing them out. We&rsquo;d lose $34B in mattresses and
furniture — same story. The bulk of our imports from China are luxury
goods that we can already make here at home<sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup> — it&rsquo;s just
cheaper to buy them from China. But cheaper for whom?</p>
<p>This gets at the heart of the reason why we&rsquo;re tied to China economically. It&rsquo;s
economically productive <em>for the 1%</em> to maintain a trade relationship with
China. The financial incentives don&rsquo;t help any Americans, and in fact, most of
us are hurt by this relationship.<sup id="fnref:6"><a href="#fn:6" class="footnote-ref" role="doc-noteref">6</a></sup> Trade is what keeps
us shackled to the Chinese Communist Party government, but it&rsquo;s not beneficial
to anyone but those who are already obscenely rich, and certainly not for our
poorest — who, going into 2020, are as likely to be high school dropouts
as they are to be doctors.<sup id="fnref:7"><a href="#fn:7" class="footnote-ref" role="doc-noteref">7</a></sup></p>
<p>So, we can cut off China. Why should we? Let&rsquo;s lay out the facts: China is
conducting human rights violations on the largest scale the world has seen since
Nazi Germany. China executes political prisoners<sup id="fnref:8"><a href="#fn:8" class="footnote-ref" role="doc-noteref">8</a></sup> and
harvests their organs for transplant to sick elites on an industrial scale,
targeting and killing civilians based on not only political, but also ethnic and
religious factors. This is commonly known as genocide. China denies using the
organs of prisoners, but there&rsquo;s credible doubt<sup id="fnref:9"><a href="#fn:9" class="footnote-ref" role="doc-noteref">9</a></sup> from
the scientific community.</p>
<p>Recent evidence directly connecting executions to organ harvesting is somewhat
unreliable, but I don&rsquo;t think China deserves the benefit of the doubt.
China is a world leader in executions, and is believed to conduct more
executions than the rest of the world combined.<sup id="fnref:10"><a href="#fn:10" class="footnote-ref" role="doc-noteref">10</a></sup>
Wait times for organ transplantation are extraordinarily low in
China,<sup id="fnref:11"><a href="#fn:11" class="footnote-ref" role="doc-noteref">11</a></sup> on the order of weeks — in most of the developed
world these timeframes are measured in terms of years,<sup id="fnref:12"><a href="#fn:12" class="footnote-ref" role="doc-noteref">12</a></sup> and China has
been unable to explain the source for tens of thousands of transplants in the
past<sup id="fnref:13"><a href="#fn:13" class="footnote-ref" role="doc-noteref">13</a></sup>. And, looking past recent evidence, China directly admitted to
using the organs of executed prisoners in 2005.<sup id="fnref:14"><a href="#fn:14" class="footnote-ref" role="doc-noteref">14</a></sup></p>
<p>These atrocities are being committed against cultural minorities to further
China&rsquo;s power. The UN published a statement in August 2018 stating that they
have credible reports of over a million ethnic Uighurs being held in internment
camps in Xinjiang,<sup id="fnref:15"><a href="#fn:15" class="footnote-ref" role="doc-noteref">15</a></sup> imprisoned with various other ethnic
minorities from the region. Leaks in November 2019 reported by the New York
Times showed that China admits the imprisoned have committed no crimes other
than dissent,<sup id="fnref:16"><a href="#fn:16" class="footnote-ref" role="doc-noteref">16</a></sup> and that the camps were to be run with, quote,
&ldquo;absolutely no mercy&rdquo;.</p>
<p>It&rsquo;s nice to believe that we would have stood up to Nazi Germany if we had been
there in the 1940&rsquo;s. China is our generation&rsquo;s chance to prove ourselves of that
conviction. We talk a big game about fighting against white nationalists in our
own country, and pride ourselves on standing up against &ldquo;fascists&rdquo;. It&rsquo;s time we
turned attention to the real fascists, on the world stage.</p>
<p>Instead, the staunch capitalism of America, and the West as a whole, has swooped
in to leverage Chinese fascism for a profit. Marriott Hotels apologized for
listing Hong Kong, Macau, and Taiwan as countries separate from China.<sup id="fnref:17"><a href="#fn:17" class="footnote-ref" role="doc-noteref">17</a></sup>
Apple removed the Taiwanese flag from iOS in China and the territories it
claims.<sup id="fnref:18"><a href="#fn:18" class="footnote-ref" role="doc-noteref">18</a></sup> Activision/Blizzard banned several players for making pro-Hong
Kong statements in tournaments and online.<sup id="fnref:19"><a href="#fn:19" class="footnote-ref" role="doc-noteref">19</a></sup> These behaviors make me
ashamed to be an American.</p>
<p>Fuck that.</p>
<p>A brief history lesson: Hong Kong was originally controlled by the United
Kingdom at the end of the Opium Wars. It&rsquo;s beyond the scope of this article, but
it&rsquo;ll suffice to say that the United Kingdom was brutal and out of line, and the
end result is that Hong Kong became a British colony. Because of this, it was
protected from direct Chinese influence during China&rsquo;s turbulent years
following, and they were insulated from the effects of the Great Leap Forward
and the Cultural Revolution, which together claimed tens of millions of lives
and secured the Communist Party of China&rsquo;s power into the present.</p>
<p>On July 1st, 1997, the <a href="https://en.wikipedia.org/wiki/Sino-British_Joint_Declaration">Sino-British Joint
Declaration</a> <a href="https://www.youtube.com/watch?v=k7YzJzq1Mvk">went
into effect</a>, and Hong Kong was
turned over to China. The agreement stipulated that Hong Kong would remain
effectively autonomous and self-governing for a period of 50 years —
until 2047. China has been gradually and illegally eroding that autonomy
ever since. Today, Hong Kong citizens have effectively no representation in
their government. The Legislative Council of Hong Kong has been deliberately
engineered by China to be pro-Beijing — a majority of the council is
selected through processes with an inherent pro-Beijing bias, giving Hong Kong
effectively no autonomous power to pass laws.<sup id="fnref:20"><a href="#fn:20" class="footnote-ref" role="doc-noteref">20</a></sup></p>
<p>Hong Kong&rsquo;s executive branch is even worse. The Chief Executive of Hong Kong
(Carrie Lam) is elected by a committee of 1,200 members largely controlled by
pro-Beijing seats, from a pool of pro-Beijing candidates, and the people have
effectively no representation in the election. The office has been held by
pro-Beijing politicians since it was established.<sup id="fnref:21"><a href="#fn:21" class="footnote-ref" role="doc-noteref">21</a></sup></p>
<p>The ongoing protests in Hong Kong were sparked by a mainland attempt to rein
in Hong Kong&rsquo;s judicial system in a similar manner, with the introduction of the
&ldquo;Fugitive Offenders and Mutual Legal Assistance in Criminal Matters Legislation
(Amendment) Bill 2019&rdquo;,<sup id="fnref:22"><a href="#fn:22" class="footnote-ref" role="doc-noteref">22</a></sup> which would have allowed the authorities
to extradite suspects awaiting trial to mainland China. These protests inspired
the Hong Kong people to stand up against all of the injustices they have faced
from China&rsquo;s illegal encroachments on their politics. The protesters have five
demands:<sup id="fnref:23"><a href="#fn:23" class="footnote-ref" role="doc-noteref">23</a></sup></p>
<ol>
<li>Complete withdrawal of the extradition bill</li>
<li>No prosecution of the protesters</li>
<li>Retraction of the characterization of the protests as &ldquo;riots&rdquo;</li>
<li>Establish an independent inquiry into police misconduct</li>
<li>Resignation of Carrie Lam and the implementation of universal suffrage</li>
</ol>
<p>Their first demand has been met, but the others are equally important and the
protests show no signs of slowing. Unfortunately, China shows no signs of
slowing their crackdown either, and have been consistently escalating the
matter. The police are now threatening to use live rounds on the
protesters,<sup id="fnref:24"><a href="#fn:24" class="footnote-ref" role="doc-noteref">24</a></sup> and people are already being shot in the
streets.<sup id="fnref:25"><a href="#fn:25" class="footnote-ref" role="doc-noteref">25</a></sup> China is going to kill the protesters, [again][tiananmen].</p>
<p>The third demand — the retraction of the characterization of the
demonstrations as &ldquo;riots&rdquo; — and the government&rsquo;s refusal to meet it,
conveys a lot about China&rsquo;s true intentions. Chinese law defines rioting as a
capital offense,<sup id="fnref:26"><a href="#fn:26" class="footnote-ref" role="doc-noteref">26</a></sup> and we&rsquo;ve already demonstrated their
willingness to execute political prisoners on a massive scale. These protesters
are going to be killed if their demands aren&rsquo;t met.<sup id="fnref:27"><a href="#fn:27" class="footnote-ref" role="doc-noteref">27</a></sup></p>
<p>Hong Kong is the place where humanity makes its stand against oppressors. The
people of Hong Kong have been constant allies to the West, and their liberty is
at stake. If we want others to stand up for us when our liberties are on the
line, then it&rsquo;s our turn to pay it forward now. The founding document of the
United States of America<sup id="fnref:28"><a href="#fn:28" class="footnote-ref" role="doc-noteref">28</a></sup> describes the rights they&rsquo;re defending as
&ldquo;unalienable&rdquo; — endowed upon all people by their Creator. The people of
Hong Kong are our friends and we&rsquo;re watching them get killed for rights that we
hold dear in our own nation&rsquo;s founding principles.</p>
<p>We have a legal basis for demanding these rights for Hong Kong&rsquo;s people —
China is blatantly violating their autonomy, which they agreed to uphold
in 1984. The United Kingdom should feel obligated to step in, but they&rsquo;ll need
the support of the international community, which we need to be prepared to give
them. We need to make an ultimatum: if China uses deadly force in Hong Kong,
the international community will respond in kind.</p>
<p>China isn&rsquo;t the only perpetrator of genocide today, but they are persecuting our
friends. China has the second highest GDP<sup id="fnref:29"><a href="#fn:29" class="footnote-ref" role="doc-noteref">29</a></sup> in the world, and somehow
this makes it okay. If we won&rsquo;t stand up to them, then who will? I call for a
worldwide boycott of Chinese products, and of companies who kowtow to their
demands or accept investment from China. I call for international condemnation
of the Communist Party of China&rsquo;s behavior and premise for governance. And I
call for an ultimatum to protect our allies from slaughter.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>An island in the sea east of China governed by the sovereign Republic of China. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Which, admittedly, raises concerns of its own. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p><a href="https://www.census.gov/foreign-trade/balance/c5700.html">US Census Bureau, International Trade Data</a> <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p><a href="https://www.lg.com/us/press-release/lg-electronics-to-build-us-factory-for-home-appliances-in-tennessee">LG</a>, <a href="http://download.intel.com/newsroom/kits/22nm/pdfs/Global-Intel-Manufacturing_FactSheet.pdf">Intel</a> (PDF) <a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p><a href="https://www.trademap.org/Bilateral_TS.aspx?nvpm=1%7C842%7C%7C156%7C%7CTOTAL%7C%7C%7C2%7C1%7C1%7C1%7C2%7C1%7C1%7C1%7C1">ITC Trade Map</a> <a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>Source(s): Ebenstein, Avraham, et al. &ldquo;Understanding the Role of China in the ‘Decline’of US Manufacturing.&rdquo; Manuscript, Hebrew University of Jerusalem (2011); <a href="https://www.epi.org/publication/the-china-toll-deepens-growth-in-the-bilateral-trade-deficit-between-2001-and-2017-cost-3-4-million-u-s-jobs-with-losses-in-every-state-and-congressional-district/">The China toll deepens</a>, Robert E. Scott and Zane Mokhiber, Economic Policy Institute <a href="#fnref:6" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>Source: Ulbrich, Timothy R., and Loren M. Kirk. &ldquo;It’s time to broaden the conversation about the student debt crisis beyond rising tuition costs.&rdquo; American journal of pharmaceutical education 81.6 (2017): 101. <a href="#fnref:7" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>A political prisoner is someone who is imprisoned for political reasons, rather than legal reasons. In the eyes of Chinese law, there may be a legal standing for the imprisonment of some of these people, but because this is often based on dissent from the single political party, I consider these prisoners political as well. A related term is &ldquo;prisoner of conscience&rdquo;, and for the purposes of this article I do not distinguish between the two; the execution of either kind of prisoner is a crime against humanity regardless. <a href="#fnref:8" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>Trey, T., et al. &ldquo;Transplant medicine in China: need for transparency and international scrutiny remains.&rdquo; American Journal of Transplantation 16.11 (2016): 3115-3120. <a href="#fnref:9" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p><a href="https://www.amnesty.org/en/latest/news/2017/04/china-must-come-clean-about-capital-punishment/">Death Penalty: World’s biggest executioner China must come clean about ‘grotesque’ level of capital punishment</a>, Amnesty International, 11 April 2017 <a href="#fnref:10" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>Jensen, Steven J., ed. The ethics of organ transplantation. CUA Press, 2011. <a href="#fnref:11" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p>UK has some of the best times in the developed world, and averages about 3 years. Source: <a href="https://web.archive.org/web/20110903084007/http://www.organdonation.nhs.uk/ukt/statistics/centre-specific_reports/pdf/waiting_time_to_transplant.pdf">NHS</a> <a href="#fnref:12" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:13" role="doc-endnote">
<p>Matas, David, and David Kilgour. &ldquo;An independent investigation into allegations of organ harvesting of Falun Gong practitioners in China.&rdquo; Electronic document accessed September 5 (2007): 2008. <a href="#fnref:13" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:14" role="doc-endnote">
<p><a href="https://web.archive.org/web/20140304045314/http://www.thetimes.co.uk/tto/news/world/asia/article2612313.ece">China to ‘tidy up’ trade in executed prisoners’ organs</a>, the UK Times, December 3 2005 <a href="#fnref:14" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:15" role="doc-endnote">
<p><a href="https://www.bbc.com/news/world-asia-china-45147972">China Uighurs: One million held in political camps, UN told</a>, BBC, 10 August 2018 <a href="#fnref:15" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:16" role="doc-endnote">
<p><a href="https://www.nytimes.com/interactive/2019/11/16/world/asia/china-xinjiang-documents.html">‘Absolutely No Mercy’: Leaked Files Expose How China Organized Mass Detentions of Muslims</a>, New York Times, 16 November 2019 <a href="#fnref:16" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:17" role="doc-endnote">
<p><a href="https://www.nytimes.com/2018/01/11/business/china-marriott-tibet-taiwan.html">Marriott to China: We Do Not Support Separatists</a>, New York Times, 11 January 2018 <a href="#fnref:17" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:18" role="doc-endnote">
<p><a href="https://qz.com/1723334/apple-removes-taiwan-flag-emoji-in-hong-kong-macau-in-ios-13-1-1/">Apple bows to China by censoring Taiwan flag emoji</a>, Quartz, 7 October 2019 <a href="#fnref:18" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:19" role="doc-endnote">
<p><a href="https://www.npr.org/2019/10/08/768245386/blizzard-entertainment-bans-esports-player-after-pro-hong-kong-comments">Blizzard Entertainment Bans Esports Player After Pro-Hong Kong Comments</a>, NPR, 8 October 2019 <a href="#fnref:19" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:20" role="doc-endnote">
<p><a href="https://en.wikipedia.org/wiki/Legislative_Council_of_Hong_Kong#Procedure">Legislative Council of Hong Kong, Wikipedia</a> <a href="#fnref:20" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:21" role="doc-endnote">
<p><a href="https://en.wikipedia.org/wiki/Chief_Executive_of_Hong_Kong#List_of_Chief_Executives_of_Hong_Kong">List of Chief Executives of Hong Kong</a>, Wikipedia <a href="#fnref:21" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:22" role="doc-endnote">
<p><a href="https://www.hklii.hk/eng/hk/legis/ord/503/index.html">https://www.hklii.hk/eng/hk/legis/ord/503/index.html</a> <a href="#fnref:22" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:23" role="doc-endnote">
<p><a href="https://focustaiwan.tw/news/acs/201906270014.aspx">https://focustaiwan.tw/news/acs/201906270014.aspx</a> <a href="#fnref:23" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:24" role="doc-endnote">
<p><a href="https://www.washingtonpost.com/world/hong-kong-police-pummel-university-with-water-cannon-as-officer-hit-by-arrow/2019/11/17/f004c978-091f-11ea-8054-289aef6e38a3_story.html">Hong Kong police move on university campus, threaten live rounds, retreat before growing flames</a>, The Washington Post, 17 November 2019
[tiananmen]: <a href="https://en.wikipedia.org/wiki/1989_Tiananmen_Square_protests">https://en.wikipedia.org/wiki/1989_Tiananmen_Square_protests</a> <a href="#fnref:24" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:25" role="doc-endnote">
<p>Source: <a href="https://streamable.com/0pexa">Video (graphic)</a> <a href="#fnref:25" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:26" role="doc-endnote">
<p>Criminal Law of the People&rsquo;s Republic of China, <a href="https://www.cecc.gov/resources/legal-provisions/criminal-law-of-the-peoples-republic-of-china">translation provided by US Congressional-Executive Commission of China</a> <a href="#fnref:26" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:27" role="doc-endnote">
<p>As pointed out by Hong Kongers reading this article, Hong Kong has a <a href="https://en.wikipedia.org/wiki/Public_Order_Ordinance">separate definition of rioting</a>, which is not a capital offense. For my part, I am not entirely convinced that China isn&rsquo;t planning to use the &ldquo;riots&rdquo; classification as justification for a violent response. <a href="#fnref:27" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:28" role="doc-endnote">
<p><a href="https://www.archives.gov/founding-docs/declaration-transcript">Declaration of Independence, full text</a> <a href="#fnref:28" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:29" role="doc-endnote">
<p><a href="https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)">List of countries by GDP (nominal) - Wikipedia</a> <a href="#fnref:29" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/11/26/Avoid-traumatic-changes.html
title: Software developers should avoid traumatic changes
url: https://drewdevault.com/2019/11/26/Avoid-traumatic-changes.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>A lot of software has gone through changes which, in retrospect, I would
describe as &ldquo;traumatic&rdquo; to their communities. I recognize these sorts of changes
by their effect: we might have pulled through in the end, but only after a lot
of heartbreak, struggle, and hours of wasted hacking; but the change left a scar
on the community.</p>
<p>There are two common cases in which a change risks introducing this kind of
trauma:</p>
<ol>
<li>It requires everyone in the community, or nearly everyone, to overhaul their
code to get it <strong>working</strong> again</li>
<li>It requires everyone in the community, or nearly everyone, to overhaul their
code to get it <strong>idiomatic</strong> again</li>
</ol>
<p>Let&rsquo;s call these cases, respectively, strong and weak trauma. While these are
both traumatic changes, the kind of trauma they inflict on the community is
different. The first kind is more severe, but the latter is a bad idea, too. We
can examine these through two case-studies in Python: the (in)famous transition
to Python 3, and the less notorious introduction of asyncio.</p>
<p>In less than one month, Python 2 will reach its end of life, and even as a
staunch advocate of Python 3, I too have some software which is not going to
make it to the finish line in time<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. There&rsquo;s no doubt that Python 3 is much,
much better than Python 2. However, the transition was poorly handled, and
upgrading can be no small task for some projects. The result has been hugely
divisive and intimately familiar to anyone who works with Python, creating
massive rifts in the community and wasting millions of hours of engineer time
addressing. This kind of &ldquo;strong&rdquo; trauma is fairly easy to spot in advance.</p>
<p>The weaker kind of traumatic change is more subtle, and less talked about. It&rsquo;s
a slow burn, and it takes a long time for its issues to manifest. Consider the
case of asyncio: clearly it&rsquo;s an improvement for Python, whose previous attempts
at concurrency have fallen completely flat. The introduction of async/await and
coroutines throughout the software ecosystem is something I&rsquo;m generally very
pleased about. You&rsquo;ll see me reach for threads to solve a problem when hell
freezes over, and no earlier, so I&rsquo;m quite fond of first-class coroutines.</p>
<p>Unfortunately, this has a chilling effect on existing Python code.  The
introduction of asyncio has made large amounts of code idiomatically obsolete.
Requests, the darling of the Python world, is effectively useless in a
theoretical idiomatic post-asyncio world. The same is true of Flask, SQLAlchemy,
and many, many other projects. Just about anything that does I/O is unidiomatic
now.</p>
<p>Since nothing has actually <em>broken</em> with this change, the effects are more
subtle than with strong traumatic changes. The effect of asyncio has been to
hasten the onset of code rot. Almost all of SourceHut&rsquo;s code pre-dates asyncio,
for example, and I&rsquo;m starting to feel the limitations of the pre-asyncio model.
The opportunity to solve this problem by rewriting with asyncio in mind,
however, also presents me a chance to rewrite in anything else, and reevaluate
my choice of Python for the project entirely. It&rsquo;s a tough decision to think
about — the mature and diverse ecosystem of libraries that help to make a
case for Python is dramatically reduced when asyncio support is a consideration.</p>
<p>It may take years for the trauma to fully manifest, but the rift is still there
and can only grow. Large amounts of code is rotting and will have to be thrown
away for the brave new asyncio world. The introduction of asyncio has made
another clear &ldquo;before&rdquo; and &ldquo;after&rdquo; in the Python ecosystem. The years in between
will be rough, because all new Python code will either leverage the rotting
pre-asyncio ecosystem or suffer through an immature post-asyncio ecosystem.
It&rsquo;ll likely turn out for the better — years from now.</p>
<p>And sometimes these changes <em>are</em> for the better, but they should be carefully
thought out, and designed to minimize the potential impact. In practical terms,
it&rsquo;s for this reason that I urge caution with ideas like adding generics to
Go. In a post-generics world, a large amount of the Go ecosystem will suddenly
become unidiomatic, and breaking changes will required to bring it up to spec.
Let&rsquo;s think carefully about it, eh?</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Eh, kind of. I&rsquo;m theoretically behind the effort to drop Python 2 from Alpine Linux, but the overhaul is tons of work and the time I can put into the effort isn&rsquo;t going to be enough to finish before 2020. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/11/29/dotorg.html
title: Take action to save .org and prosecute those who sold out the internet
url: https://drewdevault.com/2019/11/29/dotorg.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>As many of you have no doubt heard, control of the .org registry has been sold
to private interests. There have been attempts to call them to reason, like
<a href="https://savedotorg.org/">Save .ORG</a>, but let&rsquo;s be realistic: they knew what
they&rsquo;re doing is wrong, the whole time. If they were a commercial entity, our
appeals would fall on deaf ears and that would be the end of it. But, they&rsquo;re
not a commercial entity - so our appeals may fall on deaf ears, but that doesn&rsquo;t
have to be the end of it.</p>
<p>The level of corruption on display by the three organizations involved in this
scam: ICANN (Internet Corporation for Assigned Names and Numbers), ISOC (The
Internet Society), and PIR (Public Interest Registry), is astounding and very
illegal. If you are not familiar with the matter, click this to read a summary:</p>
<details>
  <summary>Summary of the corrupt privatization of .org</summary>
  <p>
    The governance of names on the internet is kind of complicated. ISOC
    oversees a lot of activities in internet standards and governance, but their
    role in this mess is as the parent company of PIR. PIR is responsible for
    the .org registry, which oversees the governance of .org directly and
    collects fees for every sale of a .org domain. ICANN is the broader
    authority which oversees all domain allocation on the internet, and also
    collects a fee for every domain sold. There's a complex web of documents and
    procedures which govern these three organizations, and the name system as a
    whole, and all three of them were involved in this process. Each of these
    organizations is a non-profit, except for PIR, which in the course of this
    deal is trying to convert to a B corp.
  </p>
  <p>
    ICANN can set price limits on the sale of .org domains. In March of 2019,
    they proposed removing these price caps entirely. During the period for
    public comment, they received 3,300 comments against, and 6 in favor. On May
    13, they removed these price caps anyway.
  </p>
  <p>
    In November 2019, ISOC announced that they had approved the sale of PIR, the
    organization responsible for .org, to Ethos Capital, for an unspecified
    amount. According to
    <a
      href="https://www.internetsociety.org/board-of-trustees/minutes/147"
      rel="nofollow noopener"
    >the minutes</a>, the decision to approve this sale was unanimously voted on
    by the board. Additionally, it seems that Goldman Sachs had been involved in
    the sale to some degree.
  </p>
  <p>
    Fadi Chehadé became the CEO of ICANN in 2012. In 2016, he leaves his
    position before it expires to start a consulting company, and he later joins
    Abry Partners. One of the 3 partners is Erik Brooks. They later acquire
    Donuts, a private company managing domains. Donuts co-founder Jon Nevett
    becomes the CEO of PIR in December 2018. On May 7th, Chehadé registers
    EthosCapital.com, and on May 13th ICANN decided to remove the price caps
    despite 0.2% support from the public. On May 14th, the following day, Ethos
    Capital was incorporated, with Brooks as the CEO. In November 2019, ISOC
    approved the acquisition of PIR by Ethos Capital, a for-profit company.
  </p>
  <p>
    These are the names of the criminals who sold the internet. If you want to
    read more, <a
      href="https://www.privateinternetaccess.com/blog/2019/11/isoc-pir-ethos-capital-deal-timeline/"
      rel="noopener"
    >Private Internet Access</a> has a good write-up.
  </p>
  <p>Okay, now let's talk about what you can do about it.</p>
</details>
<p>If you are familiar with the .org heist, then like me, you&rsquo;re probably pissed
off. Here&rsquo;s how you can take action: all of these organizations are 501c3
non-profits. The sale of a non-profit to a for-profit entity like this is
illegal without very specific conditions being met. Additionally, this kind of
behavior is not the sort the IRS likes to see in a tax-exempt organization.
Therefore, we can take the following steps to put a stop to this:</p>
<ol>
<li>Write to the CA and VA attorney general offices encouraging them to
investigate the misbehavior of these three non-profits, which are
incorporated in their respective states.</li>
<li>File form 13909 with the IRS, encouraging them to review the organization&rsquo;s
non-profit status.</li>
</ol>
<p>This kind of behavior is illegal. The sale of a non-profit requires a letter
from the Attorneys General in both California (ICANN) and Virginia (ISOC, PIR).
Additionally, much of this behavior qualifies as &ldquo;self-dealing&rdquo;, or leveraging
one&rsquo;s power within an organization for their own benefit, rather than the
benefit of the organization. To report this, I&rsquo;ve prepared a letter to the CA
and VA Attorney&rsquo;s General offices, which you can read here:</p>
<ul>
<li><a href="https://yukari.sr.ht/ag-letter.pdf">Letter to the Attorney General</a></li>
</ul>
<p>I encourage you to consider writing a letter of your own, but I would not
recommend copying and pasting this letter. However, this kind of behavior is
also illegal in the eyes of the IRS, and a form is provided for this purpose.
Form 13909 is the appropriate means for reporting this behavior. You can
download a pre-filled form here, and I do encourage you to submit one this
yourself:</p>
<ul>
<li><a href="https://yukari.sr.ht/dotorg-form-13909.pdf">Form 13909 for ICANN and ISOC complaints (PDF)</a></li>
<li><a href="https://yukari.sr.ht/dotorg-form-13909.odg">Form 13909 for ICANN and ISOC complaints (ODG)</a></li>
</ul>
<p>This only includes complaints for ICANN and ISOC, as PIR is seeking to lose its
non-profit status anyway. You can print out the PDF, fill in your details on
both pages, and mail it to the address printed on the form; or you can download
the ODG, open it up with LibreOffice Draw, and fill in the remaining details
digitally, then email it to the address shown on the page.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p>
<p>Happy Thanksgiving! Funny how this all happened right when the American public
would be distracted&hellip;</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Crash course in LibreOffice Draw: press F2, then click and drag to make a new textbox. Select text and use Ctrl+[ to reduce the font size to something reasonable. The red button on the toolbar along the top will export the result as a PDF. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/12/09/Developers-shouldnt-distribute.html
title: Developers shouldn't distribute their own software
url: https://drewdevault.com/2019/12/09/Developers-shouldnt-distribute.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>An oft-heard complaint about Linux is that software distribution often takes
several forms: a Windows version, a macOS version, and&hellip; a Debian version, an
Ubuntu version, a Fedora version, a CentOS version, an openSUSE version&hellip; but
these complaints miss the point. The true distributable form for Linux software,
and rather for Unix software in general, is a .tar.gz file containing the source
code.</p>
<p><strong>Note</strong>: This article presumes that proprietary/nonfree software is irrelevant,
and so should you.</p>
<p>That&rsquo;s not to imply that end-users should take this tarball and run <code>./configure &amp;&amp; make &amp;&amp; sudo make install</code> themselves. Rather, the responsibility for
end-user software distribution is on the distribution itself. That&rsquo;s why we call
it a <em>distribution</em>. This relationship may feel like an unnecessary middleman to
the software developer who just wants to get their code into their user&rsquo;s hands,
but on the whole this relationship has far more benefits than drawbacks.</p>
<p>As the old complaint would suggest, there are hundreds of variants of Linux
alone, not to mention the BSD flavors and any novel new OS that comes out next
week. Each of these environments has its own take on how the system as a whole
should be organized and operate, and it&rsquo;s a fools' errand for a single team to
try and make sense of it all. More often than not, software which tries to field
this responsibility itself sticks out like a sore thumb on the user&rsquo;s operating
system, totally flying in the face the conventions set out by the distribution.</p>
<p>Thankfully, each distro includes its own set of volunteers dedicated to this
specific job: packaging software for the distribution and making sure it
conforms to the norms of the target environment. This model also adds a set of
checks and balances to the system, in which the distro maintainers can audit
each other&rsquo;s work for bugs and examine the software being packaged for
anti-features like telemetry or advertisements, patching it out as necessary.
These systems keep malware out of the repositories, handle distribution of
updates, cryptographically verifying signatures, scaling the distribution out
across many mirrors - it&rsquo;s a robust system with decades of refinement.</p>
<p>The difference in trust between managed software repositories like Debian,
Alpine Linux, Fedora, and so on; and unmanaged software repositories like PyPI,
npm, Chrome extensions, the Google Play store, Flatpak, etc — is starkly
obvious. Debian and its peers are full of quality software which integrates well
into the host system and is free of malware. Unmanaged repositories, however,
are <a href="https://www.zdnet.com/article/two-malicious-python-libraries-removed-from-pypi/">constant sources</a> for crapware and malware. I don&rsquo;t trust
developers to publish software with my best interests in mind, and developers
shouldn&rsquo;t ask for that level of trust. It&rsquo;s only through a partnership with
distributions that we can build a mutually trustworthy system for software
distribution.</p>
<p>Some developers may complain that distros ship their software too slowly, but
you shouldn&rsquo;t sweat it. End-user distros ship updates reasonably quickly, and
server distros ship updates at a schedule which meets the user&rsquo;s needs. This
inconsistent pace in release schedules among free software distributions is a
feature, not a bug, and allows the system to work to the needs of its specific
audience. You should use a distro that ships updates to <em>you</em> at the pace you
wish, and let your users do the same.</p>
<p>So, to developers: just don&rsquo;t worry about distribution! Stick a tarball on your
release page and leave the rest up to distros. And to users: install packages
from your distro&rsquo;s repositories, and learn how its packaging process works so
you can get involved when you find a package missing. It&rsquo;s not as hard as it
looks, and they could use your help. For my part, I work both as a developer,
packager, and end-user, publishing my software as tarballs, packaging some of it
up for my <a href="https://pkgs.alpinelinux.org/packages?name=&amp;branch=edge&amp;arch=x86_64&amp;maintainer=Drew+DeVault">distro of choice</a>, report bugs to other maintainers, and field
requests from maintainers of other distros as necessary. Software distribution
is a social system and it works.</p>


guid: https://drewdevault.com/2019/12/15/Status-update-December-2019.html
title: Status update, December 2019
url: https://drewdevault.com/2019/12/15/Status-update-December-2019.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>It&rsquo;s December 15th and it still hasn&rsquo;t snowed here. Why did I move to this
godforsaken den of unholy heat and rain? I think I have chosen a latitude <em>just</em>
southerly enough to deprive me of the climate I yearn for. I take some comfort
in the knowledge that I&rsquo;m travelling home to see the family in a couple of
weeks, and sure enough Colorado has been covered in snow for some time now.
Anyway, none of this is relevant to my work, which is what you came here for.
Let&rsquo;s take a look at this past month.</p>
<p>I&rsquo;ve started a couple of new projects this month, the first of which I call
&ldquo;<a href="https://git.sr.ht/~sircmpwn/himitsu">himitsu</a>&rdquo;. The goal is to build a
key-value store for secure information like passwords, keys, and so on. The
design is inspired by Plan 9&rsquo;s factotum, redesigned for Unix systems and
somewhat broader in scope. One interesting goal of himitsu is the ability for
programs to establish authenticated connections without ever handling your
secret information - for example, your email client could ask himitsu to connect
to an IMAP server, log in with your authentication details, then hand the
authenticated file descriptor to the mail reader. The key-value store can also
store things like the IMAP server address &amp; port, your username, and so on,
meaning your mail reader could work out of the box with zero configuration. Work
on this project will be slow going, as I have to use extra care to make sure
that it&rsquo;s secure and correct.</p>
<p>In SourceHut news, I focused mainly on two workstreams: single-sign-on and
names.sr.ht, the upcoming DNS and domain registration service. The first finally
fixes the problems with login across *.sr.ht, and now logging in once will log
you in everywhere. Other issues with internal OAuth keys expiring have been
fixed alongside these changes, and I&rsquo;ve implemented a lot of improvements to
the billing system as well. All of these should address some inconveniences
which have been frustrating users for a while now. As for names.sr.ht, let&rsquo;s
just share another teaser screenshot:</p>
<p><a href="https://sr.ht/LLkW.png"><img src="https://sr.ht/LLkW.png" alt="Screenshot of domain contact management on names.sr.ht"></a></p>
<p>I also received my <a href="https://www.pine64.org/pinephone/">PinePhone</a> this week, and
I&rsquo;ve been terribly excited to work on it. I&rsquo;ve already sent a few patches to
postmarketOS upstream, and intend to write more, to get sway working well as a
daily driver phone interface. &ldquo;Sway Mobile&rdquo; is now starting to take shape. The
first of the projects for this is the development of a touch-friendly
application launcher, which I&rsquo;ve dubbed
&ldquo;<a href="https://git.sr.ht/~sircmpwn/casa">casa</a>&rdquo;. Other projects I intend to work on
for Sway Mobile include a new, touch-friendly bar and lock screen, a new
on-screen keyboard program, and hopefully the development of touch bindings for
the compositor itself. I&rsquo;ll be writing up my plans in more detail, along with a
review of the PinePhone itself, in a blog post next week.</p>
<p>In the course of this work, I also made a small library that readers may find
useful for their own projects:
<a href="https://gitlab.freedesktop.org/ddevault/fdicons">libfdicons</a>. It implements the
FreeDesktop icon specification in a single small C library, which I need for
Casa. In other Wayland news, I&rsquo;ve made some modest progress on the book, and I
plan on writing more for it soon. I apologise for letting it get somewhat
sidelined while I focused on other projects. I ended up overhauling the XDG
chapter somewhat, as I found it pretty weak on a later reading. I intend to
write about seats (input) next, and will likely move the XDG chapter after the
seat chapter so things flow better. I&rsquo;ve also started a new Wayland compositor,
<a href="https://git.sr.ht/~sircmpwn/sedna">sedna</a>, which aims to reach a broader
audience than Sway can, and I&rsquo;ll be working on this as time permits.</p>
<p>Speaking of Sway, the next release (1.3) has been coming along, slowly but
surely. We&rsquo;re only blocked by one change now, and with the original author busy
I&rsquo;ve stepped up to offer what time I can implementing the last few changes. Once
we get that merged, I&rsquo;ll start working on the release process for Sway 1.3.
Thank you for your patience <img src="https://drewdevault.com/img/heart.png" style="height: 1rem;
display: inline" /></p>
<p><a href="https://git.sr.ht/~sircmpwn/aerc/refs/0.3.0">aerc 0.3.0</a> was released this
month, and progress on the next version has been going strong. Improvements to
aerc have been almost entirely community driven, and I&rsquo;ve only stepped in to
write a few small patches here and there. Thanks to all of the contributors for
their help! There are already quite a few changes in for 0.4.0, and more are in
review now, including many bug fixes, more sophisticated email templates,
contacts autocompletion, bulk email management, and more. All of this is thanks
to the great community which has grown around it!</p>
<p>That&rsquo;s all the updates I have for you today. I&rsquo;m still touched by the support
the community has given me to work on these projects. I could never be this
productive without your help. Thank you.</p>


guid: https://drewdevault.com/2019/12/18/PinePhone-review.html
title: PinePhone review
url: https://drewdevault.com/2019/12/18/PinePhone-review.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p><strong>tl;dr</strong>: Holy shit! This is the phone I have always wanted. I have never been
this excited about the mobile sector before. However: the software side is
totally absent — phone calls are very dubious, SMS is somewhat dubious,
LTE requires some hacks, and everything will have to be written from the ground
up.</p>
<p>I have a PinePhone developer edition model, which I paid for out of pocket<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>
and which took an excruciatingly long time to arrive. When it finally arrived,
it came with no SIM or microSD card (expected), and the eMMC had some half-assed
version of Android on it which just boot looped without POSTing to anything
useful<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>. This didn&rsquo;t bother me in the slightest — like any other
computer I&rsquo;ve purchased, I planned on immediately flashing my own OS on it. My
Linux distribution of choice for it is
<a href="https://postmarketos.org/">postmarketOS</a>, which is basically the mobile OS I&rsquo;d
build if I wanted to build a mobile OS.</p>
<p>Let me make this clear: <strong>right now, there are very few people, perhaps only
dozens, for whom this phone is the right phone, given the current level of
software support</strong>. I am not using it as my daily driver, and I won&rsquo;t for some
time. The only kind of person I would recommend this phone to is a developer who
believes in the phone and wants to help build the software necessary for it to
work.  However, it seems to me that all of the right people <em>are</em> working on the
software end of this phone — everyone I&rsquo;d expect from the pmOS community,
from KDE, from the kernel hackers — this phone has an unprecedented level
of community support and the software <em>will</em> be written.</p>
<p>So, what&rsquo;s it actually like?</p>
<details>
  <summary>Expand for a summary of the specs</summary>
  <p>
    The device is about
    <abbr title="The thickness of a GameBoy cartridge">1 cm thick</abbr>
    and weighs
    <abbr
      title="The weight of one GameBoy Color, with batteries, without cartridge"
    >188 grams</abbr>. The screen is about 16 cm tall, of which 1.5 cm is bezel,
    and <abbr
      title="About the width and height of a GameBoy color, plus 1 inch of height"
    >7.5 cm wide</abbr> (5 mm of bezel). The physical size and weight is very
    similar to my daily driver, a Samsung Galaxy J7 Refine. It has a USB-C port,
    which I understand can be reconfigured for DisplayPort, and a standard
    headphone jack and speakers, both of which sound fine in my experience. The
    screen is 720x1440, and looks about as nice as any other phone. It has
    front- and back-facing cameras, which I've yet to get working (I understand
    that someone has got them working at some point), plus a flash/lamp on the
    back, and an <abbr
      title="Note that the only values for R, G, and B that I've managed to get working are 0.0 and 1.0 each, for a total of 7 possible colors (including off)"
    >RGB LED</abbr> on the front.
  </p>
  <p>
    The eMMC is 16G and, side note, had <em>seventeen</em> partitions on it when
    I first got the phone. 2G of RAM, 4 cores. It's not very powerful, but in my
    experience it runs lightweight UIs (such as <a
    href="https://swaywm.org">sway</a>) just fine. With very little effort by
    way of power management, and with obvious power sinks left unfixed, the
    battery lasts about 5 hours.
  </p>
</details>
<p>In short, I&rsquo;m quite satisfied with it, but I&rsquo;ve never had especially strenuous
demands of my phone. I haven&rsquo;t run any benchmarks on the GPU, but it seems
reasonably fast and the open-source Lima driver supports GLESv2. The modem is
supported by <a href="https://01.org/ofono">Ofono</a>, which is a telephony daemon based on
dbus — however, I understand that we can just open <code>/dev/ttyUSB1</code> and talk
to the modem ourselves, and I may just write a program that does this. Using
Ofono, I have successfully spun up LTE internet, sent and received SMS messages,
and placed and answered phone calls - though the last one without working
audio. A friend from KDE, Bhushan Shah, is working on this and rumor has it that
a call has successfully been placed. I have not had success with MMS, but I
think it&rsquo;s possible. WiFi works. All of this with zero blobs and a kernel which
is&hellip; admittedly, pretty heavily patched, but <a href="https://gitlab.com/pine64-org/linux">open
source</a> and making its way upstream.<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup></p>
<p>Of course, no one wants to place phone calls by typing a lengthy command into
their terminal, but that these features can be done in an annoying way means
that it&rsquo;s feasible to write applications that do this in a convenient way. For
my part, I have been working on some components of a mobile-friendly Wayland
compositor, based on Sway, which I&rsquo;m calling Sway Mobile for the time being. I&rsquo;m
not sure if Sway will actually stick around once it becomes difficult to bend to
my will (it&rsquo;s designed for keyboard-driven operation, after all), but I&rsquo;m
building mobile shell components which will translate nicely to any other
wlroots-based compositors.</p>
<p>The first of these is a simple app drawer, which I&rsquo;ve dubbed
<a href="https://git.sr.ht/~sircmpwn/casa">casa</a>. I have a lot more stuff planned:</p>
<ul>
<li>A new bar/notification drawer/quick action thing</li>
<li>A dialer &amp; call manager, maybe integrated with gnome-contacts</li>
<li>A telephony daemon which records incoming SMS messages and pulls up the call
manager for incoming phone calls. Idea: write incoming SMS messages into a
Maildir.</li>
<li>A new touch-friendly Wayland lock screen</li>
<li>An on-screen keyboard program</li>
</ul>
<p>Here&rsquo;s a video showing casa in action:</p>
<p><video
src="https://yukari.sr.ht/casa.webm?cache-break"
style="max-width: 50%; margin: 0 auto; display: block"
autoplay loop muted >
Your browser does not support webm playback. Please choose a browser which
supports free and open standards.
</video></p>
<p>The latest version has 4 columns and uses the space a bit better. Also, in the
course of this work I put together the
<a href="https://gitlab.freedesktop.org/ddevault/fdicons">fdicons</a> library, which may be
useful to some.</p>
<p>I have all sorts of other small things to work on, like making audio behave
better and improving power management. I intend to contribute these tools to
postmarketOS upstream as a nice lightweight plug-and-play UI package you can
choose from when installing pmOS, either improving their existing
postmarketos-ui-sway meta-package or making something new.</p>
<p>In conclusion: I have been waiting for this phone for years and years and years.
I have been hoping that someone would make a phone whose hardware was compatible
with upstream Linux drivers, and could <em>theoretically</em> be used as a daily driver
if only the software were up to snuff. I wanted this because I knew that the
free software community was totally capable of building the software for such a
phone, if only the hardware existed. This is actually happening — all of
the free software people I would hope are working on the PinePhone, are working
on the PinePhone. And it&rsquo;s only $150! I could buy four of them for the price of
the typical smartphone! And I just might!</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>In other words, no one paid me to or even asked me to write this review. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>I understand that the final production run of the PinePhone is going to ship with postmarketOS or something. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>The upstream kernel actually does work if you patch in the DTS, but WiFi doesn&rsquo;t work and it&rsquo;s not very stable. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2019/12/30/dotfiles.html
title: Managing my dotfiles as a git repository
url: https://drewdevault.com/2019/12/30/dotfiles.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>There are many tools for managing your dotfiles - user-specific configuration
files. GNU stow is an example. I&rsquo;ve tried a few solutions over the years, but I
settled on a very simple system several years ago which has served me very well
in the time since: my $HOME is a git repository. <a href="https://git.sr.ht/~sircmpwn/dotfiles">This
repository</a>, in fact. This isn&rsquo;t an
original idea, but I&rsquo;m not sure where I first heard it from either, and I&rsquo;ve
extended upon it somewhat since.</p>
<p>The key to making this work well is my one-byte <code>.gitignore</code> file:</p>
<pre><code>*
</code></pre><p>With this line, and git will ignore all of the files in my $HOME directory, so I
needn&rsquo;t worry about leaving personal files, music, videos, other git
repositories, and so on, in my public dotfiles repo. But, in order to track
anything at all, we need to override the gitignore file on a case-by-case basis
with <code>git add -f</code>, or <code>--force</code>. To add my vimrc, I used the following command:</p>
<pre><code>git add -f .vimrc
</code></pre><p>Then I can commit and push normally, and .vimrc is tracked by git. The gitignore
file does not apply to any files which are already being tracked by git, so any
future changes to my vimrc show up in git status, git diff, etc, and can be
easilly committed with <code>git commit -a</code>, or added to the staging area normally
with <code>git add</code> — using <code>-f</code> is no longer necessary. Setting up a new
machine is quite easy. After the installation, I run the following commands:</p>
<pre><code>cd ~
git init
git remote add origin git@git.sr.ht:~sircmpwn/dotfiles
git fetch
git checkout -f master
</code></pre><p>A quick log-out and back in and I feel right at $HOME. Additionally, I have
configured $HOME as a prefix, so that ~/bin is full of binaries, ~/lib has
libraries, and so on; though I continue to use ~/.config rather than ~/etc. I
put $HOME/bin ahead of anything else in my path, which allows me to shadow
system programs with wrapper scripts as necessary. For example, ~/bin/xdg-open
is as follows:</p>
<div class="highlight"><pre class="chroma"><code class="language-sh" data-lang="sh"><span class="cp">#!/bin/sh
</span><span class="cp"></span><span class="k">case</span> <span class="s2">&#34;</span><span class="si">${</span><span class="nv">1</span><span class="p">%%:*</span><span class="si">}</span><span class="s2">&#34;</span> in
	http<span class="p">|</span>https<span class="p">|</span>*.pdf<span class="o">)</span>
		<span class="nb">exec</span> qutebrowser <span class="s2">&#34;</span><span class="nv">$1</span><span class="s2">&#34;</span>
		<span class="p">;;</span>
	mailto<span class="o">)</span>
		<span class="nb">exec</span> aerc <span class="s2">&#34;</span><span class="nv">$1</span><span class="s2">&#34;</span>
		<span class="p">;;</span>
	*<span class="o">)</span>
		<span class="nb">exec</span> /usr/bin/xdg-open <span class="s2">&#34;</span><span class="nv">$@</span><span class="s2">&#34;</span>
		<span class="p">;;</span>
<span class="k">esac</span>
</code></pre></div><p>Replacing the needlessly annoying-to-customize xdg-open with one that just
does what I want, falling back to /usr/bin/xdg-open if necessary. Many other
non-shadowed scripts and programs are found in ~/bin as well.</p>
<p>However, not all of my computers are configured equally. Some run different
Linux (or non-Linux) distributions, or have different concerns being desktops,
servers, laptops, phones, etc. It&rsquo;s often useful for this reason to be able to
customize my configuration for each host. For example, before $HOME/bin in my
$PATH, I have $HOME/bin/$(hostname). I also run several machines on
different architectures, so I include $HOME/bin/$(uname -m)<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> as well. To
customize my sway configuration to consider the different device configurations
of each host, I use the following directive in ~/.config/sway/config:</p>
<pre><code>include ~/.config/sway/`hostname`
</code></pre><p>Then I have a host-specific configuration there, also tracked by git so I can
conveniently update one machine from another. I take a similar approach to
per-host configuration for many other pieces of software I use.</p>
<p>Rather than using (and learning) any specialized tools, I find my needs quite
adequately satisfied by a simple composition of several Unix primitives with a
tool I&rsquo;m already very familiar with: git. Version controlling your configuration
files is a desirable trait even with other systems, so why not ditch the
middleman?</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p><code>uname -m</code> prints out the system architecture. Try it for yourself, I bet it&rsquo;ll read &ldquo;x86_64&rdquo; or maybe &ldquo;aarch64&rdquo;. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/01/04/Slow.html
title: Hello world
url: https://drewdevault.com/2020/01/04/Slow.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Let&rsquo;s say you ask your programming language to do the simplest possible task:
print out &ldquo;hello world&rdquo;. Generally this takes two syscalls: write and exit.
The following assembly program is the ideal Linux x86_64 program for this
purpose. A perfect compiler would emit this hello world program for any
language.</p>
<pre><code>bits 64
section .text
global _start
_start:
	mov rdx, len
	mov rsi, msg
	mov rdi, 1
	mov rax, 1
	syscall

	mov rdi, 0
	mov rax, 60
	syscall

section .rodata
msg: db &quot;hello world&quot;, 10
len: equ $-msg
</code></pre><p>Most languages do a whole lot of other crap other than printing out &ldquo;hello
world&rdquo;, even if that&rsquo;s all you asked for.</p>
<table class="table table-bordered">
  <thead>
    <tr>
      <th>Test case</th>
      <th>Source</th>
      <th>Execution time</th>
      <th>Total syscalls</th>
      <th>Unique syscalls</th>
      <th>Size (KiB)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Assembly</strong> (x86_64)</td>
      <td>
        <a href="#tests">test.S</a>
      </td>
      <td>0.00s real</td>
      <td>2</td>
      <td>2</td>
      <td>8.6 KiB*</td>
    </tr>
    <tr>
      <td><strong>Zig</strong> (small)</td>
      <td>
        <a href="#testzig">test.zig</a>
      </td>
      <td>0.00s real</td>
      <td>2</td>
      <td>2</td>
      <td>10.3 KiB</td>
    </tr>
    <tr>
      <td><strong>Zig</strong> (safe)</td>
      <td>
        <a href="#testzig">test.zig</a>
      </td>
      <td>0.00s real</td>
      <td>3</td>
      <td>3</td>
      <td>11.3 KiB</td>
    </tr>
    <tr>
      <td><strong>C</strong> (musl, static)</td>
      <td>
        <a href="#testc">test.c</a>
      </td>
      <td>0.00s real</td>
      <td>5</td>
      <td>5</td>
      <td>95.9 KiB</td>
    </tr>
    <tr>
      <td><strong>C</strong> (musl, dynamic)</td>
      <td>
        <a href="#testc">test.c</a>
      </td>
      <td>0.00s real</td>
      <td>15</td>
      <td>9</td>
      <td>602 KiB</td>
    </tr>
    <tr>
      <td><strong>C</strong> (glibc, static*)</td>
      <td>
        <a href="#testc">test.c</a>
      </td>
      <td>0.00s real</td>
      <td>11</td>
      <td>9</td>
      <td>2295 KiB</td>
    </tr>
    <tr>
      <td><strong>C</strong> (glibc, dynamic)</td>
      <td>
        <a href="#testc">test.c</a>
      </td>
      <td>0.00s real</td>
      <td>65</td>
      <td>13</td>
      <td>2309 KiB</td>
    </tr>
    <tr>
      <td><strong>Rust</strong></td>
      <td>
        <a href="#testrs">test.rs</a>
      </td>
      <td>0.00s real</td>
      <td>123</td>
      <td>21</td>
      <td>244 KiB</td>
    </tr>
    <tr>
      <td><strong>Crystal</strong> (static)</td>
      <td>
        <a href="#testcr">test.cr</a>
      </td>
      <td>0.00s real</td>
      <td>144</td>
      <td>23</td>
      <td>935 KiB</td>
    </tr>
    <tr>
      <td><strong>Go</strong> (static w/o cgo)</td>
      <td>
        <a href="#testgo">test.go</a>
      </td>
      <td>0.00s real</td>
      <td>152</td>
      <td>17</td>
      <td>1661 KiB</td>
    </tr>
    <tr>
      <td><strong>D</strong> (dmd)</td>
      <td>
        <a href="#testd">test.d</a>
      </td>
      <td>0.00s real</td>
      <td>152</td>
      <td>20</td>
      <td>5542 KiB</td>
    </tr>
    <tr>
      <td><strong>D</strong> (ldc)</td>
      <td>
        <a href="#testd">test.d</a>
      </td>
      <td>0.00s real</td>
      <td>181</td>
      <td>21</td>
      <td>10305 KiB</td>
    </tr>
    <tr>
      <td><strong>Crystal</strong> (dynamic)</td>
      <td>
        <a href="#testcr">test.cr</a>
      </td>
      <td>0.00s real</td>
      <td>183</td>
      <td>25</td>
      <td>2601 KiB</td>
    </tr>
    <tr>
      <td><strong>Go</strong> (w/cgo)</td>
      <td>
        <a href="#testgo">test.go</a>
      </td>
      <td>0.00s real</td>
      <td>211</td>
      <td>22</td>
      <td>3937 KiB</td>
    </tr>
    <tr>
      <td><strong>Perl</strong></td>
      <td>
        <a href="#testpl">test.pl</a>
      </td>
      <td>0.00s real</td>
      <td>255</td>
      <td>25</td>
      <td>5640 KiB</td>
    </tr>
    <tr>
      <td><strong>Java</strong></td>
      <td>
        <a href="#testjava">Test.java</a>
      </td>
      <td>0.07s real</td>
      <td>226</td>
      <td>26</td>
      <td>15743 KiB</td>
    </tr>
    <tr>
      <td><strong>Node.js</strong></td>
      <td>
        <a href="#testjs">test.js</a>
      </td>
      <td>0.04s real</td>
      <td>673</td>
      <td>40</td>
      <td>36000 KiB</td>
    </tr>
    <tr>
      <td><strong>Python 3</strong> (PyPy)</td>
      <td>
        <a href="#testpy">test.py</a>
      </td>
      <td>0.68s real</td>
      <td>884</td>
      <td>32</td>
      <td>9909 KiB</td>
    </tr>
    <tr>
      <td><strong>Julia</strong></td>
      <td>
        <a href="#testjl">test.jl</a>
      </td>
      <td>0.12s real</td>
      <td>913</td>
      <td>41</td>
      <td>344563 KiB</td>
    </tr>
    <tr>
      <td><strong>Python 3</strong> (CPython)</td>
      <td>
        <a href="#testpy">test.py</a>
      </td>
      <td>0.02s real</td>
      <td>1200</td>
      <td>33</td>
      <td>15184 KiB</td>
    </tr>
    <tr>
      <td><strong>Ruby</strong></td>
      <td>
        <a href="#testrb">test.rb</a>
      </td>
      <td>0.04s real</td>
      <td>1401</td>
      <td>38</td>
      <td>1283 KiB</td>
    </tr>
  </tbody>
</table>
<div style="text-align: right">
  <small>* See notes for this test case</small>
</div>
<p>This table is sorted so that the number of syscalls goes up, because I reckon
more syscalls is a decent metric for how much shit is happening that you didn&rsquo;t
ask for (i.e. <code>write(&quot;hello world\n&quot;); exit(0)</code>). Languages with a JIT fare much
worse on this than compiled languages, but I have deliberately chosen not to
account for this.</p>
<p>These numbers are real. This is more complexity that someone has to debug, more
time your users are sitting there waiting for your program, less disk space
available for files which actually matter to the user.</p>
<h3 id="environment">Environment</h3>
<p>Tests were conducted on January 3rd, 2020.</p>
<ul>
<li>gcc 9.2.0</li>
<li>glibc 2.30</li>
<li>musl libc 1.1.24</li>
<li>Linux 5.4.7 (Arch Linux)</li>
<li>Linux 4.19.87 (vanilla, Alpine Linux) is used for musl libc tests</li>
<li>Go 1.13.5</li>
<li>Rustc 1.40.0</li>
<li>Zig 0.5.0</li>
<li>OpenJDK 11.0.5 JRE</li>
<li>Crystal 0.31.1</li>
<li>NodeJS 13.5.0</li>
<li>Julia 1.3.1</li>
<li>Python 3.8.1</li>
<li>PyPy 7.3.0</li>
<li>Ruby 2.6.4p114 (2019-10-01 rev 67812)</li>
<li>dmd 1:2.089.0</li>
<li>ldc 2:1.18.0</li>
<li>Perl 5.30.1</li>
</ul>
<p>For each language, I tried to write the program which would give the most
generous scores without raising eyebrows at a code review. The size of all
files which must be present at runtime (interpreters, stdlib, libraries, loader,
etc) are included. Binaries were stripped where appropriate.</p>
<p>This was not an objective test, this is just an approximation that I hope will
encourage readers to be more aware of the consequences of their abstractions,
and their exponential growth as more layers are added.</p>
<h3 id="tests">test.S</h3>
<pre><code>bits 64
section .text
global _start
_start:
	mov rdx, len
	mov rsi, msg
	mov rdi, 1
	mov rax, 1
	syscall

	mov rdi, 0
	mov rax, 60
	syscall

section .rodata
msg: db &quot;hello world&quot;, 10
len: equ $-msg
</code></pre><pre><code>nasm -f elf64 test.S
gcc -o test -static -nostartfiles -nostdlib -nodefaultlibs
strip test: 8.6 KiB
</code></pre><p><strong>Notes</strong></p>
<ul>
<li>This program only works on x86_64 Linux.</li>
<li>The size depends on how you measure it:<br />
<em>Instructions + data alone</em>: 52 bytes<br />
<em>Stripped ELF</em>: 8.6 KiB<br />
<em>Manually minified ELF</em>: <a href="http://timelessname.com/elfbin/">142 bytes</a></li>
</ul>
<h3 id="testzig">test.zig</h3>
<pre><code>const std = @import(&quot;std&quot;);

pub fn main() !void {
    const stdout = try std.io.getStdOut();
    try stdout.write(&quot;hello world\n&quot;);
}
</code></pre><pre><code># small
zig build-exe test.zig --release-small --strip
# safe
zig build-exe test.zig --release-safe --strip
</code></pre><p><strong>Notes</strong></p>
<ul>
<li>Written with the assistance of Andrew Kelly (maintainer of Zig)</li>
</ul>
<h3 id="testc">test.c</h3>
<pre><code>int puts(const char *s);

int main(int argc, char *argv[]) {
    puts(&quot;hello world&quot;);
    return 0;
}
</code></pre><pre><code># dynamic
gcc -O2 -o test test.c
strip test

# static
gcc -O2 -o test -static test.c
strip test
</code></pre><p><strong>Notes</strong></p>
<ul>
<li>glibc programs can never truly be statically linked. The size reflects this.</li>
</ul>
<h3 id="testrs">test.rs</h3>
<pre><code>fn main() {
    println!(&quot;hello world&quot;);
}
</code></pre><pre><code>rustc -C opt-levels=s test.rs
</code></pre><p><strong>Notes</strong></p>
<ul>
<li>The final binary is dynamically linked with glibc, which is included in the
size.</li>
</ul>
<h3 id="testgo">test.go</h3>
<pre><code>package main

import &quot;os&quot;

func main() {
    os.Stdout.Write([]byte(&quot;hello world\n&quot;))
}
</code></pre><pre><code># dynamic
go build -o test test.go

# static w/o cgo
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o test -ldflags '-extldflags &quot;-f no-PIC -static&quot;' -buildmode pie -tags 'osusergo netgo static_build' test.go
</code></pre><p>Aside: it is getting way too goddamn difficult to build static Go binaries.</p>
<p><strong>Notes</strong></p>
<ul>
<li>The statically linked test was run on Alpine Linux with musl libc. It doesn&rsquo;t
link to libc in theory, but hey.</li>
</ul>
<h3 id="testjava">Test.java</h3>
<pre><code>public class Test {
    public static void main(String[] args) {
        System.out.println(&quot;hello world&quot;);
    }
}
</code></pre><pre><code>javac Test.java
java Test
</code></pre><h3 id="testcr">test.cr</h3>
<pre><code>puts &quot;hello world\n&quot;
</code></pre><pre><code># Dynamic
crystal build -o test test.cr

# Static
crystal build --static -o test test.cr
</code></pre><p><strong>Notes</strong></p>
<ul>
<li>The Crystal tests were run on Alpine Linux with musl libc.</li>
</ul>
<h3 id="testjs">test.js</h3>
<pre><code>console.log(&quot;hello world&quot;);
</code></pre><pre><code>node test.js
</code></pre><h3 id="testjl">test.jl</h3>
<pre><code>println(&quot;hello world&quot;)
</code></pre><pre><code>julia test.jl
</code></pre><p><strong>Notes</strong></p>
<ul>
<li>Julia numbers were provided by a third party</li>
</ul>
<h3 id="testpy">test.py</h3>
<pre><code>print(&quot;hello world&quot;)
</code></pre><pre><code># cpython
python3 test.py
# pypy
pypy3 test.py
</code></pre><h3 id="testpl">test.pl</h3>
<pre><code>print &quot;hello world\n&quot;
</code></pre><pre><code>perl test.pl
</code></pre><p><strong>Notes</strong></p>
<ul>
<li>Passing /dev/urandom into perl is equally likely to print &ldquo;hello world&rdquo;</li>
</ul>
<h3 id="testd">test.d</h3>
<pre><code>import std.stdio;
void main()
{
    writeln(&quot;hello world&quot;);
}
</code></pre><pre><code># dmd
dmd -O test.d
# ldc
ldc -O test.d
</code></pre><h3 id="testrb">test.rb</h3>
<pre><code>puts &quot;hello world\n&quot;
</code></pre><pre><code>ruby test.rb
</code></pre>


guid: https://drewdevault.com/2020/01/08/Re-Slow.html
title: Following up on "Hello world"
url: https://drewdevault.com/2020/01/08/Re-Slow.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>This is a follow-up to my last article, <a href="https://drewdevault.com/2020/01/04/Slow.html">Hello
world</a>, which is easily the most
negatively received article I&rsquo;ve written — a remarkable feat for someone
who&rsquo;s written as much flame bait as me. Naturally, the fault lies with the
readers.</p>
<p><a href="https://xkcd.com/1984/" rel="noopener"><img src="https://imgs.xkcd.com/comics/misinterpretation_2x.png" width="294" /></a></p>
<p>All jokes aside, I&rsquo;ll try to state my point better. The &ldquo;Hello world&rdquo; article
was a lot of work to put together — frustrating work — by the time
I had finished collecting numbers, I was exhausted and didn&rsquo;t pay much mind to
putting context to them. This left a lot of it open to interpretation, and a lot
of those interpretations didn&rsquo;t give the benefit of the doubt.</p>
<p>First, it&rsquo;s worth clarifying that the assembly program I gave is a
<em>hypothetical, idealized</em> hello world program, and in practice not even the
assembly program is safe from bloat. After it&rsquo;s wrapped up in an ELF, even after
stripping, the binary bloats up to <strong>157×</strong> the size of the
actual machine code. I had hoped this would be more intuitively clear, but the
take-away is that the ideal program is a pipe dream, not a standard to which the
others are held. As the infinite frictionless plane in vacuum is to physics,
that assembly program is to compilers.</p>
<p>I also made the mistake of including the runtime in the table. What I wanted you
to notice about the timestamp is that it <em>rounds to zero</em> for 15 of the 21 test
cases, and arguably only one or two approach the realm of human perception.
It&rsquo;s meant to lend balance to the point I&rsquo;m making with the number of syscalls:
despite the complexity on display, the user generally can&rsquo;t even tell. The other
problem with including the runtimes is that it makes it look like a benchmark,
which it&rsquo;s not (you&rsquo;ll notice that if you grep for &ldquo;benchmark&rdquo;, you will find no
results).</p>
<p>Another improvement would have been to group rows of the table by orders of
magnitude (in terms of number of syscalls), and maybe separate the outliers in
each group. There is little difference between many of the languages in the
middle of the table, but when one of them is your favorite language, &ldquo;stacking
it up&rdquo; against its competitors like this is a good way to get the reader&rsquo;s blood
pumping and bait some flames. If your language appears to be represented
unfavorably on this chart, you&rsquo;re likely to point out the questionable
methodology, golf your way to a more generous sample code, etc; things I could
have done myself were I trying to make a benchmark rather than a point about
complexity.</p>
<p>And hidden therein is my actual point: complexity. There has long been a trend
in computing of endlessly piling on the abstractions, with no regard for the
consequences. The web is an ever growing mess of complexity, with larger and
larger blobs of inscrutable JavaScript being shoved down pipes with no regard
for the pipe&rsquo;s size or the bridge toll charged by the end-user&rsquo;s telecom.
Electron apps are so far removed from hardware that their jarring non-native UIs
can take seconds to respond and eat up the better part of your RAM to merely
show a text editor or chat application.</p>
<p>The PC in front of me is literally five thousand times faster than the graphing
calculator in my closet - but the latter can boot to a useful system in a
fraction of a millisecond, while my PC takes almost a minute. Productivity per
CPU cycle per Watt is the lowest it&rsquo;s been in decades, and is orders of
magnitude (plural) beneath its potential. So far as most end-users are
concerned, computers haven&rsquo;t improved in meaningful ways in the past 10 years,
and in many respects have become worse. The cause is well-known: programmers
have spent the entire lifetime of our field recklessly piling abstraction on top
of abstraction on top of abstraction. We&rsquo;re more concerned with shoving more
spyware at the problem than we are with optimization, outside of a small number
of high-value problems like video decoding.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> Programs have grown fat and
reckless in scope, and it affects literally everything, even down to the last
bastion of low-level programming: C.</p>
<p>I use syscalls as an approximation of this complexity. Even for one of the
simplest possible programs, there is a huge amount of abstraction and complexity
that comes with many approaches to its implementation. If I just print &ldquo;hello
world&rdquo; in Python, users are going to bring along almost a million lines of code
to run it, the fraction of which isn&rsquo;t dead code is basically a rounding error.
This isn&rsquo;t <em>always</em> a bad thing, but it often is and no one is thinking about
it.</p>
<p>That&rsquo;s the true message I wanted you to take away from my article: most
programmers aren&rsquo;t thinking about this complexity. Many choose tools because
it&rsquo;s easier for them, or because it&rsquo;s what they know, or because developer time
is more expensive than the user&rsquo;s CPU cycles or battery life and the engineers
aren&rsquo;t signing the checks. I hoped that many people would be surprised at just
how much work their average programming language could end up doing even when
given simple tasks.</p>
<p>The point was not that your programming language is wrong, or that being higher
up on the table is better, or that programming languages should be blindly
optimizing these numbers. The point is, if these numbers surprised you, then you
should find out why! I&rsquo;m a systems programmer — I want you to be
interested in your systems! And if this surprises you, I wonder what else
might&hellip;</p>
<p>I know that article didn&rsquo;t do a good job of explaining any of this. I&rsquo;m sorry.</p>
<hr>
<p>Now to address more specific comments:</p>
<p><strong>What the fuck is a syscall</strong>?</p>
<p>This question is more common with users of the languages which make more of
them, ironically. A syscall is when your program asks the kernel to do something
for it. This causes a transition from <em>user space</em> to <em>kernel space</em>. This
transition is one of the more expensive things your programs can do, but a
program that doesn&rsquo;t make any syscalls is not a useful program: syscalls are
necessary to do any kind of I/O (input or output). <a href="https://en.wikipedia.org/wiki/System_call">Wikipedia
page</a>.</p>
<p>On Linux, you can use the <a href="https://linux.die.net/man/1/strace">strace</a> tool to
analyze the syscalls your programs are making, which is how I obtained the
numbers in the original article.</p>
<p><strong>This &ldquo;benchmark&rdquo; is biased against JIT&rsquo;d and interpreted languages</strong>.</p>
<p>Yes, it is. It <em>is</em> true that many programming environments have to factor
in a &ldquo;warm up&rdquo; time. This argument on its face-value is apparently validated by
the cargo-culted (and often correct) wisdom that benchmarks should be conducted
with timers in-situ, post warm-up period, with the measured task being
repeated many times so that trends become more obvious.<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> It&rsquo;s precisely these
details, which the conventional benchmarking wisdom aims to obscure, that I&rsquo;m
trying to cast a light on. While a benchmark which shows how quickly a bunch of
programming languages can print &ldquo;hello world&rdquo; a million times<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> might be
interesting, it&rsquo;s not what I&rsquo;m going for here.</p>
<p><strong>Rust is doing important things with those syscalls</strong>.</p>
<p>My opinion on this is mixed: yes, stack guards are useful. However, my &ldquo;hello
world&rdquo; program has zero chance of causing a stack overflow. In theory, Rust
should be able to reckon whether or not many programs are at risk of stack
overflow.  If not, it can ask the programmer to specify some bounds, or it can
emit the stack guards <em>only in those cases</em>. The worst option is panicking, and
I&rsquo;m surprised that Crustaceans feel like this is sufficient. Funny, given their
obsession with &ldquo;zero cost&rdquo; abstractions, that a nonzero-cost abstraction would
be so fiercely defended. They&rsquo;re already used to overlong compile times, adding
more analysis probably won&rsquo;t be noticed ;)</p>
<p><strong>Go is doing important things with those syscalls</strong>.</p>
<p>On this I wholly disagree. I hate the Go runtime, it&rsquo;s the worst thing about an
otherwise great language. Go programs are almost impossible to debug for having
to sift through mountains of unrelated bullshit the program is doing, all to
support a concurrency/parallelism model that I also strongly dislike. There are
some bad design decisions in Golang and stracing the average Go program brings a
lot of them to light. Illumos has many of its own problems, but <a href="http://dtrace.org/blogs/wesolows/2014/12/29/golang-is-trash/">this
article</a> about
porting Go to it covers a number of related problems.</p>
<p><strong>Wow, Zig is competitive with assembly?</strong></p>
<p>Yeah, I totally had the same reaction. I&rsquo;m interested to see how it measures up
under more typical workloads. People keep asking me what I think about Zig in
general, and I think it has potential, but I also have a lot of complaints. It&rsquo;s
not likely to replace C for me, but it might have a place somewhere in my stack.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>For efficient display of unskippable 30 second video ads, of course. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>This approach is the most &ldquo;fair&rdquo; for comparison&rsquo;s sake, but it also often obscures a lot of the practical value of the benchmark in the first place. For example, how often is the branch predictor and L1 cache going to be warmed up in favor of the measured code in practice? <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>All of them being handily beaten by <code>/bin/yes &quot;hello world&quot;</code> <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/01/15/Status-update-January-2020.html
title: Status update, January 2020
url: https://drewdevault.com/2020/01/15/Status-update-January-2020.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I forgot to write this post this morning, and I&rsquo;m on cup 3 of coffee while
knee-deep in some arcane work with tarballs in Python. Forgive the brevity of
this introduction. Let&rsquo;s get right into the status update.</p>
<p>First of all, <a href="https://fosdem.org/2020/">FOSDEM 2020</a> is taking place on February 1st and 2nd, and
I&rsquo;m planning on being there again this year. I hope to see you there! I&rsquo;ll be
hosting another <a href="https://fosdem.org/2020/schedule/event/bof_sourcehut/">small session</a> for SourceHut and aerc users
where I&rsquo;ll take questions, demo some new stuff, and give out stickers.</p>
<p>In Wayland news, the upcoming Sway 1.3 release is getting very close - rc3 is
planned to ship later today. We&rsquo;ve confirmed that it&rsquo;ll ship with VNC support
via <a href="https://github.com/any1/wayvnc">wayvnc</a> and improvements to input latency.
I haven&rsquo;t completed much extra work on Casa (and &ldquo;Sway Mobile&rdquo; alongside it),
but there have been some small improvements. I did find some time to work on
<a href="https://git.sr.ht/~sircmpwn/sedna">Sedna</a>, however. We&rsquo;ve decided to use it as
a proving grounds for the new wlroots scene graph API, which plans to
incorporate Simon Ser&rsquo;s <a href="https://github.com/emersion/libliftoff">libliftoff</a> and put to rest the eternal
debate over how wlroots renderer should take shape. This&rsquo;ll be <em>lots</em> of work
but the result will be a remarkably good foundation on which we can run
performant compositors on a huge variety of devices — and, if we&rsquo;re
lucky, might help resolve the Nvidia problem. I also did a bit more work on the
<a href="https://wayland-book.com">Wayland Book</a>, refactoring some of the chapter
ordering to make more sense and getting started with the input chapter. More
soon.</p>
<p>On SourceHut, lots of new developments have been underway. The latest round of
performance improvements for git.sr.ht finally landed with the introduction of
new server hardware, and it&rsquo;s finally competitive with its peers in terms of
push and web performance. I&rsquo;ve also overhauled our monitoring infrastructure
<a href="https://metrics.sr.ht">and made it public</a>. Our <a href="https://sourcehut.org/blog/2020-01-13-sourcehut-q4-2019-financial-report/">Q4 2019 financial
report</a> was also published earlier this week. I&rsquo;m currently
working on pushing forward through the self-service data ownership goals, and
we&rsquo;ve already seen some improvements in that todo.sr.ht can now re-import
tracker exports from itself or other todo.sr.ht instances.</p>
<p>I&rsquo;ve also been working more on <a href="https://git.sr.ht/~sircmpwn/himitsu">himitsu</a>
recently, though I&rsquo;m taking it pretty slowly because it&rsquo;s a security-sensitive
project. Most of the crypto code has been written at this point - writing
encrypted secrets to disk, reading and writing the key index - but reading
encrypted secrets back from the disk remains to be implemented. I know there are
some bugs in the current implementation, which I&rsquo;ll be sorting out before I
write much more code. I also implemented most of the support code for the Unix
socket RPC, and implemented a couple of basic commands which have been helpful
with proving out the secret store code (proving that it&rsquo;s wrong, at least).</p>
<p>Simon Ser&rsquo;s <a href="https://mrsh.sh">mrsh</a> has also been going very well lately, and is
now a nearly complete implementation of the POSIX shell. I&rsquo;ve started working on
something I&rsquo;ve long planned to build on top of mrsh: a comfortable interactive
shell, inspired by fish&rsquo;s interactive mode, but with a strictly POSIX syntax. I
call the project <a href="https://git.sr.ht/~sircmpwn/imrsh">imrsh</a>, for interactive
mrsh. I&rsquo;ve already got it in somewhat good shape, but many of the features
remain to be implemented. The bulk of the work was in Simon&rsquo;s mrsh, so it
shouldn&rsquo;t be too hard to add a pretty interface on top. We&rsquo;ll see how it goes.</p>
<p>That&rsquo;s all for today. In the coming month I hope to expand on each of these, and
I&rsquo;m also working on a new Secret Project which may start bearing fruits soon
(but likely not). Thank you for your continued support! I&rsquo;ll see you at FOSDEM.</p>


guid: https://drewdevault.com/2020/01/17/Effective-project-governance.html
title: A philosophy of project governance
url: https://drewdevault.com/2020/01/17/Effective-project-governance.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I&rsquo;ve been in the maintainer role for dozens of projects for a while now, and
have moderated my fair share of conflicts. I&rsquo;ve also been on the other side,
many times, as a minor contributor watching or participating in conflict within
other projects. Over the years, I&rsquo;ve developed an approach to project governance
which I believe is lightweight, effective, and inclusive.</p>
<p>I hold the following axioms to be true:</p>
<ol>
<li>Computer projects are organized by humans, creating a social system.</li>
<li>Social systems are fundamentally different from computer systems.</li>
<li>Objective rules cannot be programmed into a social system.</li>
</ol>
<p>And the following is true of individuals within those systems:</p>
<ol>
<li>Project leadership is in a position to do anything they want.</li>
<li>Project leadership will ultimately do whatever they want, even if they have
to come up with an interpretation of the rules which justifies it.</li>
<li>Individual contributors who have a dissonant world-view from project
leadership will never be welcome under those leaders.</li>
</ol>
<p>Any effective project governance model has to acknowledge these truths. To this
end, the simplest effective project governance model is a BDFL, which scales a
lot further than people might expect.</p>
<p>The BDFL (Benevolent Dictator for Life) is a term which was first used to
describe Python&rsquo;s governance model with Guido van Rossum at the helm. The &ldquo;for
life&rdquo; in BDFL is, in practice, until the &ldquo;dictator&rdquo; resigns from their role.
Transfers of power either involve stepping away and letting lesser powers decide
between themselves how to best fill the vacuum, or simply directly appointing a
successor (or successors). In this model, a single entity is in charge —
often the person who started the project, at first — and while they may
delegate their responsibilities, they ultimately have the final say in all
matters.</p>
<p>This decision-making authority derives from the BDFL. Consequently, the
project&rsquo;s values are a reflection of that BDFL&rsquo;s values. Conflict resolution and
matters of exclusion or inclusion of specific people from the project is the
direct responsibility of the BDFL. If the BDFL delegates this authority to other
groups or project members, that authority derives from the BDFL and is exercised
at their leisure, on their terms. In practice, for projects of a certain size,
most if not all of the BDFL&rsquo;s authority is delegated across many people, to the
point where the line between BDFL and core contributor is pretty blurred. The
relationships in the project are built on trust between individuals, not trust
in the system.</p>
<p>As a contributor, you should evaluate the value system of the leadership and
make a personal determination as to whether or not it aligns with your own. If
it does, participate. If it does not, find an alternative or fork the
project.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p>
<p>Consider the main competing model: a Code of Conduct as the rule of law.</p>
<p>These attempt to boil subjective matters down into objectively enforcible rules.
Not even in sovereign law do we attempt this. Programmers can easily fall into
the trap of thinking that objective rules can be applied to social systems, and
that they can deal with conflict by executing a script. This is quite untrue,
and attempting to will leave loopholes big enough for bad actors to drive a
truck through.</p>
<p>Additionally, governance models which provide a scripted path onto the decision
making committee can often have this path exploited by bad actors, or by people
for whom the politics are more important than the software. By implementing this
system, the values of the project can easily shift in ways the leaders and
contributors don&rsquo;t expect or agree with.</p>
<p>The worst case can be that a contributor is ostracized due to the letter of the
CoC, but not the spirit of it. Managing drama is a sensitive, subjective issue,
but objective rules break hearts. Enough of this can burn out the leaders,
creating a bigger power vacuum, without a plan to fill it.</p>
<p>In summary:</p>
<p><strong>For leaders</strong>: Assume good faith until proven otherwise.</p>
<p>Do what you think is right. If someone is being a
dickhead<small><sup>†</sup></small>,
tell them to stop.  If they don&rsquo;t stop, kick them out. Work with contributors
you trust to elevate their role in the project so you can delegate
responsibilities to them and have them act as good role models for the
community. If you&rsquo;re not good at moderating discussions or conflict resolution,
find someone who is among your trusted advisors and ask them to exercise their
skills.</p>
<p>If you need to, sketch up informal guidelines to give an approximation of your
values, so that contributors know how to act and what to expect, but make it
clear that they&rsquo;re guidelines rather than rules. Avoid creating complex systems
of governance. Especially avoid setting up systems which create paths that
untrusted people can use to quickly weasel their way into positions of power.
Don&rsquo;t give power to people who don&rsquo;t have a stake in the project.</p>
<p><strong>For contributors</strong>: Assume good faith until proven otherwise.</p>
<p>Do what you think is right. If someone is being a
dickhead<small><sup>†</sup></small>,
talk to the leadership about it. If you don&rsquo;t trust the project leadership, the
project isn&rsquo;t for you, and future conflicts aren&rsquo;t going to go your way. Be
patient with your maintainers — remember that you have the easier job.</p>
<p><small><sup>†</sup> According to your subjective definition of dickhead.</small></p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Note that being able to fork is the escape hatch which makes this model fair and applicable to free &amp; open source projects. The lack of a similarly accessible escape hatch in, for example, the governments of soverign countries, prevents this model from generalizing well. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/01/21/Stress-and-happiness.html
title: The happinesses and stresses of full-time FOSS work
url: https://drewdevault.com/2020/01/21/Stress-and-happiness.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>In the past few days, several free software maintainers have come out to discuss
the stresses of their work. Though the timing was suggestive, my article last
week on the philosophy of project governance was, at best, only tangentially
related to this topic - I had been working on that article for a while. I do
have some thoughts that I&rsquo;d like to share about what kind of stresses I&rsquo;ve
dealt with as a FOSS maintainer, and how I&rsquo;ve managed (or often mismanaged) it.</p>
<p>February will mark one year that I&rsquo;ve been working on self-directed free
software projects full-time. I was planning on writing an optimistic
retrospective article around this time, but given the current mood of the
ecosystem I think it would be better to be realistic. In this stage of my
career, I now feel at once happier, busier, more fulfilled, more engaged, more
stressed, and more depressed than I have at any other point in my life.</p>
<p>The good parts are numerous. I&rsquo;m able to work on my life&rsquo;s passions, and my
projects are in the best shape they&rsquo;ve ever been thanks to the attention I&rsquo;m
able to pour into them. I&rsquo;ve also been able to do more thoughtful, careful work;
with the  extra time I&rsquo;ve been able to make my software more robust and reliable
than it&rsquo;s ever been. The variety of projects I can invest my time into has also
increased substantially, with what was once relegated to minor curiosities now
receiving a similar amount of attention as my larger projects were receiving in
my spare time before. I can work from anywhere in the world, at any time, not
worrying about when to take time off and when to put my head down and crank out
a lot of code.</p>
<p>The frustrations are numerous, as well. I often feel like I&rsquo;ve bit off more than
I can chew. This has been the default state of affairs for me for a long time;
I&rsquo;m often neglecting half of my projects in order to obtain progress by leaps
and bounds in just a few. Working on FOSS full-time has cast this model&rsquo;s
disadvantages into greater relief, as I focus on a greater breadth of projects
and spend more time on them.</p>
<p>The attention and minor fame I&rsquo;ve received as a result of my prolific efforts
also has profound consequences. On the positive line of thought, I&rsquo;m somewhat
embarrassed to admit that I&rsquo;ve noticed my bug reports and feature requests on
random projects (or even my own projects) being taken more seriously now, which
is almost certainly more related to name recognition than merit. I often receive
thanks and words of admiration from my&hellip; fans? I guess I have those now.
Sometimes these are somewhat unwelcome, with troubled individuals writing
difficult to decipher half-rants laden with strange praises and bizarre
questions. Other times I&rsquo;m asked out of the blue to join a discussion I was
unaware of, to comment on some piece of technology I&rsquo;ve never used or to take a
stand on some argument which I wasn&rsquo;t privy to. I don&rsquo;t enjoy these kinds of
comments. But, they&rsquo;re not far removed from the ones I like - genuine,
thoughtful praise arrives in my inbox fairly often and it makes the job a lot
more worthwhile.</p>
<p>Of course, a similar sort of person exists on the opposite extreme. There are
many people who hate my guts and anything I&rsquo;ve ever worked on, and who&rsquo;ll go out
of their way to let me and anyone else who&rsquo;ll listen to them know how they feel.
Of course, I have <em>earned</em> the ire of no small number of people, and I regret
many of these failed interpersonal relationships. These cases are in the
minority, however - most of the people who will tell tales of my evil are people
who I&rsquo;ve never met. There&rsquo;s a lot of spaces online that I just won&rsquo;t visit
anymore because of them. As for the less extreme of this sort of person, I&rsquo;ll
also reiterate what others have said - the negative effects of entitled,
arrogant, or outright toxic users is profound. Don&rsquo;t be that person.</p>
<p>In either case, I can never join new communities on the same terms as anyone
else does. At least one person in every new community already has some
preconception of me when I arrive. Often I think about making an alias just to
enjoy the privilege of anonymity again.</p>
<p>A great help has been my daily interactions with the many friends and colleagues
who are dear to me. I&rsquo;ve made lifelong friends of many of the people I&rsquo;ve met
through these projects.  Thanks to FOSS, I have met an amazing number of kind,
talented, generous people.  Every day, I&rsquo;m thankful to and amazed by the
hundreds of people who have found my ideas compelling, and who come together to
contribute their own ideas and set aside their precious time to work together
realizing our shared dreams. If I&rsquo;m feeling blue, often all it takes to snap me
out of it is to reflect on the gratitude I feel for these wonderful people. I&rsquo;ll
never be able to thank my collaborators enough, but hell, I could stand to do it
some more anyway.</p>
<p>I also have mixed feelings about how <em>busy</em> I am. Every day I wake up to a
hundred new emails, delete half of them, and spend 3-4 hours working on the
rest. Patches, questions, support inquiries, monitoring &amp; reports, it&rsquo;s endless.
On top of that, I have dozens of things I already need to work on. The CI work
distribution algorithm needs to be completely redone; I need to provision new
hardware — oh yeah, and, the hardware that I need ran into shipping
issues, again; I need to improve monitoring; I need to plan for FOSDEM; I need
to finish the Wayland book; I need to figure out the memory issues in himitsu
— not to mention write the rest of the software; I need to file taxes,
twice as much work when you own a business; I need to implement data export
&amp; account deletion; I need to finish the web-driven patch review UI; I need
to finish writing docs for Alpine; I have to work more on the PinePhone; I have
a legacy server which needs to be overhauled and is now on the clock because of
ACMEv1; names.sr.ht needs to be finished&hellip;</p>
<p>Not to mention the tasks which have been on hold for longer now than they&rsquo;ve
been planned for in the first place. Alpine is still going to have hundreds of
Python 2 packages by EoL; the ppc64le server is gathering dust in the
datacenter; there&rsquo;s been some bug with fosspay for several months, in which it
doesn&rsquo;t show Patreon figures unless I reboot the process every now and then;
RISC-V work is stalled because the work is currently blocked by a large problem
that I can&rsquo;t automate; the list of blog posts I want to write is well over 100
entries long. There are <em>several dozen</em> other loose ends I haven&rsquo;t mentioned
here but am painfully aware of anyway.</p>
<p>That&rsquo;s not even considering any personal goals, which I have vanishingly little
time for. I get zero exercise, and though my diet is mostly reasonable the
majority of it is delivery unless I get the odd 2 hours to visit the grocery
store. That is, unless I want to spend those 2 hours with my friends, which
means it&rsquo;s back to delivery. My dating life is almost nonexistent. I want to
spend more time studying Japanese, but it&rsquo;s either that or keeping up with my
leisure reading. Lofty goals of also studying Chinese or Arabic are but dust in
the wind. I&rsquo;m addicted to caffeine, again.</p>
<p>There have been healthy ways and unhealthy ways of dealing with the occasional
feelings of being overwhelmed by all of this. The healthier ways have included
taking walks, reading my books, spending a few minutes with my cat, doing
chores, and calling my family to catch up. Less healthy ways have included
walking to the corner store to buy unhealthy comfort foods, consuming alcohol or
weed too much or too often, getting in stupid internet arguments, being mean to
my friends and colleagues, and googling myself to read negative comments.</p>
<p>Despite being swamped with all of this work, it&rsquo;s all work that I love. I love
writing code, and immeasurably more so when writing <em>my</em> code. Sure, there are
tech debt skeletons in the closet here and they&rsquo;re keeping me awake at night, but
on the whole I feel lucky to be able to write the software I want to write, the
way I want to write it. I&rsquo;ve been trying to do that my entire life —
writing code for someone else has always been a huge drain on my emotional
well-being.  That&rsquo;s why I worked on my side projects in the first place, to have
an outlet through which I could work on self-directed projects without making
compromises for some arbitrary deadline.</p>
<p>When I&rsquo;m in the zone, writing lots of code for a project I&rsquo;m interested in,
knowing it&rsquo;s going to have a meaningful impact on my users, knowing that it&rsquo;s
being written under my terms, it&rsquo;s the most rewarding work I&rsquo;ve ever done. I get
to do that every day.</p>
<p>This isn&rsquo;t the retrospective I wanted to write, but it&rsquo;s nice to drop the veneer
for a few minutes and share an honest take on what this is like. This year has
been nothing like what I expected it to be - it&rsquo;s both terrible and wonderful
and very busy, very goddamn busy. In any case, I&rsquo;m extremely grateful to be here
doing it, and it&rsquo;s thanks to many, many supportive people - users, contributors,
co-maintainers, and friends. Thank you, thank you, thank you, thank you.</p>


guid: https://drewdevault.com/2020/01/27/KnightOS-was-interesting.html
title: KnightOS was an interesting operating system
url: https://drewdevault.com/2020/01/27/KnightOS-was-interesting.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p><a href="https://knightos.org">KnightOS</a> is an operating system I started writing about
10 years ago, for Texas Instruments line of z80 calculators — the TI-73,
TI-83+, TI-84+, and similar calculators are supported. It still gets the rare
improvements, but these days myself and most of the major contributors are just
left with starry eyed empty promises to themselves that one day they&rsquo;ll do one
of those big refactorings we&rsquo;ve been planning&hellip; for 4 or 5 years now.</p>
<p>Still, it was a really interesting operating system which was working under some
challenging constraints, and overcame them to offer a rather nice Unix-like
environment, with a filesystem, preemptive multiprocessing <em>and</em> multithreading,
assembly and C programming environments, and more. The entire system was written
in handwritten z80 assembly, almost 50,000 lines of it, on a compiler toolchain
we built from scratch.</p>
<p>There was only 64 KiB of usable RAM. The kernel stored <em>all</em> of its state in
1024 bytes of statically allocated RAM. Many subsystems used overlapping parts
of this memory, which was carefully planned for to avoid conflicts. The
userspace memory allocator used a simple linked list for tracking allocations,
to minimize the overhead of each allocation and maximize the usable space for
userspace programs. There was no MMU in the sense that we have on modern
computers, so any program could freely overwrite any other programs. In fact,
the &ldquo;userspace&rdquo; task switching GUI would read the kernel&rsquo;s process table
directly to make a list of running programs.</p>
<p>The non-volatile storage was NOR Flash, which presents some interesting
constraints. In the worst case we only had 512 KiB of storage, and even in the
best case just 4MiB (this for a device released in 2013). This space was shared
with the kernel, whose core code was less than 4KiB, and including high-address
subsystems still clocked in at less than 16KiB. Due to the constraints of NOR
Flash, a custom filesystem was designed which did all daily operations by only
<em>resetting</em> bits in the underlying storage. In order to <em>set</em> any bits, we had
to set the entire 64 KiB sector to 1. Overhead was also kept to a bare minimum
here to maximize storage space available to users.</p>
<p>Writing to Flash storage also renders it unreadable while the operation is in
progress. The kernel normally executes directly from Flash, resident at the
bottom of the memory. Therefore, in order to modify Flash, the kernel&rsquo;s Flash
driver copies part of itself to RAM, jumps to it, and then jumps back after the
operation is complete. Recall that all of the kernel&rsquo;s memory is statically
allocated, and there&rsquo;s not much of it — we used only 128 bytes for the
code which runs in RAM, and it&rsquo;s shared with some other stuff that we had to
plan around. In order to meet these constraints, we employ <em>self modifying code</em>
— the Flash driver copies some of itself into RAM, then pre-computes some
information and <em>modifies</em> that machine code in-place before jumping to it.</p>
<p>We also had some basic networking support. The calculator has a 2.5mm jack,
similar to headphone jacks — if you had a 3.5mm adapter, we had a music
player which would play MIDI or WAV files. The kernel had direct control of the
voltages on the ring and tip, and had to bitbang them directly in software<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.
Based on this we built some basic networking support, which supported
calculator-to-calculator and calculator-to-PC information exchange. Later models
had a mini-USB controller (which, funnily enough, can also be bitbanged in
software), but we never ended up writing a driver for it.</p>
<p>The KnightOS kernel also includes some code which is the first time I ever wrote
<a href="https://github.com/KnightOS/kernel/blob/e257f54e021ee743306a2a4a5a152860728fb3f8/src/00/restarts.asm#L129-L130">&ldquo;here be dragons&rdquo;</a>
into a comment, and I don&rsquo;t think I&rsquo;ve topped it since.</p>
<p>Despite these constraints, KnightOS is completely booted up to a useful
Unix-like (with a graphical interface) faster than you can lift your finger off
of the power button. The battery could last the entire semester, if you&rsquo;re
lucky. Can the device you&rsquo;re reading this on claim the same?<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup></p>
<p><video controls src="https://yukari.sr.ht/knightos.webm"></video></p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Newer hardware revisions had some support hardware which was capable of transferring a single byte without software intervention. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>The device I&rsquo;m writing this blog post with is 3500× faster than my calculator, has 262,144× more RAM, and 2.1×10<sup>6</sup> times more storage space. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/02/06/Dependencies-and-maintainers.html
title: Dependencies and maintainers
url: https://drewdevault.com/2020/02/06/Dependencies-and-maintainers.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I&rsquo;m 34,018 feet over the Atlantic at the moment, on my way home from FOSDEM. It
was as always a lovely event, with far too many events of interest for any
single person to consume. One of the few talks I was able to attend<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> left a
persistent worm of thought in my brain. This talk was put on by representatives
of Microsoft and GitHub and discusses whether or not there is a sustainability
problem in open source (<a href="https://fosdem.org/2020/schedule/event/foss_sustainability_issues/">link</a>). The content of the talk,
interpreted within the framework in which it was presented, was moderately
interesting. It was more fascinating to me, however, as a lens for interpreting
GitHub&rsquo;s (and, indirectly, Microsoft&rsquo;s) approach to open source, and of the
mindset of developers who approach problems in the same ways.</p>
<p>The presenters drew attention to a few significant crises open-source
communities have faced in recent years — left-pad, in which a trivial
library was removed from npm and was unknowingly present in thousands of
dependency graphs; event-stream, in which a maintainer transferred project
ownership to an unknown individual who added crypto mining; and heartbleed, in
which a bug in a critical security library caused mass upgrades and panic
— and asks whether or not these can be considered sustainability issues.
The talk has a lot to dissect and will frame my thinking and writings for a
while. Today I&rsquo;ll focus on one specific problem, which I called attention to
during the Q&amp;A.</p>
<p>At a few points, the presenters spoke from the perspective of a business which
depends on up to thousands of open-source libraries or tools. In such a context,
how do you prioritize which of your thousands of dependencies requires
attention, for financial support, contributions upstream, and so on? I found
this worldview dissonant, and asked the following question: &ldquo;why do you have
thousands of dependencies in the first place?&rdquo; Because this approach seems to be
fast becoming the norm, this may seem like a stupid question.<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> If any Node
developers are reading, scan through your nearest node_modules directory and see
how many of these dependencies you&rsquo;ve even heard of.</p>
<p>Such an environment is primed to fail in the ways enumerated by this talk.
Consider the case of the maintainer who lost interest and gave their project to
an untrusted third party. If I had depended on this library, I would have
noticed long ago that the project was effectively unmaintained. It&rsquo;s likely that
I or my peers would have sent patches to this project, given that bugfixes would
have stopped coming from upstream. We would be aware of the larger risk this
project posed, and have studied alternatives. Earlier than that, I would
probably have lent my ear to the maintainer to vent their frustrations, and
offered my help where possible.</p>
<p>For most of my projects, I can probably list the entire dependency graph,
including transitive dependencies, off of the top of my head. I can name most of
their maintainers, and many of their contributors. I have shaken the hands of
these people, shared drinks and meals with them, and count many of them among my
close friends. The idea of depending on a library I&rsquo;ve never heard of, several
degrees removed via transitive dependencies, maintained by someone I&rsquo;ve never
met and have no intention of speaking to, is <em>absolutely nuts</em> to me. I know of
these problems well in advance because I know the people affected as my friends.
If someone is frustrated or overworked, I&rsquo;m right there with them trying to find
solutions and correct the over-burden. If someone is in dire financial
straights, I&rsquo;m helping them touch up their resume and introducing them to
companies that I know are looking for their skillset, or helping them work on
more sustainable sources of donations and grants. They do the same for me, and
for each other.</p>
<p>Quit treating open-source projects like a black box that conveniently solves
your problem. Engage with the human beings who work on it, participate in the
community, and <em>make</em> it healthy and sustainable. You shouldn&rsquo;t be surprised
when the 3 AM alarm goes off if the most you see of a project is a line in your
<code>package.json</code>.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>And strictly speaking I even had to slip in under the radar to attend in the first place — the room was full. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>If so, you may be pleased by a Microsoft&rsquo;s ridiculous answer: &ldquo;we have 60,000 developers, that&rsquo;s why.&rdquo; <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/02/15/Status-update-February-2020.html
title: Status update, February 2020
url: https://drewdevault.com/2020/02/15/Status-update-February-2020.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Today I thought it&rsquo;d try out something new: I have an old family recipe
simmering on the stove right now, but instead of beef I&rsquo;m trying out impossible
beef. It cooked up a bit weird — it doesn&rsquo;t brown up in the same way I
expect of ground beef, and it made a lot more fond than I expected. Perhaps the
temperature is too high? We&rsquo;ll see how it fares when it&rsquo;s done. In the
meanwhile, let&rsquo;s get you up to speed on my free software projects.</p>
<p>First, big thanks to everyone who stopped by to say &ldquo;hello&rdquo; at FOSDEM! Putting
faces to names and getting to know everyone on a personal level is really
important to me, and I would love FOSDEM even if that was all I got out of it.
Got a lot of great feedback on the coming plans for SourceHut and aerc, too.</p>
<p>That aside, what&rsquo;s new? On the Wayland scene, the long-promised Sway 1.3^W1.4
was finally released, and with it wlroots 0.10.0. I&rsquo;ve been talking it up for a
while, so I won&rsquo;t bore you by re-listing all of the cool stuff in this release -
it&rsquo;ll suffice to say that I think you&rsquo;ll enjoy it. The related tools —
swayidle, swaylock, swaybg — all saw releases around the same time. The
other release this month was scdoc 1.10.1, which was a simple patch release.
Beyond releases, there&rsquo;s been some Wayland development work as well: wev
received a simple bugfixes, and casa&rsquo;s OpenGL-based renderer rewrite has been
completed nicely.</p>
<p>aerc progresses nicely this month as well, thanks to the support of its many
dedicated contributors. Many bugfixes have landed, alongside contextual
configuration options — so you can have different config settings, for
example, when you have an email selected whose subject line matches a regex. A
series of notmuch patches should be landing soon as well. himitsu has also seen
slow progress — this pace being deliberate, as this is security-sensitive
software. Several bugs have been fixed in the existing code, but there are a few
more to address still. imrsh also had a little bit of improvement this month,
as I started down the long road towards properly working UTF-8 support.</p>
<p>SourceHut improvements have also landed recently. I did some work shoring up our
accessibility standards throughout the site, and SourceHut is now fully
complaint with the WCAG accessibility guidelines. We now score 100% on standard
performance, accessibility, and web standards compliance tests. SourceHut is the
lightest weight, most usable forge. I recently fixed a bug report from a Lynx,
user, too 😉 In terms of feature development, the big addition this month is
support for attaching files to annotated git tags, so you can attach binaries,
PGP signatures, and so on to your releases. More cool SourceHut news is coming
in the post to sr.ht-announce later today.</p>
<p>This month&rsquo;s update is a little bit light on content, I&rsquo;ll admit. Between FOSDEM
and taking some personal time, I&rsquo;ve had less time for work this month. However,
there&rsquo;s another reason: I have a new secret project which I&rsquo;ve been working on.
I intend to keep this project under wraps for a while still, because I don&rsquo;t
want people to start using it before I know if it&rsquo;s going to pan out or not.
This project is going to take a lot of time to complete, so I hope you&rsquo;ll bear
with me for a while and trust that the results will speak for themselves. As
always, thank you for your support, and I&rsquo;m looking forward to another month of
awesome FOSS work.</p>


guid: https://drewdevault.com/2020/02/18/Fucking-laptops.html
title: Fucking laptops
url: https://drewdevault.com/2020/02/18/Fucking-laptops.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>The best laptop ever made is the ThinkPad X200, and I have two of them. The
caveats are: I get only 2-3 hours of battery life even with conservative use;
and it struggles to deal with 1080p videos.</p>
<p>The integrated GPU, Bluetooth and WiFi, internal sensors, and even the
fingerprint reader can all be driven by the upstream Linux kernel. In fact, the
hardware is so well understood that I have successfully used almost all of the
laptop&rsquo;s features on Linux, FreeBSD, NetBSD, Minix, Haiku, and Plan 9. Plan
fucking 9. It can run coreboot, too. The back of the laptop has all of the
screws (Phillips head) labelled so you know which to remove to service which
parts. User replacable parts include the screen, keyboard (multiple layouts are
available and are interchangeable), the RAM, hard drive (I put a new SSD in one
of mine a few weeks ago, and it took about 30 seconds) — actually, there
are a total of 26 replacable parts in this laptop.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> There is a detailed
278-page service manual to assist you or your local repair tech in addressing
any problems that arise.</p>
<p>They&rsquo;re quite durable, too. Mine still looks like it just rolled off the
assembly line yesterday. In fact, it was built 12 years ago.</p>
<p>The X200 was made in 2008. In the time since, the modern laptops' battery life
and video decoding performance has improved. In every other respect, the market
is regressive, half-assed garbage.</p>
<p>I am usually near power, so I&rsquo;ve been reasonably happy even with the pithy
battery life of the X200. I also have a T520, which sucks in its own way<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>,
but can decode 1080p videos just fine. I generally don&rsquo;t need a lot of power -
compiling most programs is fast enough that I don&rsquo;t really notice, especially
with incremental compilation, and for any large workloads I just SSH out into a
build server somewhere. However, I&rsquo;ve been planning some astronomy outings
lately, and the battery life matters for this - so I was looking for a laptop I
could run <a href="https://stellarium.org/">Stellarium</a> on to drive my telescope into
the wee hours of the night.</p>
<p>It has since come to my attention that in 2020, every laptop <em>still</em> fucking
sucks. Even the ones people pretend to like have crippling, egregious flaws. The
Dell XPS series has a firmware so bad that its engineers should be strung up in
the town square for building it - if yours works, it&rsquo;s because you were <em>lucky</em>.
System76 laptops are bulky and priced at 2x or 3x what they&rsquo;re worth. Same goes
for Purism, plus a company I have no desire to support any longer, and they&rsquo;re
out of stock anyway. Pine64 requires nonfree blobs, patched kernels, and booting
up ARM devices is a fucking nightmare, and they&rsquo;re out of stock anyway. The Star
Lite looks promising, but they&rsquo;re out of stock too. Huewei laptops are shameless
Macbook ripoffs with the same shitty keyboards, and you can&rsquo;t buy them in the US
anyway. Speaking of Macbooks, even Apple fanboys are fed up with them these
days.</p>
<p>The laptop market is in an atrocious state, folks. If you work at any of these
companies and you&rsquo;re proud of the garbage you&rsquo;re shipping, then I&rsquo;m disappointed
in you. Come on, let&rsquo;s get our shit together and try to make a laptop which is
<em>at least</em> as good as the 12 year-old one I&rsquo;m stuck with now.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Which just means you can basically take the entire thing apart and replace almost any part. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>It barely gets an hour and a half of battery life on a good day. And there&rsquo;s an Nvidia optimus GPU, which is just, ugh. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/02/21/Thoughts-on-performance.html
title: Thoughts on performance & optimization
url: https://drewdevault.com/2020/02/21/Thoughts-on-performance.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>The idea that programmers ought to or ought not to be called &ldquo;software
engineers&rdquo; is a contentious one. How you approach optimization and performance
is one metric which can definitely push my evaluation of a developer towards the
engineering side. Unfortunately, I think that a huge number of software
developers today, even senior ones, are approaching this problem poorly.</p>
<p>Centrally, I believe that you cannot effectively optimize a system which you do
not understand. Say, for example, that you&rsquo;re searching for a member of a
linked list, which is an O(n) operation. You know this can be improved by
switching from a linked list to a sorted array and using a binary search. So,
you spend a few hours refactoring, commit the implementation, and&hellip; the
application is no faster. What you failed to consider is that the lists are
populated from data received over the network, whose latency and bandwidth
constraints make the process much slower than any difference made by the kind of
list you&rsquo;re using.  If you&rsquo;re not optimizing your bottleneck, then you&rsquo;re
wasting your time.</p>
<p>This example seems fairly obvious, and I&rsquo;m sure you, our esteemed reader, would
not have made this mistake. In practice, however, the situation is usually more
subtle. Thinking about your code really hard, making assumptions, and then
acting on them is not the most effective way to make performance improvements.
Instead, we apply the scientific method: we think really hard, <em>form a
hypothesis</em>, make predictions, test them, and then apply our conclusions.</p>
<p>To implement this process, we need to describe our performance in factual terms.
All software requires a certain amount of resources — CPU time, RAM, disk
space, network utilization, and so on. These can also be described over time,
and evolve as the program takes on different kinds of work. For example, we
could model our program&rsquo;s memory use as bytes allocated over time, and perhaps
we can correlate this with different stages of work — &ldquo;when the program
starts task C, the rate of memory allocation increases by 5MiB per second&rdquo;. We
identify bottlenecks — &ldquo;this program&rsquo;s primary bottleneck is disk I/O&rdquo;.
When we hit performance problems, then we know that we need to upgrade to SSDs,
or predict what reads will be needed later and prep them in advance, cache data
in RAM, etc.</p>
<p>Good optimizations are based on factual evidence that the program is not
operating within its constraints in certain respects, then improving on those
particular problems. You should always conduct this analysis before trying to
solve your problems. I generally recommend conducting this analysis in advance,
so that you can predict performance issues before they occur, and plan for them
accordingly. For example, if you know that your disk utilization grows by 2 GiB
per day, and you&rsquo;re on a 500 GiB hard drive, you&rsquo;ve got about 8 months to plan
your next upgrade, and you shouldn&rsquo;t be surprised by an ENOSPC when the time
comes.</p>
<p>For CPU bound tasks, this is also where a general understanding of the
performance characteristics of various data structures and algorithms is useful.
When you know you&rsquo;re working on something which <em>will become</em> the application&rsquo;s
bottleneck, you would be wise to consider algorithms which can be implemented
efficiently. However, it&rsquo;s equally important to re-prioritize performance when
you&rsquo;re not working on your bottlenecks, and instead consider factors like
simplicity and conciseness more seriously.</p>
<p>Much of this will probably seem obvious to many readers. Even so, I think the
general wisdom described here is institutional, so it&rsquo;s worth writing down. I
also want to call out some specific behaviors that I see in software today which
I think don&rsquo;t take this well enough into account.</p>
<p>I opened by stating that I believe that you cannot effectively optimize a system
which you do not understand. There are two groups of people I want to speak to
with this in mind: library authors (especially the standard library), and
application programmers. There are some feelings among library authors that
libraries should be fairly opaque, and present high-level abstractions over
their particular choices of algorithms, data structures, and so on. I think this
represents a fundamental lack of trust with the programmer downstream. Rather
than write idiot-proof abstractions, I think it&rsquo;s better to trust the downstream
programmer, explain to them how your system works, and equip them with the tools
to audit their own applications. After all: your library is only a small
component of <em>their</em> system, not yours — and you cannot optimize a system
you don&rsquo;t understand.</p>
<p>And to the application programmer, I urge you to meet your dependencies in the
middle. Your entire system is your responsibility, including your dependencies.
When the bottleneck lies in someone else&rsquo;s code, you should be prepared to dig
into their code, patch it, and send a fix upstream, or to restructure your code
to route the bottleneck out. Strive to understand how your dependencies, up to
and including the stdlib, compiler, runtime, kernel, and so on, will perform in
your scenario. And again to the standard library programmer: help them out by
making your abstractions thin, and your implementations simple and debuggable.</p>


guid: https://drewdevault.com/2020/03/03/Abiopause.html
title: The Abiopause
url: https://drewdevault.com/2020/03/03/Abiopause.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>The sun has an influence on its surroundings. One of these is in the form of
small particles that are constantly ejected from the sun in all directions,
which exerts an outward pressure, creating an expanding sphere of particles that
moves away from the sun. These particles are the solar wind. As the shell of
particles expands, the density (and pressure) falls. Eventually the solar wind
reaches the <em>interstellar medium</em> — the space between the stars —
which, despite not being very dense, is not empty. It exerts a pressure that
pushes inwards, towards the sun.</p>
<p>Where the two pressures balance each other is an interesting place. The sphere
up to this point is called the <em>heliosphere</em> — which can be roughly
defined as the zone in which the influence of the sun is the dominant factor.
The <em>termination shock</em> is where the change starts to occur. The plasma from the
sun slows, compresses, and heats, among other changes. The physical interactions
here are interesting, but aren&rsquo;t important to the metaphor. At the
termination shock begins the <em>heliosheath</em>. This is a turbulent place where
particles from the sun and from the interstellar medium mix. The interactions in
this area are complicated and interesting, you should read up about it later.</p>
<p><img src="https://legacy.sr.ht/_FIT.svg" alt="Picture of a faucet pouring into a sink"></p>
<div class="text-center">
  <small>Yanpas via Wikimedia Commons, CC-BY-SA</small>
</div>
<p>Finally, we reach the <em>heliopause</em>, beyond which the influence of the
interstellar medium is dominant. Once crossing this threshold, you are said to
have left the solar system. The Voyager 1 space probe, the first man-made object
to leave the solar system, crossed this point on August 25th, 2012. Voyager 2
completed the same milestone on November 12th, 2018<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</p>
<p>In the world of software, the C programming language clearly stands out as the
single most important and influential programming language.  Everything
forming the critical, foundational parts of your computer is written in it:
kernels, drivers, compilers, interpreters, runtimes, hypervisors, databases,
libraries, and more are almost all written in C.<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> For this reason, any
programming language which wants to get anything useful done is certain to
support a C FFI (foreign function interface), which will allow programmers to
communicate with C code from the comfort of a high-level language. No other
language has the clout or ubiquity to demand this level of deference from
everyone else.</p>
<p>The way that an application passes information back and forth with its
subroutines is called its <em>ABI</em>, or application binary interface. There are a
number of ABIs for C, but the most common is the System-V ABI, which is used on
most modern Unix systems. It specifies details like which function parameters to
put in which registers, what goes on the stack, the structure and format of
these values, and how the function returns a value to the caller. In order to
interface with C programs, the FFI layers in other programs have to utilize this
ABI to pass information to and from C functions.</p>
<p>Other languages often have their own ABIs. C, being a different programming
language from $X, naturally has different semantics. The particular semantics of
C don&rsquo;t necessarily line up to the semantics the language designers want $X to
have, so the typical solution is to define functions with C &ldquo;linkage&rdquo;, which
means they&rsquo;re called with the C ABI. It&rsquo;s from this that we get keywords like
<code>extern &quot;C&quot;</code> (C++, Rust), <code>export</code> in Go, <code>[DllImport]</code> in C#, and so on.
Naturally, these keywords come with a lot of constraints on how the function
works, limiting the user to the mutually compatible subset of the two ABIs, or
else using some kind of translation layer.</p>
<p>I like to think of the place where this happens as the &ldquo;abiopause&rdquo;, and draw
comparisons with the solar system&rsquo;s heliopause. Within the &ldquo;abiosphere&rdquo;, the
programming language you&rsquo;re using is the dominant influence. The idioms and
features of the language are used to their fullest extent to write idiomatic
code. However, the language&rsquo;s sphere of influence is but a bubble in a sea of C
code, and the interface between these two areas of influence is often quite
turbulent. Directly using functions with C linkage from the abiosphere is not
pleasant, as the design of good C APIs do not match the semantics of good
$X APIs. Often there are layers to this transition, much like our solar
system, where some attempt is made to wrap the C interface in a more idiomatic
abstraction.</p>
<p>I don&rsquo;t really like this boundary, and I think most programmers who have worked
here would agree. If you like C, you&rsquo;re stuck either writing bad C code or using
poorly-suited tools to interface badly with an otherwise good API. If you like
$X, you&rsquo;re stuck writing very non-idiomatic $X code to interface with a foreign
system. I don&rsquo;t know how to fix this, but it&rsquo;s interesting to me that the
&ldquo;abiopause&rdquo; appears to be an interface full of a similar turbulence and
complexity as we find in the heliopause.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>It took longer because Voyager 2 went on to see Uranus and Neptune. Voyager 1 just swung around Saturn and was shot directly up and out of the solar system. Three other man-made objects are currently on trajectories which will leave the solar system. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Even if you don&rsquo;t like C, it would be ridiculous to dismiss its influence and importance. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/03/07/Open-letter-to-Senator-Casey.html
title: An open letter to Senator Bob Casey on end-to-end encryption
url: https://drewdevault.com/2020/03/07/Open-letter-to-Senator-Casey.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>To Senator Bob Casey, I&rsquo;m writing this open letter.</p>
<p>As your constituent, someone who voted for you in 2018, and an expert in
software technology, I am disappointed in your support of the EARN IT Act. I am
aware that encryption is a challenging technology to understand, even for us
software engineers, and that it raises difficult problems for the legislature.
The EARN IT Act does not protect our children, and it has grave implications for
the freedoms of our citizens.</p>
<p>The mathematics underlying strong end-to-end encryption have been proven to be
unbreakable. Asking service providers to solve them or stop using it is akin to
forcing us to solve time travel or quit recording history. Banning the use of a
technology without first accomplishing a sisyphean task is equivalent to banning
the technology outright. Ultimately, these efforts are expensive and futile. The
technology necessary to implement unbreakable encryption can be described
succinctly on a single 8.5&quot;x11&quot; sheet of paper. I would be happy to send such a
paper to your office, if you wish. The cat is out of the bag: encryption is not
a secret, and its use to protect our citizens is a widespread industry standard.
Attempting to ban it is equivalent to trying to ban algebra or trigonometry.</p>
<p>Citizen use of end-to-end encryption is necessary to uphold our national
security. One way that child abuse material is often shared is via the Tor
secure internet network. This system utilizes strong end-to-end encryption to
secure the communications of its users, which makes it well-suited to hiding
the communications of child abusers. However, the same guarantees that enable
the child abusers to securely share materials are also essential for
journalists, activists, watchdog groups - and for our national security. The
technology behind Tor was designed by the US Navy and DARPA and the ability for
the public to use it to secure their communications is essential to the
network&rsquo;s ability to delivery on its national security guarantees as well.</p>
<p>Protecting our children is important, but this move doesn&rsquo;t help. Breaking
end-to-end encryption is no substitute for good police work and effective
courts. Banning end-to-end encryption isn&rsquo;t going to make it go away - the
smart criminals are still going to use it to cover their tracks, and law
enforcement still needs to be prepared to solve cases with strong encryption
involved. Even on the Tor network, where strong end-to-end encryption is
utilized, many child abusers have been caught and brought to justice thanks to
good investigative work. It&rsquo;s often difficult to conduct an investigation within
the limits of the law and with respect to the rights of our citizens, but it&rsquo;s
necessary for law enforcement to endure this difficulty to protect our freedom.</p>
<p>End-to-end encryption represents an important tool for the preservation of our
fundamental rights, as enshrined in the bill of rights. Time and again, our
alleged representatives levy attacks on this essential technology. It doesn&rsquo;t
get any less important each time it&rsquo;s attacked - rather, the opposite seems to
be true. On the face of it, the EARN IT Act appears to use important and morally
compelling problems of child abuse as a front for an attack on end-to-end
encryption. Using child abuse as a front to attack our fundamental right to
privacy is reprehensible, and I&rsquo;m sure that you&rsquo;ll reconsider your position.</p>
<p>As freedom of the press is an early signal for the failure of democracy and rise
of tyranny, so holds for the right to encrypt. I am an American, I am free to
speak my mind. I am free to solve a simple mathematical equation which
guarantees that my thoughts are shared only with those I choose. The right to
private communications is essential to a functioning democracy, and if you claim
to represent the American people, you must work to defend that right.</p>


guid: https://drewdevault.com/2020/03/13/GitHub-notifications.html
title: GitHub's new notifications: a case of regressive design
url: https://drewdevault.com/2020/03/13/GitHub-notifications.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p><em>Disclaimer: I am the founder of a company which competes with GitHub. However,
I still use tools like GitHub, GitLab, and so on, as part of regular
contributions to projects all over the FOSS ecosystem. I don&rsquo;t dislike GitHub,
and I use it frequently in my daily workflow.</em></p>
<p>GitHub is rolling out a new notifications UI. A few weeks ago, I started seeing
the option to try it. Yesterday, I received a warning that the old UI will soon
be deprecated. At this pace, I would not be surprised to see the new UI become
mandatory in a week or two. I&rsquo;m usually optimistic about trying out new
features, but this change worried me right away. I still maintain a few projects
on GitHub, and I frequently contribute to many projects there. Using the
notification page to review these projects is a ritual I usually conduct several
times throughout the workday. So, I held my breath and tried it out.</p>
<p>The new UI looks a lot more powerful initially. The whole page is used to
present your notifications, and there are a lot more buttons to click, many of
them with cute emojis to quickly convey meaning. The page is updated in
real-time, so as you interact with the rest of the website your notifications
page in the other tab will be updated accordingly.</p>
<p>Let&rsquo;s stop and review my workflow using the <em>old</em> UI. I drew this beautiful
graphic up in GIMP to demonstrate:</p>
<p><img src="https://cmpwn.com/system/media_attachments/files/000/659/354/original/d9abc4befe1a074c.png" alt=""></p>
<p>I open the page, then fix my eyes on the notification titles. I move my mouse to
the right, and while reading titles I move the mouse down, clicking to mark any
notifications as read that I don&rsquo;t need to look at, and watching in my
peripheral vision to see that the mouse hits its mark over the next button. The
notifications are grouped by repository, so I can read the name of the repo then
review all of its notifications in one go. The page is fairly narrow, so reading
the titles usually leads my eyes naturally into reading any other information I
might need, like the avatars of participants or age of the notification.</p>
<p>I made an equally beautiful picture for the new UI<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>:</p>
<p><img src="https://cmpwn.com/system/media_attachments/files/000/659/353/original/b15f20de0ae35cd3.png" alt=""></p>
<p>This one is a lot harder to scan quickly or get into your muscle memory. The
title of the notification no longer stands out, as it&rsquo;s the same size as the
name of the repo that was affected. They&rsquo;re no longer grouped by repo, either,
so I have to read both every time to get the full context. I then have to move
my eyes <em>all the way</em> across the page to review any of those other details,
through vast fields of whitespace, where I can easily lose my place and end up
on a different row.</p>
<p>Once I&rsquo;ve decided what to do with it, I have to move my mouse over the row, and
wait for the action buttons to appear. They were invisible a second ago, so I
have to move my mouse again to get closer to the target. Clicking it will mark
it as read. Then, because I have it filtered to unread (because &ldquo;all&rdquo;
notifications is really <em>all</em> notifications, and there&rsquo;s no &ldquo;new&rdquo; notifications
like the old UI had), the row disappears, making it difficult to undo if it was
a mistake. Then I heave my eyes to the left again to read the next one.</p>
<p>This page is updated in real-time. In the old UI, after I had marked everything
as read that I didn&rsquo;t need to look at, I would middle click on each remaining
notification to open it in a new tab. On the new real-time page, as soon as the
other tab loads, the notification I clicked disappears (again, because I have it
filtered to &ldquo;unread&rdquo;). This isn&rsquo;t immediate, though — it takes at least as
long as it takes for the new tab to load. Scanning the list and middle-clicking
every other message becomes a Sisyphean task.</p>
<p>And the giant sticky header that follows you around! A whole 160 pixels, 14% of
my vertical space, is devoted to a new header which shows up on the next page
when I follow through a notification. And it&rsquo;s implemented with JavaScript and
done in a bizzare way, so writing a user style to get rid of it was rather
difficult.</p>
<p>Aside: I tried adding a custom filter to show only pull requests, but it seems
to silently fail, and I just see all of my notifications when I use it.</p>
<hr>
<p>Anyway, we&rsquo;re probably stuck with this. Now that they&rsquo;ve announced the imminent
removal of the old UI, we can probably assume that this feature is on the
non-stop release train. Negative feedback almost never leads to cancelling the
roll-out of a change, because the team&rsquo;s pride is on the line.</p>
<p>I haven&rsquo;t spoken to anyone who likes the new UI. Do you?</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Both of these pictures were sent to GitHub as feedback on the feature, three weeks ago. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/03/18/Reckless-limitless-scope.html
title: The reckless, infinite scope of web browsers
url: https://drewdevault.com/2020/03/18/Reckless-limitless-scope.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Since the first browser war between Netscape and Internet Explorer, web browsers
have been using features as their primary means of competing with each other.
This strategy of unlimited scope and perpetual feature creep is reckless, and
has been allowed to go on for far too long.</p>
<p>I used wget to download all 1,217 of the <a href="https://www.w3.org/TR/">W3C specifications</a>
which have been published at the time of writing<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>, of which web browsers need
to implement a substantial subset in order to provide a modern web experience.
I ran a word count on all of these specifications. How complex would you guess
the web is?</p>
<p>The total word count of the W3C specification catalogue is 114 million words at
the time of writing. If you added the combined word counts of the C11, C++17,
UEFI, USB 3.2, and POSIX specifications, all 8,754 published RFCs, and the
combined word counts of everything on Wikipedia&rsquo;s <a href="https://en.wikipedia.org/wiki/List_of_longest_novels">list of longest
novels</a>, you would be 12 million words short of the W3C
specifications.<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup></p>
<p>I conclude that <strong>it is impossible to build a new web browser</strong>. The complexity
of the web is <em>obscene</em>. The creation of a new web browser would be comparable
in effort to the Apollo program or the Manhattan project.</p>
<p>It is impossible to:</p>
<ul>
<li>Implement the web correctly</li>
<li>Implement the web securely</li>
<li>Implement the web <strong>at all</strong></li>
</ul>
<p>Starting a bespoke browser engine with the intention of competing with Google or
Mozilla is a fool&rsquo;s errand. The last serious attempt to make a new browser,
Servo, has become one part incubator for Firefox refactoring, one part
playground for bored Mozilla engineers to mess with technology no one wants, and
zero parts viable modern web browser. But WebVR is cool, right? Right?</p>
<p>The consequences of this are obvious. Browsers are the most expensive piece of
software a typical consumer computer runs. They&rsquo;re infamous for using all of
your RAM, pinning CPU and I/O, draining your battery, etc. <em>Web browsers are
responsible for more than 8,000 CVEs</em>.<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup></p>
<p>Because of the monopoly created by the insurmountable task of building a
competitive alternative, browsers have also been free to stop being the &ldquo;user
agent&rdquo; and start being the agents of their creators instead. Firefox is filling
up with ads, tracking, and mandatory plugins. Chrome is used as a means for
Google to efficiently track your eyeballs and muscle anti-technologies like DRM
and AMP into the ecosystem.  The browser duopoly is only growing stronger, too,
as Microsoft drops Edge and WebKit falls well behind its competition.</p>
<p>The major projects are open source, and usually when an open-source project
misbehaves, we&rsquo;re able to fork it to offer an alternative. But even this
is an impossible task where web browsers are concerned. The number of W3C
specifications grows at an average rate of 200 new specs per year, or about 4
million words, or about one POSIX every 4 to 6 months. How can a new team
possibly keep up with this on top of implementing the outrageous scope web
browsers already have <em>now</em>?</p>
<p>The browser wars have been allowed to continue for far too long. They should
have long ago focused on competing in terms of performance and stability, not in
adding new web &ldquo;features&rdquo;. This is absolutely ridiculous, and it has to stop.</p>
<p><em>Note: I have prepared a write-up on <a href="https://paste.sr.ht/~sircmpwn/13c1951014a256e9f551296a129bf6d10e9303dc">how I arrived at these word counts</a>.</em></p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Not counting WebGL, which is maintained by Khronos. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>You could fit the 5,038 page Intel x86 ISA manual into the remainder, six times. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Combined search results for CVEs mentioning &ldquo;firefox&rdquo;, &ldquo;chrome&rdquo;, &ldquo;safari&rdquo;, and &ldquo;internet explorer&rdquo;, on cve.mitre.org. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/03/25/Designing-a-replacement-part-for-my-truck.html
title: Designing and 3D printing a new part for my truck
url: https://drewdevault.com/2020/03/25/Designing-a-replacement-part-for-my-truck.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I drove a car daily for many years while I was living in Colorado, California,
and New Jersey, but since I moved to Philadelphia I have not needed a car. The
public transit here is not great, but it&rsquo;s good enough to get where I need to be
and it&rsquo;s a lot easier than worrying about parking a car. However, in the past
couple of years, I have been moving more and more large server parts back and
forth to the datacenter for SourceHut. I&rsquo;ve also developed an interest in
astronomy, which benefits from being able to carry large equipment to remote
places. These reasons, among others, put me into the market for a vehicle once
again.</p>
<p>I think of a vehicle strictly as a functional tool. Some creature comforts are
nice, but I consider them optional. Instead, I prioritize utility. A truck makes
a lot of sense — lots of room to carry things in. And, given my expected
driving schedule of &ldquo;not often&rdquo;, I wasn&rsquo;t looking to spend a lot of money or
get a loan. There are other concerns: modern cars are very complicated machines,
and many have lots of proprietary computerized components which make end-user
maintenance very difficult. Sometimes manufacturers even use cryptography and
legal threats to bring cars into their dealerships, bullying out third-party
repairs.</p>
<p>To avoid these, I got an older truck: a 1984 Dodge D250. It&rsquo;s a much simpler
machine than most modern cars, and learning how to repair and maintain it is
something I can do in my spare time.</p>
<p>It&rsquo;s an old truck, and the previous owners were not negligent, but also didn&rsquo;t
invest a lot of time or money in the vehicle&rsquo;s upkeep. The first problem I hit
was the turn signal lever snapping and becoming slack, which I fixed by pulling
open the steering column, re-aligning the lever, and tightening an internal
screw. The more interesting problem, however, was this:</p>
<p><img src="https://l.sr.ht/OWVw.jpg" alt="Picture of a broken latch on the window over the truck bay"></p>
<p>This plastic part holds an arm in place, which is engaged by a lever in the
center of the window which folds closed over the truck bay. It&rsquo;s used to hold
the window in place and provides a weak locking mechanism. When the arm is
allowed to move freely, it can clang around while I&rsquo;m driving, and can make
opening the truck bay a frustrating procedure. I have been looking for a reason
to learn how to use <a href="http://solvespace.com/index.pl">solvespace</a>, and this
seemed like a great start.</p>
<p>I ordered a caliper<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> and measured the dimensions of the broken part, and took
pictures of it from several angles for later reference. I took some notes:</p>
<p><img src="https://l.sr.ht/20eR.jpg" alt="Picture of my notes on the measurements of the part"></p>
<p>Then, I used solvespace to design the following part:</p>
<p><img src="https://l.sr.ht/rVPV.png" alt="Screenshot of the replacement part in solvespace"></p>
<p>This was the third iteration — I printed one version, brought it out to
the truck to compare with the broken part, made refinements to the design, then
rinse and repeat. Here&rsquo;s an earlier revision being compared with the broken
piece:</p>
<p><img src="https://l.sr.ht/CUPM.jpg" alt="A hand holds up a 3D printed part for comparison with the original"></p>
<p>Finally, I arrived at a design I liked and sent it to the printer.</p>
<p><img src="https://l.sr.ht/xh9h.jpg" alt="Picture of my 3D printer working on the part"></p>
<p>I took some pliers to the remaining plastic bits from the broken part, and sawed
off the rivets. I attached the replacement with superglue and ta-da!</p>
<p><img src="https://l.sr.ht/3AGi.jpg" alt="Picture of the replacement part in place"></p>
<p>If the glue fails, I&rsquo;ll drill out what&rsquo;s left of the rivets and secure it with
screws. This may require another revision of the design, which will also give me
a chance to address some minor shortcomings. I don&rsquo;t expect to need this,
though, because this is not part under especially high stress.</p>
<p>You can get the CAD files and an STL from my <a href="https://git.sr.ht/~sircmpwn/open-dodge-d250">repository
here</a>, which I intend to keep
updating as I learn more about this truck and encounter more fun problems to
solve.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Oh man, I&rsquo;ve always wanted a caliper, and now I have an excuse! <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/04/06/My-weird-branchless-git-workflow.html
title: My unorthodox, branchless git workflow
url: https://drewdevault.com/2020/04/06/My-weird-branchless-git-workflow.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I have been using git for a while, and I took the time to learn about it in
great detail. Equipped with an understanding of its internals and a comfortable
familiarity with tools like <a href="https://git-rebase.io">git rebase</a> — and a
personal, intrinsic desire to strive for minimal and lightweight solutions
— I have organically developed a workflow which is, admittedly, somewhat
unorthodox.</p>
<p>In short, I use git branches very rarely, preferring to work on my local master
branch almost every time. When I want to work on multiple tasks in the same
repository (i.e. often), I just&hellip; work on all of them on master. I waste no
time creating a new branch, or switching to another branch to change contexts; I
just start writing code and committing changes, all directly on master,
intermixing different workstreams freely.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> This reduces my startup time to
zero, both for starting new tasks and revisiting old work.</p>
<p>When I&rsquo;m ready to present some or all of my changes to upstream, I grab git
rebase and reorganize all of these into their respective features, bugfixes, and
so on, forming a series of carefully organized, self-contained patchsets. When I
receive feedback, I just start correcting the code right away, then fixup the
old commits during the rebase. Often, I&rsquo;ll bring the particular patchset I&rsquo;m
ready to present upstream to the front of my master branch at the same time, for
convenient access with <a href="https://git-send-email.io">git send-email</a>.</p>
<p>I generally set my local master branch to track the remote master branch,<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> so
I can update my branch with <code>git pull --rebase</code>.<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> Because all of my
work-in-progress features are on the master branch, this allows me to quickly
address any merge conflicts with upstream for <em>all</em> of my ongoing work at once.
Additionally, by keeping them all on the same branch, I can be assured that my
patches are mutually applicable and that there won&rsquo;t be any surprise conflicts
in feature B after feature A is merged upstream.</p>
<p>If I&rsquo;m working on my own projects (where I can push to upstream master), I&rsquo;ll
still be working on master. If I end up with a few commits queued up and I need
to review some incoming patches, I&rsquo;ll just apply them to master, rebase them
behind my WIP work, and then use <code>git push origin HEAD~5:refs/heads/master</code> to
send them upstream, or something to that effect.<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup> Bonus: this instantly
rebases my WIP work on top of the new master branch.</p>
<p>This workflow saves me time in several ways:</p>
<ul>
<li>No time spent creating new branches for new features.</li>
<li>No time spent switching between branches to address feedback.</li>
<li>All of my features are guaranteed to be mutually applicable to master, saving
me time addressing conflicts.</li>
<li>Any conflicts with upstream are addressed in all of my workstreams at once,
without switching between branches or allowing any branch to get stale.</li>
</ul>
<p>I know that lightweight branches are one of git&rsquo;s flagship features, but I don&rsquo;t
really use them. I know it&rsquo;s weird, sue me.</p>
<p>Sometimes I do use branches, though, when I know that a workstream is going to
be a lot of work — it involves lots of large-scale refactoring, or will
take several weeks to complete. This isolates it from my normal workflow on
small-to-medium patches, acknowledging that the large workstream is going to be
more prone to conflicts. By addressing these separately, I don&rsquo;t waste my time
fixing up the error-prone branch all the time while I&rsquo;m working on my smaller
workstreams.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>I will occasionally use <code>git add -p</code> or even just <code>git commit -p</code> to quickly separate any changes in my working directory into separate commits for their respective workstreams, to make my life easier later on. This is usually the case when, for example, I have to fix problem A before I can address problem B, and additional issues with problem A are revealed by my work on problem B. I just fix them right away, <code>git commit -p</code> the changes separately, then file each commit into their respective patchsets later. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>&ldquo;What?&rdquo; Okay, so in git, you have <em>local</em> branches and <em>remote</em> branches.  The default behavior is reasonably sane, so I would forgive you for not noticing. Your local branches can <em>track</em> remote branches, so that when you <code>git pull</code> it automatically updates any local <em>tracking branches</em>. <code>git pull</code> is actually equivalent to doing <code>git fetch</code> and then <code>git merge origin/master</code> assuming that the current branch (your <em>local</em> master) is <em>tracking</em> <code>origin/master</code>. <code>git pull --rebase</code> is the same thing, except it uses <code>git rebase</code> instead of <code>git merge</code> to update your local branch. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>In fact, I have <code>pull.rebase = true</code> in my git config, which makes <code>--rebase</code> the default behavior. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>&ldquo;What?&rdquo; Okay, so <code>git push</code> is shorthand for <code>git push origin master</code>, if you have a tracking branch set up for your local master branch to <code>origin/master</code>. But this itself is also shorthand, for <code>git push &lt;remote&gt; &lt;local&gt;:&lt;remote&gt;</code>, where <code>&lt;local&gt;</code> is the local branch you want to push, and <code>&lt;remote&gt;</code> is the remote branch you want to update. But, remember that branches are just references to commits. In git, there are other ways to reference commits. <code>HEAD~5</code>, for example, gets the commit which is 5 commits earlier than <code>HEAD</code>, which is the commit you have checked out right now. So <code>git push origin HEAD~5:refs/for/master</code> updates the <code>origin</code>&rsquo;s <code>refs/for/master</code> reference (i.e. the master branch) to the local commit at <code>HEAD~5</code>, pushing any commits that upstream master doesn&rsquo;t also have in the process. <a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/04/15/Status-update-April-2020.html
title: Status update, April 2020
url: https://drewdevault.com/2020/04/15/Status-update-April-2020.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Wow, it&rsquo;s already time for another status update? I&rsquo;m starting to lose track of
the days stuck inside. I have it easier than many - I was already used to
working from home before any of this began. But, weeks and weeks of not spending
IRL time with anyone else is starting to get to me. Remember to call your
friends and family and let them know how you&rsquo;re doing. Meanwhile, I&rsquo;ve had a
productive month - let&rsquo;s get you up to date!</p>
<p>In the Wayland world, I&rsquo;ve made some more progress on the book. The input
chapter is now finished, including the example code. The main things which
remain to be written are the XDG positioner section (which I am dreading), drag
and drop, and protocol extensions. On the code side of things, wlroots continues
to see gradual improvements — the DRM (not the bad kind) implementation
continues to see improvements, expanding to more and more use-cases with even
better performance. Sway has also seen little bug fixes here and there, and
updates to keep up with wlroots.</p>
<p>For my part, I&rsquo;ve mostly been focused on SourceHut and Secret Project this
month. On the SourceHut side of things, I&rsquo;ve been working on hub.sr.ht, and on
an experimental GraphQL-based API for git.sr.ht. The former is progressing quite
well, and I hope to ship an early version before the next status update. As for
the latter, it&rsquo;s still very experimental, but I am optimistic about it. I have
felt that the current REST API design was less than ideal, and the best time to
change it would be during the alpha. The GraphQL design, while it has its
limitations, is a lot better than the REST design and should make it a lot
easier for services to interop with each other - which is a core design need for
sr.ht.</p>
<p>Here&rsquo;s a little demo of hub.sr.ht as of a few weeks ago to whet your appetite:</p>
<video src="https://yukari.sr.ht/hub.sr.ht.webm" muted autoplay loop controls>
  Your web browser does not support the webm video codec. Please consider using
  web browsers that support free and open standards.
</video>
<p>As far as the secret project is concerned, here&rsquo;s another teaser:</p>
<pre><code>fn printf (fmt: str, ...) int;

fn f (ptr: &amp;int) int =
{
	let x: int = *ptr;
	free ptr;
	printf(&quot;value: %d\n&quot;, x)
};

export fn main int =
{
	let x = alloc &amp;int 10;
	f(^x);
	0
};
</code></pre><pre><code>$ [redacted] -o example [redacted...]
$ ./example
value: 10
</code></pre><p>That&rsquo;s all for today! I&rsquo;ll see you again next month. Thank you for your support!</p>


guid: https://drewdevault.com/2020/04/20/Configuring-aerc-for-git.html
title: Configuring aerc for git via email
url: https://drewdevault.com/2020/04/20/Configuring-aerc-for-git.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I use <a href="https://aerc-mail.org">aerc</a> as my email client (naturally — I
wrote it, after all), and I use <a href="https://git-send-email.io">git send-email</a> to
receive patches to many of my projects. I designed aerc specifically to be
productive for this workflow, but there are a few extra things that I use in my
personal aerc configuration that I thought were worth sharing briefly. This blog
post will be boring and clerical, feel free to skip it unless it&rsquo;s something
you&rsquo;re interested in.</p>
<p>When I want to review a patch, I first tell aerc to <code>:cd sources/&lt;that project&gt;</code>, then I open up the patch and give it a read. If it needs work, I&rsquo;ll
use &ldquo;rq&rdquo; (&ldquo;reply quoted&rdquo;), a keybinding which is available by default, to open
my editor with the patch pre-quoted to trim down and reply with feedback inline.
If it looks good, I use the first of my custom keybindings: &ldquo;ga&rdquo;, short for git
am. The entry in <code>~/.config/aerc/binds.conf</code> is:</p>
<pre><code>ga = :pipe -mb git am -3&lt;Enter&gt;
</code></pre><p>This pipes the entire message (-m, in case I&rsquo;m viewing a message part) into <code>git am -3</code> (-3 uses a three-way merge, in case of conflicts), in the background
(-b). Then I&rsquo;ll use C-t (ctrl-T), another keybinding which is included by
default, to open a terminal tab in that directory, where I can compile the code,
run the tests, and so on. When I&rsquo;m done, I use the &ldquo;gp&rdquo; keybinding to push the
changes:</p>
<pre><code>gp = :term git push&lt;Enter&gt;
</code></pre><p>This runs the command in a new terminal, so I can monitor the progress. Finally,
I like to reply to the patch, letting the contributor know their work was merged
and thanking them for the contribution. I have a keybinding for this, too:</p>
<pre><code>rt = :reply -Tthanks&lt;Enter&gt;
</code></pre><p>My &ldquo;thanks&rdquo; template is at <code>~/.config/aerc/templates/thanks</code> and looks like
this:</p>
<pre><code>Thanks!

{% raw %}
{{exec &quot;{ git remote get-url --push origin;
    git reflog -2 origin/master --pretty=format:%h; }
  | xargs printf 'To %s\n   %s..%s  master -&gt; master\n'&quot; &quot;&quot;}}
{% endraw %}
</code></pre><p>That git command prints a summary of the most recent push to master. The result
is that my editor is pre-filled with something like this:</p>
<pre><code>Thanks!

To git@git.sr.ht:~sircmpwn/builds.sr.ht
   7aabe74..191f4a0  master -&gt; master
</code></pre><p>I occasionally append a few lines asking questions about follow-up work or
clarifying the deployment schedule for the change.</p>


guid: https://drewdevault.com/2020/04/22/How-to-store-data-forever.html
title: How to store data forever
url: https://drewdevault.com/2020/04/22/How-to-store-data-forever.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>As someone who has been often maligned by the disappearance of my data for
various reasons — companies going under, hard drive failure, etc —
and as someone who is responsible for the safekeeping of other people&rsquo;s data,
I&rsquo;ve put a lot of thought into solutions for long-term data retention.</p>
<p>There are two kinds of long-term storage, with different concerns: cold storage
and hot storage. The former is like a hard drive in your safe — it stores
your data, but you&rsquo;re not actively using it or putting wear on the storage
medium. By contrast, hot storage is storage which is available immediately and
undergoing frequent reads and writes.</p>
<h2 id="what-storage-medium-to-use">What storage medium to use?</h2>
<p>There are some bad ways to do it. The worst way I can think of is to store
it on a microSD card. These fail <em>a lot</em>. I couldn&rsquo;t find any hard data, but
anecdotally, 4 out of 5 microSD cards I&rsquo;ve used have experienced failures
resulting in permanent data loss. Low volume writes, such as from a digital
camera, are unlikely to cause failure. However, microSD cards have a tendency to
get hot with prolonged writes, and they&rsquo;ll quickly leave their safe operating
temperature and start to accumulate damage. Nearly all microSD cards will let
you perform writes fast enough to drive up the temperature beyond the operating
limits — after all, writes per second is a marketable feature — so
if you want to safely move lots of data onto or off of a microSD card, you need
to monitor the temperature and throttle your read/write operations.</p>
<p>A more reliable solution is to store the data on a hard drive<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. However, hard
drives are rated for a limited number of read/write cycles, and can be expected
to fail eventually. Backblaze publishes some great articles on <a href="https://www.backblaze.com/blog/hard-drive-stats-for-2019/">hard drive
failure rates</a> across
their fleet. According to them, the average annual failure rate of hard drives
is almost 2%. Of course, the exact rate will vary with the frequency of use and
storage conditions. Even in cold storage, the shelf life of a magnetic platter
is not indefinite.</p>
<p>There are other solutions, like optical media, tape drives, or more novel
mediums, like the <a href="https://en.wikipedia.org/wiki/Rosetta_Project">Rosetta Disk</a>.
For most readers, a hard drive will be the best balance of practical and
reliable. For serious long-term storage, if expense isn&rsquo;t a concern, I would
also recommend hot storage over cold storage because it introduces the
possibility of active monitoring.</p>
<h2 id="redundancy-with-raid">Redundancy with RAID</h2>
<p>One solution to this is redundancy — storing the same data across multiple
hard drives. For cold storage, this is often as simple as copying the data onto
a second hard drive, like an external backup HDD. Other solutions exist for hot
storage. The most common standard is <a href="https://en.wikipedia.org/wiki/RAID">RAID</a>, which offers different
features with different numbers of hard drives. With two hard drives (RAID1), for
example, it utilizes mirroring, which writes the same data to both disks. RAID
gets more creative with three or more hard drives, utilizing <em>parity</em>, which
allows it to reconstruct the contents of failed hard drives from still-online
drives. The basic idea relies on the XOR operation. Let&rsquo;s say you write the
following byte to drive A: <code>0b11100111</code>, and to drive B: <code>0b10101100</code>. By XORing
these values together:</p>
<pre><code>  11100111 A
^ 10101100 B
= 01001011 C
</code></pre>
<p>We obtain the value to write to drive C. If any of these three drives fail, we
can XOR the remaining two values again to obtain the third.</p>
<pre><code>  11100111 A
^ 01001011 C
= 10101100 B

  10101100 B
^ 01001011 C
= 11100111 A
</code></pre>
<p>This allows any drive to fail while still being able to recover its contents,
and the recovery can be performed online. However, it&rsquo;s often not that simple.
Drive failure can dramatically reduce the performance of the array while it&rsquo;s
being rebuilt — the disks are going to be seeking constantly to find the
parity data to rebuild the failed disk, and any attempts to read from the disk
that&rsquo;s being rebuilt will require computing the recovered value on the fly. This
can be improved upon by using lots of drives and multiple levels of redundancy,
but it is still likely to have an impact on the availability of your data if not
carefully planned for.</p>
<p>You should also be monitoring your drives and preparing for their failure in
advance.  Failing disks can show signs of it in advance — degraded
performance, or via S.M.A.R.T reports. Learn the tools for monitoring your
storage medium, such as smartmontools, and set it up to report failures to you
(and <em>test</em> the mechanisms by which the failures are reported to you).</p>
<h3 id="other-raid-failure-modes">Other RAID failure modes</h3>
<p>There are other common ways a RAID can fail that result in permanent data loss.
One example is using hardware RAID — there was an argument to be made for
them at one point, but these days hardware RAID is <em>almost always</em> a mistake.
Most operating systems have software RAID implementations which can achieve the
same results without a dedicated RAID card. With hardware RAID, if the RAID card
itself fails (and they often do), you might have to find the exact same card to
be able to read from your disks again. You&rsquo;ll be paying for new hardware, which
might be expensive or out of production, and waiting for it to arrive before you
can start recovering data. With software RAID, the hard drives are portable
between machines and you can always interpret the data with general purpose
software.</p>
<p>Another common failure is <em>cascading</em> drive failures. RAID can tolerate partial
drive failure thanks to parity and mirroring, but if the failures start to pile
up, you can suffer permanent data loss. Many a sad administrator has been in
panic mode, recovering a RAID from a disk failure, and at their lowest
moment&hellip; another disk fails. Then another. They&rsquo;ve suddenly lost their data,
and the challenge of recovering what remains has become ten times harder. When
you&rsquo;ve been distributing read and write operations consistently across all of
your drives over the lifetime of the hardware, they&rsquo;ve been receiving a similar
level of wear, and failing together is not uncommon.</p>
<p>Often, failures like this can be attributed to using many hard drives from the
same batch. One strategy I recommend to avoid this scenario is to use drives
from a mix of vendors, model numbers, and so on. Using a RAID improves
performance by distributing reads and writes across drives, using the time one
drive is busy to utilize an alternate. Accordingly, any differences in the
performance characteristics of different kinds of drives will be smoothed out in
the wash.</p>
<h2 id="zfs">ZFS</h2>
<p>RAID is complicated, and getting it right is difficult. You don&rsquo;t want to wait
until your drives are failing to learn about a gap in your understanding of
RAID. For this reason, I recommend ZFS to most. It automatically makes good
decisions for you with respect to mirroring and parity, and gracefully handles
rebuilds, sudden power loss, and other failures. It also has features which are
helpful for other failure modes, like snapshots.</p>
<p>Set up Zed to email you reports from ZFS. Zed has a debug mode, which will send
you emails even for working disks — I recommend leaving this on, so that
their conspicuous absence might alert you to a problem with the monitoring
mechanism. Set up a cronjob to do monthly scrubs and review the Zed reports when
they arrive. ZFS snapshots are cheap - set up a cronjob to take one every 5
minutes, perhaps with <a href="https://github.com/zfsonlinux/zfs-auto-snapshot">zfs-auto-snapshot</a>.</p>
<h2 id="human-failures-and-existential-threats">Human failures and existential threats</h2>
<p>Even if you&rsquo;ve addressed hardware failure, you&rsquo;re not done yet. There are other
ways still in which your storage may fail. Maybe your server fans fail and burn
out all of your hard drives at once. Or, your datacenter could suffer a total
existence failure — what if a fire burns down the building?</p>
<p>There&rsquo;s also the problem of human failure. What if you accidentally <code>rm -rf / *</code>
the server? Your RAID array will faithfully remove the data from all of the hard
drives for you. What if you send the sysop out to the datacenter to decommission
a machine, and no one notices that they decommissioned the wrong one until it&rsquo;s
too late?</p>
<p>This is where off-site backups come into play. For this purpose, I recommend
<a href="https://www.borgbackup.org/">Borg backup</a>. It has sophisticated features for compression and
encryption, and allows you to mount any version of your backups as a filesystem
to recover the data from. Set this up on a cronjob as well for as frequently as
you feel the need to make backups, and send them off-site to another location,
which itself should have storage facilities following the rest of the
recommendations from this article. Set up another cronjob to run <code>borg check</code>
and send you the results on a schedule, so that their conspicuous absence may
indicate that something fishy is going on. I also use <a href="https://prometheus.io/">Prometheus</a> with
<a href="https://github.com/prometheus/pushgateway">Pushgateway</a> to make a note every time that a backup is run, and
set up an alarm which goes off if the backup age exceeds 48 hours. I also have
periodic test alarms, so that the alert manager&rsquo;s own failures are noticed.</p>
<h2 id="are-you-prepared-for-the-failure">Are you prepared for the failure?</h2>
<p>When your disks are failing and everything is on fire and the sky is falling,
this is the worst time to be your first rodeo. You should have <em>practiced</em> these
problems before they became problems. Do training with anyone expected to deal
with failures. Yank out a hard drive and tell them to fix it. Have someone in
sales come yell at them partway through because the website is unbearably slow
while the RAID is rebuilding and the company is losing $100 per minute as a
result of the outage.</p>
<p>Periodically produce a working system from your backups. This proves (1) the
backups are still working, (2) the backups have coverage over everything which
would need to be restored, and (3) you know how to restore them. Bonus: if
you&rsquo;re confident in your backups, you should be able to replace the production
system with the restored one and allow service to continue as normal.</p>
<h2 id="actually-storing-data-forever">Actually storing data <em>forever</em></h2>
<p>Let&rsquo;s say you&rsquo;ve managed to keep your data around. But will you still know how
to interpret that data in the future? Is it in a file format which requires
specialized software to use? Will that software still be relevant in the future?
Is that software open-source, so you can update it yourself? Will it still
compile and run correctly on newer operating systems and hardware? Will the
storage medium still be compatible with new computers?</p>
<p>Who is going to be around to watch the monitoring systems you&rsquo;ve put in place?
Who&rsquo;s going to replace the failing hard drives after you&rsquo;re gone? How will they
be paid? Will the dataset still be comprehensible after 500 years of evolution
of written language? The dataset requires constant maintenance to remain intact,
but also to remain useful.</p>
<p>And ultimately, there is one factor to long-term data retention that you cannot
control: future generations will decide what data is worth keeping — not
us.</p>
<p>In summary: no matter what, definitely don&rsquo;t do this:</p>
<p><img src="https://l.sr.ht/ig3R.jpg" alt="Picture of a SATA card for RAIDing 10 microSD cards together"></p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Or SSDs, which I will refer to interchangeably with HDDs in this article. They have their own considerations, but we&rsquo;ll get to that. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/05/05/We-are-complicit-in-our-employers-deeds.html
title: We are complicit in our employer's deeds
url: https://drewdevault.com/2020/05/05/We-are-complicit-in-our-employers-deeds.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Tim Bray&rsquo;s excellent &ldquo;<a href="https://www.tbray.org/ongoing/When/202x/2020/04/29/Leaving-Amazon">Bye Amazon</a>&rdquo; post inspired me to take this
article off of my backlog, where it has been sitting for a few weeks. I applaud
Tim for stepping down from a company that has demonstrated itself incompatible
with his sense of right and wrong, and I want to take a moment to remind you
that the rest of us in the tech industry have the same opportunity — no,
the same <em>obligation</em> as Tim did.</p>
<p>As software engineers, we enjoy high salaries and extremely good job security. A
good software engineer with only a couple of years of experience under their
belt can expect to have an offer within 1 or 2 months of starting their search.
It can seem a little scary and stressful, but if you&rsquo;re a programmer already
working at $company and you&rsquo;re looking for a change, you&rsquo;re better off than 99%
of your non-technical friends. In tech, hardly anyone is &ldquo;trapped&rdquo; at a bad job;
or at least we don&rsquo;t have a good excuse for not trying for something better.</p>
<p>Tim calls out Amazon&rsquo;s terrible, unhealthy working conditions and retaliation
against staff who speak up or try to organize.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> Google conducts mass
surveillance, kowtows to oppressive regimes, and punishes workers who stand up
to them. Less obvious stuff, too — Apple builds walled gardens and makes
targeted attacks on open standards, Facebook is a giant surveillance tool which
routinely disregards the law, the same behavior which made Uber and Airbnb into
the giants they are today, all while fostering a &ldquo;gig&rdquo; culture in which the poor
have no stability or security. Mass surveillance, contempt of the law, tax
evasion, oppression of the poor, of minorities&hellip; this is what our industry is
known for, and it&rsquo;s <em>our</em> fault.</p>
<p>This is why I hold my peers accountable for working at companies which are
making a negative impact on the world around them. As a general rule, it costs a
business your salary × 1.5 to employ you, given the overhead of benefits,
HR, training, and so on. When you&rsquo;re making a cool half-million annual salary
from $bigcorp, it&rsquo;s because they expect to make at least ¾ of a million that
they wouldn&rsquo;t be making without you. It does not make economic sense for them to
hire you if this weren&rsquo;t the case. Your contribution makes a big difference.</p>
<p>If the best defense we have for working at these companies is the <a href="https://en.wikipedia.org/wiki/Superior_orders">Nuremberg
defense</a>, that doesn&rsquo;t reflect well on us. But, maybe you would
object, maybe you would have the courage to say &ldquo;no&rdquo; when asked to do these
things. Maybe you would, but someday, a cool project will come across your
inbox - machine learning!  Big data! Cloud scale! It&rsquo;s everything you were
promised when you took the job, and have more fun with it for a few months than
you have had in a long time. Your superiors are thrilled - &ldquo;it&rsquo;s perfect!&rdquo;, they
say, and it&rsquo;s not until they take it and start feeding it real-world data that
<a href="https://observer.com/2020/04/amazon-whole-foods-anti-union-technology-heat-map/">you realize exactly what you have built</a>. Doublethink quickly
steps in to protect your ego from the cognitive dissonance, and you take another
little step towards becoming the person you once swore never to be.</p>
<p>The rapid computerization of society has decreased the time necessary to build
novel machines one thousand-fold. This endows us with a great responsibility,
because whatever we build with them, the changes they bring to society will be
upon us much, much faster than any changes to come before. Every software
developer possesses alone the potential of 50 engineers living just 100 years
ago. We can apply this power for good or for ill, but it&rsquo;s up to each of us to
make a deliberate choice on the matter.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p><a href="https://www.amazon.com/mc/pipelines/cancellation">Here&rsquo;s a link</a> to cancel Amazon Prime, by the way. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2020/05/15/Status-update-May-2020.html
title: Status update, May 2020
url: https://drewdevault.com/2020/05/15/Status-update-May-2020.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Hello, future readers! I am writing to you from one day in the past. I finished
my plans for today early and thought I&rsquo;d get a head start on writing the status
updates for tomorrow, or rather, for today. From your reference frame, that is.</p>
<p>Let&rsquo;s start with Wayland. First, as you might have heard, <a href="https://wayland-book.com">The Wayland
Protocol</a> is now free for anyone to read, and has been
relicensed as CC-BY-SA. Enjoy! It&rsquo;s still not quite done, but most of it&rsquo;s
there. In development news, wlroots continues to enjoy incremental improvements,
and is being refined further and further towards a perfect citizen of the
ecosystem in which it resides. Sway as well has seen many small bugfixes and
improvements. Both have been been stable for a while now: the only meaningful
changes will be, for the most part, a steady stream of bug fixes and performance
improvements.</p>
<p>Moving on from Wayland, then, there are some interesting developments in the
world of email as well. aerc has seen some minor changes to how it handles
templates and custom email headers, and a series of other small features and
improvements: drafts, a <code>:choose</code> meta-command, and fixes for OpenBSD and Go
1.15. Additionally, I&rsquo;ve joined <a href="https://emersion.fr/">Simon Ser</a> to work on
<a href="https://sr.ht/~emersion/alps/">Alps</a> together, to put the finishing touches on
our lightweight &amp; customizable webmail client before
<a href="https://www.migadu.com/en/index.html">Migadu</a> puts it into production.</p>
<p>On the SourceHut front, lots of cool stuff came out this month. You might have
seen the <a href="https://sourcehut.org/blog/2020-05-11-sourcehut-plus-plan-9/">announcement this week</a> that we&rsquo;ve added Plan 9 support to the
CI — a world first :D I also just published the first bits of the new,
experimental GraphQL API for git.sr.ht, which you can <a href="https://git.sr.ht/graphql">play with here</a>.
And, of course, the long-awaited project hub was released this month! <a href="https://sr.ht">Check it
out here</a> to get your projects listed. I&rsquo;ll post about all of
this in more detail on the sr.ht-announce mailing list later today.</p>
<p>That&rsquo;s all for today! I&rsquo;ll see you next month. Thank you once more for your
wonderful support.</p>
<details>
  <summary>...</summary>
<pre>/* sys::write */
fn write(fd: int, buf: *void, count: size) size;
<p>fn puts(s: str) size =
{
let n = write(1, s: *char, len(s));
n += write(1, &ldquo;\n&rdquo;: *char, 1);
n;
};</p>
<p>export fn main int =
{
puts(&ldquo;Hello world!&quot;);
0;
};
</pre></p>
<pre>
$ ./[redacted] < example.[redacted] | qbe > example.S
$ as -o example.o example.S
$ ld -o example lib/sys/[redacted]s.o example.o lib/sys/lib[redacted]rt.a
$ wc -c example
9640
$ ./example
Hello world!
</pre>
</details>


guid: https://drewdevault.com/2020/06/06/Add-a-contrib-directory.html
title: Add a "contrib" directory to your projects
url: https://drewdevault.com/2020/06/06/Add-a-contrib-directory.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>There&rsquo;s a common pattern among free- and open-source software projects to
include a &ldquo;contrib&rdquo; directory at the top of their source code tree. I&rsquo;ve seen
this in many projects for many years, but I&rsquo;ve seen it discussed only rarely
— so here we are!</p>
<p>The contrib directory is used as an unorganized (or, at best, lightly organized)
bin of various useful things <strong>contrib</strong>uted by the community around the
software, but which is not necessarily a good candidate for being a proper part
of the software. Things in contrib should not be wired into your build system,
shouldn&rsquo;t be part of your automated testing, shouldn&rsquo;t be included in your
documentation, and should not be installed with your packages. contrib entries
are not supported by the maintainers, and are given only a light code review at
the most. There is no guarantee whatsoever of workitude or maintenance for
anything found in contrib.</p>
<p>Nevertheless, it is often useful to have such a place to put various little
scripts, config files, and so on, which provide a helpful leg-up for users
hoping to integrate the software with some third-party product, configure it to
fit nicely into an unusual environment, coax it into some unusual behavior, or
whatever else the case may be. The idea is to provide a place to drop a couple
of files which might save a future someone facing similar problems from doing
all of the work themselves. Such people can contribute back small fixes or
improvements, and the maintenance burden of such contributions lies entirely
with the users.</p>
<p>If the contributor wants to take on a greater maintenance burden, this kind of
stuff is better suited to a standalone project, with its own issue tracking,
releases, and so on. If you just wrote a little script and want somewhere to
drop it so that others may find it useful, then contrib is the place for you.</p>
<p>For a quick example, let&rsquo;s consult Sway&rsquo;s <a href="https://github.com/swaywm/sway/tree/master/contrib">contrib
folder</a>:</p>
<pre><code>_incr_version
autoname-workspaces.py
grimshot
grimshot.1
grimshot.1.scd
inactive-windows-transparency.py
</code></pre><p>The <code>_incr_version</code> script is something that I use myself to help with preparing
new releases. It is a tool useful only to maintainers, and therefore is not
distributed with the project.</p>
<p>Looking at <code>autoname-workspaces.py</code> next, from which we can see that the quality
criteria is reduced for members of contrib — none of Sway&rsquo;s upstream code
is written in Python, and the introduction of such a dependency would be
controversial. This script automatically changes your workspace name based on
what applications you&rsquo;re running in it — an interesting workflow, but
quite different from the <abbr title="out-of-the-box">OOTB</abbr> experience.</p>
<p><code>grimshot</code> is a shell script which ties together many third-party programs
(grim, slurp, wl-copy, jq, and notify-send) to make a convenient way of taking
screenshots. Adding this upstream would introduce <em>a lot</em> of third-party
dependencies for a minor convenience. This tool has had a bit more effort put
into it: notice that a man page is provided as well. Because the contrib
directory does not participate in the upstream build system, the contributor has
also added a pre-compiled man page so that you can skip this step when
installing it on your system.</p>
<p>Last, we have <code>inactive-windows-transparency.py</code>, which is a script for making
all windows other than your focused one semi-transparent. Some people may want
this, but again, it&rsquo;s not really something we&rsquo;d consider appropriate for the
OOTB experience. Perfect for contrib!</p>


guid: https://drewdevault.com/2020/06/12/Can-we-talk-about-client-side-certs.html
title: Can we talk about client-side certificates?
url: https://drewdevault.com/2020/06/12/Can-we-talk-about-client-side-certs.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I&rsquo;m working on improving the means by which API users authenticate with the
SourceHut API. Today, I was reading <a href="https://tools.ietf.org/html/rfc6749">RFC 6749</a> (OAuth2) for this purpose,
and it got me thinking about the original OAuth spec. I recalled vaguely that it
had the API clients actually <em>sign</em> every request, and&hellip; <a href="https://tools.ietf.org/html/rfc5849">yep, indeed it
does</a>. This also got me thinking: what else signs requests? TLS!</p>
<p>OAuth is very complicated. The RFC is 76 pages long, the separate bearer token
RFC (6750) is another 18, and no one has ever read either of them. Add JSON Web
Tokens (RFC 7519, 30 pages), too. The process is complicated and everyone
implements it themselves — a sure way to make mistakes in a
security-critical component. Not all of the data is authenticated, no
cryptography is involved at any step, and it&rsquo;s easy for either party to end up
in an unexpected state. The server has to deal with problems of revocation and
generating a secure token itself. Have you ever met anyone who feels positively
about OAuth?</p>
<p>Now, take a seat. Have a cup of coffee. I want to talk about client-side
certificates. Why didn&rsquo;t they take off? Let&rsquo;s sketch up a hypothetical TLS-based
protocol as an alternative to OAuth.  Picture the following&hellip;</p>
<ol>
<li>You, an API client developer, generate a certificate authority and
intermediate, and you upload your CA certificate to the Service Provider as
part of your registration as a user agent.</li>
<li>When you want a user to authorize you to access their account, you generate a
certificate for them, and redirect them to the Service Provider&rsquo;s
authorization page with a <abbr title="Certificate Signing Request">CSR</abbr>
in tow. Your certificate includes, among other things, the list of authorized
scopes for which you want to be granted access. It is already signed with
your client CA key, or one of its intermediates.</li>
<li>The client reviews the desired access, and consents. They are redirected back
to your API client application, along with the signed certificate.</li>
<li>Use this client-side certificate to authenticate your API requests. Hooray!</li>
</ol>
<p>Several advantages to this approach occur to me.</p>
<ul>
<li>You get strong encryption and authentication guarantees for free.</li>
<li>TLS is basically the single most ironclad, battle-tested security mechanism on
the internet, and mature implementations are available for every platform.
Everyone implements OAuth themselves, and often poorly.</li>
<li>Client-side certificates are stateless. They contain all of the information
necessary to prove that the client is entitled to access.</li>
<li>If you handle SSL termination with nginx, haproxy, etc, you can reject
unauthorized requests before your application backend ever even sees them.</li>
<li>The service provider can untrust the client&rsquo;s CA in a single revocation, if
they are malicious or lose their private keys.</li>
<li>The API client and service provider are both always certain that the process
was deliberately initiated by the API client. No weird state tokens to carry
through the process like OAuth uses!</li>
<li>Lots of free features: any metadata you like, built-in expirations, API
clients can self-organize into intermediates at their discretion, and so on.</li>
<li>Security-concious end users can toggle a flag in their account which would, as
part of the consent process, ask them to sign the API client&rsquo;s certificate
themselves, before the signed certificate is returned to the API client. Then
any API request authorized for that user&rsquo;s account has to be signed by the API
client, the service provider, <em>and</em> the user to be valid.</li>
</ul>
<p>Here&rsquo;s another example: say your organization has several services, each of
which interacts with a subset of Acme Co&rsquo;s API on behalf of their users. Your
organization generates a single root CA, and signs up for Acme Co&rsquo;s API with it.
Then you issue intermediate CAs to each of your services, which are <em>only</em>
allowed to issue CSRs for the subset of scopes they require. If any service is
compromised, it can&rsquo;t be used to get more access than it already had, and you
can revoke just that one intermediate without affecting the rest.</p>
<p>Even some famous downsides, such as <abbr title="Certificate Revocation
Lists">CRLs</abbr> and <abbr title="Online Certificate Status
Protocol">OCSP</abbr>, are mitigated here, because the system is much more
centralized. You control all of the endpoints which will be validating
certificates, you can just distribute revocations directly to them as soon as
they come in.</p>
<p>The advantages are clearly numerous. Let&rsquo;s wrap it up in a cute, Google-able
name, write some nice tooling and helper libraries for it, and ship it!</p>
<p>Or, maybe not. I have a nagging feeling that I&rsquo;m missing something here. It
doesn&rsquo;t seem right that such an obvious solution would have been left on the
table, by everyone, for decades. Maybe it&rsquo;s just that the whole certificate
signing dance has left a bad taste in everyone&rsquo;s mouth — many of us have
not-so-fond memories of futzing around with the awful OpenSSL CLI to generate a
CSR. But, there&rsquo;s no reason why we couldn&rsquo;t do it better, and more streamlined,
if we had the motivation to.</p>
<p>There are also more use-cases for client-side certificates that seem rather
compelling, such as an alternative to user passwords. Web browser support for
client-side certificates totally sucks, but that is a solvable problem.</p>
<p>For the record, I have no intention of using this approach for the SourceHut
API. This thought simply occurred to me, and I want to hear what you think. Why
aren&rsquo;t we using client-side certificates?</p>


guid: https://drewdevault.com/2020/06/15/Status-update-June-2020.html
title: Status update, June 2020
url: https://drewdevault.com/2020/06/15/Status-update-June-2020.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Like last month, I am writing to you from the past, preparing this status update
a day earlier than usual. This time it&rsquo;s because I expect to be busy with
planned sr.ht maintenance tomorrow, so I&rsquo;m getting the status updates written
ahead of time.</p>
<p>aerc has seen lots of patches merged recently thanks to the hard work of
co-maintainer Reto Brunner and the many contributors who sent patches, ranging
from a scrollable folder list to improvements and bugfixes for PGP support. We
wrapped all of this up in the aerc 0.4.0 release in late May. Thanks to Reto and
all of the other contributors for their hard work on aerc!</p>
<p>Wayland improvements have also continued at a good pace. I&rsquo;ve mentioned before
that wlroots is a crucial core component tying together a lot of different parts
of the ecosystem — DRM/KMS, GBM, OpenGL, libinput, udev, and more —
bringing together integrations for many disparate systems and providing a single
unified multiplexer for them over the Wayland protocol. Taking full advantage of
all of these systems and becoming a more perfect integration of them is a
long-term goal, and we&rsquo;ve been continuing to make headway on these goals over
the past few weeks. We are working hard to squeeze every drop of performance out
of your system.</p>
<p>In the SourceHut world, I&rsquo;ve been working mainly on GraphQL support, as well as
Alpine 3.12 upgrades (the latter being the source of the planned outage). I
wrote in some detail <a href="https://sourcehut.org/blog/2020-06-10-how-graphql-will-shape-the-alpha/">on the sourcehut.org blog</a> about why and how
the GraphQL backends are being implemented, if you&rsquo;re curious. The main
development improvements in this respect which have occured since the last
status updates are the introduction of a JavaScript-free GraphQL playground, and
a GraphQL API for meta.sr.ht. Coming improvements will include an overhaul to
authentication and OAuth2 support, and a dramatically improved approach to
webhooks. Stay tuned!</p>
<p>That&rsquo;s all for the time being. Thank you for your support and attention, and
stay safe out there. I&rsquo;ll see you next month!</p>
<details>
<summary>...</summary>
<pre>
$ cat strconv/itos.$redacted
use bytes;
use types;
<p>/***</p>
<ul>
<li>
<p>Converts an i64 to a string, in base 10. The return value is statically</p>
</li>
<li>
<p>allocated and will be overwritten on subsequent calls; see [strings::dup] to</p>
</li>
<li>
<p>duplicate the result, or [strconv::itosb] to pass your own string buffer.</p>
</li>
<li></li>
<li>
<p>let a = strconv::i64tos(1234);</p>
</li>
<li>
<p>io::printf(&quot;%s&quot;, a); // 1234</p>
</li>
<li></li>
<li>
<p>let a = strconv::i64tos(1234);</p>
</li>
<li>
<p>let b = strconv::i64tos(4321);</p>
</li>
<li>
<p>io::printf(&quot;%s %s&quot;, a, b); // 4321 4321
*/
export fn i64tos(i: i64) const <em>str =
{
static assert(types::I64_MAX == 9223372036854775807,
&ldquo;Maximum integer value exceeds buffer length&rdquo;);
static let s = struct {
l: size = 0,
b: [22]u8 = [0: u8&hellip;], /</em> 20 digits plus NUL and &lsquo;-&rsquo; */
};
s.l = 0;
s.b = [0: u8&hellip;];</p>
<p>const isneg = i &lt; 0;
if (isneg) {
s.b[s.l] = &lsquo;-': u8;
s.l += 1;
i = -i;
} else if (i == 0) {
s.b[s.l] = &lsquo;0&rsquo;: u8;
s.l += 1;
};</p>
<p>while (i &gt; 0) {
s.b[s.l] = &lsquo;0&rsquo;: u8 + (i % 10): u8;
s.l += 1;
i /= 10;
};</p>
<p>const x: size = if (isneg) 1 else 0;
bytes::reverse(s.b[x..s.l]);</p>
<p>s.b[s.l] = 0: u8;
return &amp;s: *str;
};
</pre></p>
</li>
</ul>
</details>


guid: https://drewdevault.com/2020/06/19/Mail-service-provider-recommendations.html
title: Email service provider recommendations
url: https://drewdevault.com/2020/06/19/Mail-service-provider-recommendations.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Email is important to my daily workflow, and I&rsquo;ve built many tools which
encourage productive use of it for software development. As such, I&rsquo;m often
asked for advice on choosing a good email service provider. Personally, I run
my own mail servers, but about a year ago I signed up for and evaluated many
different service providers available today so that I could make informed
recommendations to people. Here are my top picks, as well as the criteria by
which they were evaluated.</p>
<p>Unfortunately, almost all mail providers fail to meet my criteria.  As such, I
can only recommend two: Migadu and mailbox.org.</p>
<h1 id="1-migadu">#1: Migadu</h1>
<p><a href="https://www.migadu.com/">Migadu</a> is my go-to recommendation
for a mail service provider.</p>
<p><strong>Advantages</strong></p>
<ul>
<li>Migadu is a small company with strong values and no outside capital (i.e.
no profit-motivated external influence). Email support and a human being
answers, and their leadership is accessible if you have questions or feedback.</li>
<li>Their pricing is based on bandwidth usage, and does not rely on artificial
scarcity like limited domain names or mailboxes.</li>
<li>Has lots of features for your postmaster - you can treat it as a managed mail
server for your organization.</li>
</ul>
<p><strong>Disadvantages</strong></p>
<ul>
<li>They have suffered from some outages in the past. The global mail system is
tolerant of such outages - you don&rsquo;t have to worry about messages being lost
if they were sent during an outage. Still, being unable to access your mail is
a problem.</li>
<li><del>If you are on a trial account, they will put an advertisement into your email
signature. I don&rsquo;t think that it&rsquo;s ever appropriate for a mail service
provider to edit your outgoing emails for any reason, and certainly not to
advertise.</del> Updated 2021-03-12: this is no longer the case.</li>
</ul>
<p>Full disclosure: SourceHut and Migadu agreed to a consulting arrangement to
build their <a href="https://git.sr.ht/~migadu/alps">new webmail system</a>, which should
be going into production soon. However, I had evaluated and started recommending
Migadu prior to the start of this project, and I believe that Migadu fares well
under the criteria I give at the end of this post.</p>
<h1 id="2-mailboxorg">#2: mailbox.org</h1>
<p><a href="https://mailbox.org/en/">Mailbox.org</a> may be desirable if you wish to have a
more curated experience, and less hands-on access to postmaster-specific
features.</p>
<p><strong>Advantages</strong></p>
<ul>
<li>Excellent first-class support for PGP, and many other strong security and
privacy features are available.</li>
<li>Was able to speak to the CEO directly to discuss my concerns and feedback, and
have my questions answered. Raised some bugs and they were fixed in short
order.</li>
</ul>
<p><strong>Disadvantages</strong></p>
<ul>
<li>The interface is a little bit too JavaScript heavy for my tastes, and suffer
from some bugs and lack of polish.</li>
<li>They are a German company serving mostly German customers - German text leaks
into the UI and documentation in some places.</li>
<li>Completing a Google captcha is required to sign up.</li>
</ul>
<h1 id="others">Others</h1>
<p>Evaluated but not recommended: disroot, fastmail, posteo.de, poste.io,
protonmail, tutanota, riseup, cock.li, teknik, runbox, megacorp mail (gmail,
outlook, etc).</p>
<h1 id="criteria-for-a-good-mail-service-provider">Criteria for a good mail service provider</h1>
<p>The following criteria are objective and non-negotiable:</p>
<ol>
<li>Support for open standards including IMAP and SMTP</li>
<li>Support for users who wish to bring their own domain</li>
</ol>
<p>This is necessary to preserve the user&rsquo;s ownership of their data by making it
accessible over open and standardized protocols, and their right to move to
another service provider by not fixing their identity to a domain name
controlled by the email provider. It is for these reasons that Posteo,
ProtonMail, and Tutanota are not considered suitable.</p>
<p>The remaining criteria are subjective:</p>
<ol>
<li>Is the business conducted ethically? Are their incentives aligned with their
customers, or with their investors?</li>
<li>Is it sustainable? Can I expect them to be around in 10 years? 20? 30?</li>
<li>Do they make unfounded claims about security or privacy, or develop
techniques which ultimately rely on trusting them instead of supporting or
improving standards which rely on encryption?<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></li>
<li>If they make claims about privacy or security, do they explain the
limitations and trade-offs, or do they let you believe it&rsquo;s infallible?</li>
<li>Do you trust them with your personal data? What if they&rsquo;re compelled by law
enforcement? What is their government like?<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup></li>
</ol>
<p>Bonus points:</p>
<ul>
<li>What is their relationship with open source?</li>
<li>Can I sign up without an existing email address? Is there a chicken and egg
problem here?<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup></li>
<li>How well do they handle plaintext email? Do they meet the criteria for
recommended clients at
<a href="https://useplaintext.email/#implementation-recommendations">useplaintext.email</a>?</li>
</ul>
<p>If you represent a mail service provider which you believe meets this criteria,
please <a href="mailto:sir@cmpwn.com">send me an email</a>.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>This also rules out ProtonMail and Tutanota, doubly damning them, especially because it provides an excuse for skipping IMAP and SMTP, which conveniently enables vendor lock-in. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>This rules out Fastmail because of their government (Australia)&rsquo;s hostile and subversive laws regarding encryption. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Alarmingly rare, this one. It seems to be either this, or a captcha like mailbox.org does. I would be interested in seeing the use of client-side proof of work, or requiring someone to enter their payment details and successfully complete a charge instead. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


