# rss_item

guid: https://drewdevault.com/2015/11/01/Please-stop-using-slack.html
title: Please don't use Slack for FOSS projects
url: https://drewdevault.com/2015/11/01/Please-stop-using-slack.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I&rsquo;ve noticed that more and more projects are using things like Slack as the chat
medium for their open source projects. In the past couple of days alone, I&rsquo;ve
been directed to Slack for Babel and Bootstrap. I&rsquo;d like to try and curb this
phenomenon before it takes off any more.</p>
<h2 id="problems-with-slack">Problems with Slack</h2>
<p>Slack&hellip;</p>
<ul>
<li>is closed source</li>
<li>has only one client (<em>update: errata at the bottom of this article</em>)</li>
<li>is a walled garden</li>
<li>requires users to have a different tab open for each project they want to be
involved in</li>
<li>requires that Heroku hack to get open registration</li>
</ul>
<p>The last one is a real stinker. Slack is not a tool built for open source
projects to use for communication with their userbase. It&rsquo;s a tool built for
teams and it is ill-suited to this use-case. In fact, Slack has gone on record
as saying that it <em>cannot</em> support this sort of use-case: &ldquo;it’s great that
people are putting Slack to good use&rdquo; but unfortunately &ldquo;these communities are
not something we have the capacity to support given the growth in our existing
business.&rdquo; <sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p>
<h2 id="what-is-irc">What is IRC?</h2>
<p>IRC, or Internet Relay Chat&hellip;</p>
<ul>
<li>is a standardized and well-supported protocol <sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup></li>
<li>has hundreds of open source clients, servers, and bots <sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup></li>
<li>is a distributed design with several networks</li>
<li>allows several projects to co-exist on the same network</li>
<li>has no hacks for registration and is designed to be open</li>
</ul>
<h3 id="no-irc-is-not-dead">No, IRC is not dead</h3>
<p>I often hear that IRC is dead. Even my dad pokes fun at me for using a 30 year
old protocol, but not after I pointed out that he still uses HTTP. Despite the
usual shtick from the valley, old is not necessarily a synonym for bad.</p>
<p>IRC has been around since forever. You may think that it&rsquo;s not popular anymore,
but there are still tons of people using it. There are 87,762 users <em>currently
online</em> (at time of writing) on Freenode. There are 10,293 people on OFTC.
22,384 people on Rizon. In other words, it&rsquo;s still going strong, and I put a lot
more faith in something that&rsquo;s been going full speed ahead since the 80s than in
a Silicon Valley fad startup.</p>
<h2 id="problems-with-irc-that-slack-solves">Problems with IRC that Slack solves</h2>
<p>There are several things Slack tries to solve about IRC. They are:</p>
<p><strong>Code snippets</strong>: Slack has built-in support for them. On IRC you&rsquo;re just asked
to use a pastebin like Gist.</p>
<p><strong>File transfers</strong>: Slack does them. IRC also does them through XDCC, but this
can be difficult to get working.</p>
<p><strong>Persistent sessions</strong>: Slack makes it so that you can see what you missed when
you return. With IRC, you don&rsquo;t have this. If you want it, you can set up an IRC
bouncer like <a href="http://znc.in/">ZNC</a>.</p>
<p><strong>Integrations</strong>: with things like build bots. This was never actually a problem
with IRC. IRC has always been significantly better at this than Slack. There is
<em>definitely</em> an IRC client library for your favorite programming language, and
you can write your own client from scratch in a matter of minutes anyway.
There&rsquo;s an <a href="https://github.com/nandub/hubot-irc">IRC</a> backend for Hubot, too.
GitHub has a built-in hook for announcing repository activity in an IRC channel.</p>
<h2 id="other-projects-are-using-irc">Other projects are using IRC</h2>
<p>Here&rsquo;s a short, incomplete list of important FOSS projects using IRC:</p>
<ul>
<li>Debian</li>
<li>Docker</li>
<li>Django</li>
<li>jQuery</li>
<li>Angular</li>
<li>ReactJS</li>
<li>NeoVim</li>
<li>Node.js</li>
<li>everyone else</li>
</ul>
<p>The list goes on for a while. Just fill in another few hundred bullet points
with your imagination. Seriously, just join <code>#&lt;project-name&gt;</code> on Freenode. It
probably exists.</p>
<h2 id="irc-is-better-for-your-company-too">IRC is better for your company, too</h2>
<p>We use IRC at <a href="https://www.linode.com/">Linode</a>, even for non-technical people.
It works great. If you want to reduce the barrier to entry for non-technicals,
set up something like <a href="https://github.com/erming/shout">shout</a> instead. You can
also have a pretty no-brainer link to webchat on almost every network, <a href="http://webchat.esper.net/?nick=&amp;channels=truecraft">like
this</a>. If you need file
hosting, you can deploy an instance of
<a href="https://github.com/SirCmpwn/sr.ht/">sr.ht</a> or something like it. You can also
host IRC servers on your own infrastructure, which avoids leaving sensitive
conversations on someone else&rsquo;s servers.</p>
<h2 id="please-use-irc">Please use IRC</h2>
<p>In short, I&rsquo;d really appreciate it if we all quit using Slack like this. It&rsquo;s
not appropriate for FOSS projects. I would much rather join your channel with
the client I already have running. That way, I&rsquo;m more likely to stick around
after I get help with whatever issue I came to you for, and contribute back by
helping others as I idle in your channel until the end of time. On Slack, I
leave as soon as I&rsquo;m done getting help because tabs in my browser are precious
real estate.</p>
<p><a href="https://news.ycombinator.com/item?id=10486541">First discussion on Hacker News</a></p>
<p><a href="https://news.ycombinator.com/item?id=11013136">Second discussion on Hacker News</a></p>
<h2 id="updates">Updates</h2>
<p>Addressing feedback on this article.</p>
<p><strong>Slack IRC bridge</strong>: Slack provides an IRC bridge that lets you connect to
Slack with an IRC client. I&rsquo;ve used it - it&rsquo;s a bit of a pain in the ass to set
up, and once you have it, it&rsquo;s not ideal. They did put some effort into it,
though, and it&rsquo;s usable. I&rsquo;m not suggesting that Slack as a product is worse
than IRC - I&rsquo;m just saying that it&rsquo;s not better than IRC for FOSS projects, and
probably not that much better for companies.</p>
<p><strong>Clients</strong>: Slack has several clients that use the API. That being said, there
are fewer of them and for fewer platforms than IRC clients, and there are more
libraries around IRC than there are for Slack. Also, the bigger issue is that I
already have an IRC client, which I use for the hundreds of FOSS projects that
use IRC, and I don&rsquo;t want to add a Slack client for one or two projects.</p>
<p><strong>Gitter</strong>: Gitter is bad for many of the same reasons Slack is. Please don&rsquo;t
use it over IRC.</p>
<p><strong>ircv3</strong>: Check it out: <a href="http://ircv3.net">ircv3.net</a></p>
<p><strong>irccloud</strong>: Is really cool and solves all of the problems. <a href="https://www.irccloud.com/">irccloud.com</a></p>
<p><strong>2018-03-12</strong>: Slack is shutting down the IRC and XMPP gateways.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p><a href="http://thenextweb.com/insider/2015/03/24/slack-is-quietly-unintentionally-killing-irc/">Slack is quietly, unintentionally killing IRC - The Next Web</a> <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p><a href="https://www.rfc-editor.org/rfc/rfc1459.txt">RFC 1459</a> <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p><a href="https://github.com/search?o=desc&amp;q=irc&amp;s=stars&amp;type=Repositories&amp;utf8=%E2%9C%93">Github search for IRC</a> <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2015/11/11/Bring-more-tor-into-your-life.html
title: Bring more Tor into your life
url: https://drewdevault.com/2015/11/11/Bring-more-tor-into-your-life.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p><a href="https://www.torproject.org/">Tor</a> is a project that improves your privacy
online by encrypting and bouncing your connection through several nodes before
leaving for the outside world. It makes it much more difficult for someone
spying on you to know who you&rsquo;re talking to online and what you&rsquo;re saying to
them. Many people use it with the Tor Browser (a fork of Firefox) and only use
it with HTTP.</p>
<p>What some people do not know is that Tor works at the TCP level, and can be used
for any kind of traffic. There is a glaring issue with using Tor for your daily
browsing - it&rsquo;s significantly slower. That being said, there are several things
you run on your computer where speed is not quite as important. I am personally
using Tor for several things (this list is incomplete):</p>
<ul>
<li>IRC (chat)</li>
<li>Email client</li>
<li>DNS lookups (systemwide)</li>
<li>Downloading system updates</li>
</ul>
<p>Anything that supports downloading through a SOCKS proxy can be used through
Tor. You can also use programs like
<a href="https://trac.torproject.org/projects/tor/wiki/doc/TorifyHOWTO">torify</a> to
transparently wrap syscalls in Tor for any program (this is how I got my email
to use Tor).</p>
<p>Of course, Tor can&rsquo;t help you if you compromise yourself. You should not use
bittorrent over Tor, and you should check your other applications. You should
also be using SSL/TLS/etc on top of Tor, so that exit nodes can&rsquo;t be evil with
your traffic.</p>
<h2 id="orbot">Orbot</h2>
<p>I also use Tor on my phone. I run all of my phone&rsquo;s traffic through Tor, since I
don&rsquo;t use the internet on my phone much. I have whitelisted apps that need to
stream video or audio, though, for the sake of speed. You can do this, too - set
up a black or whitelist of apps on your phone whose networking will be done
through Tor. The app for this is
<a href="https://guardianproject.info/apps/orbot/">here</a>.</p>
<h2 id="why-bother">Why bother?</h2>
<p>The easy answer is &ldquo;secure everything&rdquo;. If you don&rsquo;t have a good reason to
remain insecure, you should default to secure. That argument doesn&rsquo;t work on
everyone, though, so here are some others.</p>
<ul>
<li>Securing trivial traffic makes more noise to hide the things you care about</li>
<li>You can have more peace of mind about using public WiFi networks if you&rsquo;re
using Tor.</li>
<li>ISPs can&rsquo;t inject extra ads and tracking into things you&rsquo;re using over Tor.</li>
<li>The NSA targets people who use Tor. If you &ldquo;have nothing to hide&rdquo;, then you
can help defend those who do by adding more noise and giving agencies that
engage in illegal spying a bigger haystack. Bonus: Tor helps make sure that
even though you&rsquo;re being looked at, you&rsquo;re secure.</li>
</ul>


guid: https://drewdevault.com/2015/12/20/State-of-sway-December-2015.html
title: State of Sway - December 2015
url: https://drewdevault.com/2015/12/20/State-of-sway-December-2015.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I wrote sway&rsquo;s <a href="https://github.com/SirCmpwn/sway/commit/6a33e1e3cddac31b762e4376e29c03ccf8f92107">initial commit</a>
4 months ago, on August 4th. At the time of writing, there are now 1,070 commits
from 29 different authors, totalling 10,682 lines of C (and 1,176 lines of
header files). This has been done over the course of 256 pull requests and 118
issues. Of the 73 <a href="https://github.com/SirCmpwn/sway/issues/2">i3 features we&rsquo;re
tracking</a>, 51 are now supported, and
I&rsquo;ve been using sway as my daily driver for a while now. Today, sway looks like
this:</p>
<p><a href="https://sr.ht/NCx_.png"><img src="https://sr.ht/NCx_.png" alt=""></a></p>
<p>For those who are new to the project, <a href="https://github.com/SirCmpwn/sway">sway</a>
is an i3-compatible Wayland compositor. That is, your existing
<a href="http://i3wm.org/">i3</a> configuration file will work as-is on sway, and your
keybindings will be the same and the colors and font configuration will be the
same, and so on. It&rsquo;s i3, but on Wayland.</p>
<p>Sway initially made the rounds on <a href="https://redd.it/3he5hn">/r/linux</a> and
<a href="https://redd.it/3he48j">/r/i3wm</a> and
<a href="https://www.phoronix.com/scan.php?page=news_item&amp;px=Wayland-i3-Sway-Tiling">Phoronix</a>
on August 17th, 13 days after the initial commit. I was already dogfooding it by
then, but now I&rsquo;m actually using it 100% of the time, and I hear others have
started to as well. What&rsquo;s happened since then? Well:</p>
<ul>
<li>Floating windows</li>
<li>Multihead support</li>
<li>XDG compliant config</li>
<li>Fullscreen windows</li>
<li>gaps</li>
<li>IPC</li>
<li>Window criteria</li>
<li>58 i3 commands and 1 command unique to sway</li>
<li>Wallpaper support</li>
<li>Resizing/moving tiled windows with the mouse</li>
<li>swaymsg, swaylock, <strong>swaybar</strong> as in i3-msg, i3lock, i3bar</li>
<li>Hundreds of bug fixes and small improvements</li>
</ul>
<p>Work on sway has also driven improvements in our dependencies, such as
<a href="https://github.com/Cloudef/wlc">wlc</a>, which now has improved xwayland support,
support for Wayland protocol extensions (which makes swaybg and swaylock and
swaybar possible), and various bugfixes and small features added at the bequest
of sway. Special thanks to Cloudef for helping us out with so many things!</p>
<p>All of this is only possible thanks to the hard work of dozens of contributors.
Here&rsquo;s the breakdown of <strong>lines of code per author</strong> for the top ten authors:</p>
<table class="table">
    <tbody>
        <tr><td>3516</td><td>Drew DeVault</td></tr>
        <tr><td>2400</td><td>taiyu</td></tr>
        <tr><td>1786</td><td>S. Christoffer Eliesen</td></tr>
        <tr><td>1127</td><td>Mikkel Oscar Lyderik</td></tr>
        <tr><td>720</td><td>Luminarys</td></tr>
        <tr><td>534</td><td>minus</td></tr>
        <tr><td>200</td><td>Christoph Gysin</td></tr>
        <tr><td>121</td><td>Yacine Hmito</td></tr>
        <tr><td>79</td><td>Kevin Hamacher</td></tr>
    </tbody>
</table>
<p>And here&rsquo;s the total <strong>number of commits per author</strong> for each of the top 10
committers:</p>
<table class="table">
    <tbody>
        <tr><td>514</td><td> Drew DeVault</td></tr>
        <tr><td>191</td><td> taiyu</td></tr>
        <tr><td>102</td><td> S. Christoffer Eliesen</td></tr>
        <tr><td>97</td><td> Luminarys</td></tr>
        <tr><td>56</td><td> Mikkel Oscar Lyderik</td></tr>
        <tr><td>46</td><td> Christoph Gysin</td></tr>
        <tr><td>34</td><td> minus</td></tr>
        <tr><td>9</td><td> Ben Boeckel</td></tr>
        <tr><td>6</td><td> Half-Shot</td></tr>
        <tr><td>6</td><td> jdiez17</td></tr>
    </tbody>
</table>
<p>As the maintainer of sway, <em>a lot</em> of what I do is reviewing and merging
contributions from others. So these statistics change a bit if we use <strong>number
of commits per author, excluding merge commits</strong>:</p>
<table class="table">
    <tbody>
        <tr><td>279</td><td> Drew DeVault</td></tr>
        <tr><td>175</td><td> taiyu</td></tr>
        <tr><td>102</td><td> S. Christoffer Eliesen</td></tr>
        <tr><td>96</td><td> Luminarys</td></tr>
        <tr><td>56</td><td> Mikkel Oscar Lyderik</td></tr>
        <tr><td>46</td><td> Christoph Gysin</td></tr>
        <tr><td>34</td><td> minus</td></tr>
        <tr><td>9</td><td> Ben Boeckel</td></tr>
        <tr><td>6</td><td> jdiez17</td></tr>
        <tr><td>5</td><td> Yacine Hmito</td></tr>
    </tbody>
</table>
<p>These stats only cover the top ten in each, but there are more - check out the
<a href="https://github.com/SirCmpwn/sway/graphs/contributors">full list</a>.</p>
<p>So, what does this all mean for sway? Well, it&rsquo;s going very well. If you&rsquo;d like
to live on the edge, you can use sway right now and have a productive workflow.
The important features that are missing include stacking and tabbed layouts,
window borders, and some features on the bar. I&rsquo;m looking at starting up a beta
when these features are finished. Come try out sway! Test it with us, open
GitHub issues with your gripes and desires, and <a href="http://webchat.freenode.net/?channels=sway&amp;uio=d4">chat
with us on IRC</a>.</p>
<p><em>This blog post was composed from sway.</em></p>


guid: https://drewdevault.com/2016/03/22/Integrating-a-VT220-into-my-life.html
title: Integrating a VT220 into my life
url: https://drewdevault.com/2016/03/22/Integrating-a-VT220-into-my-life.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I bought a DEC VT220 terminal a while ago, and put it next to my desk at work. I
use it to read emails on mutt now, and it&rsquo;s actually quite pleasant. There was
some setup involved in making it as comfortable as possible, though.</p>
<p><a href="https://sr.ht/BnAH.jpg"><img src="https://sr.ht/BnAH.jpg" alt="My desk at work"></a></p>
<p>Here&rsquo;s the terminal up close:</p>
<p><a href="https://sr.ht/TnC6.jpg"><img src="https://sr.ht/TnC6.jpg" alt="The terminal itself"></a></p>
<h2 id="hardware">Hardware</h2>
<p>First, I have several pieces of hardware involved in this:</p>
<ul>
<li>VT220 terminal</li>
<li>LK201 keyboard (later made obsolete)</li>
<li><a href="http://amzn.com/B00IDSM6BW">USB to serial adapter</a></li>
<li><a href="http://amzn.com/B00066HL50">DB9-&gt;DB29 null modem cable</a></li>
</ul>
<p>It took a while to get all of these things, but I was able to get a nice
refurbished terminal and a couple of crappy LK201 keyboards. Luckily I was able
to eventually remove the need for the keyboard.</p>
<h2 id="basic-setup">Basic Setup</h2>
<p>Getting this working on Linux is actually pretty simple thanks to decades of
backwards compatability. Plug all of the cords together, turn on the machine,
and (on Arch, at least) run:</p>
<pre><code>systemctl start serial-agetty@ttyUSB0.service
</code></pre>
<p>This will start up a getty for you to log into on your terminal. For a while I
would use the LK201 to log in to this getty and spin up a mail cilent.</p>
<p>I did have to make a couple of changes to serial-agetty@.service, though:</p>
<pre><code>ExecStart=-/sbin/agetty -h -L 19200 %I vt220
</code></pre>
<p>This specifies the TERM variable as &ldquo;vt220&rdquo; and sets the baud rate to 19200. I
had to also set the baud rate in the terminal&rsquo;s settings to 19200 baud as well,
to get the fastest possible terminal.</p>
<p>I eventually got into the habit of logging into the terminal with the LK201,
then running tmux and attaching to tmux from my desktop session. I would then
hide this tmux terminal in the upper left corner of my display, and move my
mouse over to it when I wanted to interact with the terminal. This let me use
the same keyboard I used for the rest of my computer experience to interact with
the VT220, instead of trying to use the LK201 as well. This was a bit annoying,
so eventually I did some more customization.</p>
<h2 id="removing-the-keyboard">Removing the keyboard</h2>
<p>I wanted to be able to make everything automatic, so I could just boot my
computer and log in normally and treat the VT220 almost like a fourth monitor. I
started by automating the process of logging in and running tmux.</p>
<p>First, I created a user for the terminal:</p>
<pre><code>useradd vt220
</code></pre>
<p>Then, I wrote a shell script that would serve as the user&rsquo;s login shell and
would start tmux:</p>
<div class="highlight"><pre class="chroma"><code class="language-bash" data-lang="bash"><span class="cp">#!/usr/bin/env bash
</span><span class="cp"></span><span class="k">if</span> <span class="o">[[</span> <span class="nv">$TERM</span> <span class="o">==</span> <span class="s2">&#34;screen&#34;</span> <span class="o">]]</span>
<span class="k">then</span>
	sudo /usr/local/bin/login-sircmpwn
<span class="k">else</span>
	tmux -S /var/tmux/vt220.sock
<span class="k">fi</span>
</code></pre></div><p>I made that directory, <code>/var/tmux/</code>, and made sure both the vt220 user and my
normal user had access to it. I also edited my sudoers file so that vt220 could
run that command as root:</p>
<pre><code>vt220 ALL=(ALL) NOASSWD: /usr/local/bin/login-sircmpwn
</code></pre>
<p>I put the script into <code>/usr/local/bin</code> and added it to <code>/etc/shells</code>, then made
it the login shell for the vt220 user with <code>chsh</code>. I then moved to my own
systemd unit for starting the getty on ttyUSB0, this time with autologin:</p>
<pre><code>#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Serial Getty on %I
Documentation=man:agetty(8) man:systemd-getty-generator(8)
Documentation=http://0pointer.de/blog/projects/serial-console.html
BindsTo=dev-%i.device
After=dev-%i.device systemd-user-sessions.service plymouth-quit-wait.service

# If additional gettys are spawned during boot then we should make
# sure that this is synchronized before getty.target, even though
# getty.target didn't actually pull it in.
Before=getty.target
IgnoreOnIsolate=yes

[Service]
ExecStart=-/sbin/agetty -a vt220 -h -L 19200 %I vt220
Type=idle
Restart=always
UtmpIdentifier=%I
TTYPath=/dev/%I
TTYReset=yes
TTYVHangup=yes
KillMode=process
IgnoreSIGPIPE=no
SendSIGHUP=yes

[Install]
WantedBy=getty.target
</code></pre>
<p>The only difference here is that it invokes agetty with <code>-a vt220</code> to autologin
as that user. <code>systemctl enable vtgetty@ttyUSB0.service</code> makes it so that on
boot, the getty would run on ttyUSB0 and autologin as vt220. Then the script
from earlier will run tmux, and within tmux will run <code>sudo /usr/local/bin/login-sircmpwn</code>, which is this shell script:</p>
<div class="highlight"><pre class="chroma"><code class="language-bash" data-lang="bash"><span class="cp">#!/usr/bin/env bash
</span><span class="cp"></span><span class="k">until</span> who <span class="p">|</span> grep sircmpwn 2&gt;<span class="p">&amp;</span><span class="m">1</span> &gt;/dev/null
<span class="k">do</span>
	sleep <span class="m">1</span>
<span class="k">done</span>
sudo -iu sircmpwn
</code></pre></div><p>What this does is pretty straightforward - it loops until I log in as sircmpwn,
then enters an interactive session with sudo as sircmpwn.</p>
<p>The net of all of this is that now, I can boot up my machine, and when I log in,
the VT220 starts up with tmux running and logged in as me. Then I went back to
the old way of attaching to this tmux session with a terminal on my desktop
session hidden in a corner of the screen. And now I could ditch the clunky old
LK201 keyboard!</p>
<h2 id="treating-the-terminal-as-another-output">Treating the terminal as another output</h2>
<p>I said earlier that my goal was to treat the terminal as a fake &ldquo;output&rdquo; that I
could switch to from my desktop session just like I switch between my three
graphical outputs. I run <a href="https://github.com/SirCmpwn/sway">sway</a>, of course, so
I decided to add a fake output in sway and see where that went. I made a
somewhat complicated <a href="https://github.com/SirCmpwn/sway/compare/vt220">branch</a>
for this purpose, but the important change is here:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"><span class="gh">diff --git a/sway/handlers.c b/sway/handlers.c
</span><span class="gh">index cec6319..60f8406 100644
</span><span class="gh"></span><span class="gd">--- a/sway/handlers.c
</span><span class="gd"></span><span class="gi">+++ b/sway/handlers.c
</span><span class="gi"></span><span class="gu">@@ -704,6 +704,21 @@ static void handle_wlc_ready(void) {
</span><span class="gu"></span> 		free(line);
 		list_del(config-&gt;cmd_queue, 0);
 	}
<span class="gi">+	// VT220 stuff
</span><span class="gi">+	// Adds a made up output that we can use for a tmux window
</span><span class="gi">+	// connected to my vt220
</span><span class="gi">+	swayc_t *output = new_swayc(C_OUTPUT);
</span><span class="gi">+	output-&gt;name = &#34;VT220&#34;;
</span><span class="gi">+	output-&gt;handle = UINTPTR_MAX;
</span><span class="gi">+	output-&gt;width = 1000;
</span><span class="gi">+	output-&gt;height = 1000;
</span><span class="gi">+	output-&gt;unmanaged = create_list();
</span><span class="gi">+	output-&gt;bg_pid = -1;
</span><span class="gi">+	add_child(&amp;root_container, output);
</span><span class="gi">+	output-&gt;x = -1000;
</span><span class="gi">+	output-&gt;y = 0;
</span><span class="gi">+	new_workspace(output, &#34;__VT220&#34;);
</span><span class="gi">+	// End VT220 stuff
</span><span class="gi"></span> }
</code></pre></div><p>This creates a fake output and puts it to the far left, then adds a workspace to
it called __VT220. I assigned it the output handle of UINTPTR_MAX and everywhere
in sway that it would try to use the output handle to manipulate a real output,
I changed to to avoid doing so if the handle is UINTPTR_MAX. Then I added this
to my sway config:</p>
<pre><code>for_window [title=&quot;__VT220&quot;] move window to workspace __VT220 
</code></pre>
<p>And run this command when sway starts:</p>
<pre><code>urxvt -T &quot;__VT220&quot; -e tmux -S /var/tmux/vt220.sock a
</code></pre>
<p>Which spawns a terminal whose window title is __VT220 running tmux attached to
the session running on the terminal. The for_window rule I added to my sway
config automatically moves this to the VT220 fake output and tada! It works. Now
I have a nice and comfortable way to use my terminal to read emails at work. Now
if only I could convince people to stop sending me HTML emails! I just bought a
second VT220 for use at home, too. Life&rsquo;s good~</p>
<p><a href="https://news.ycombinator.com/item?id=11339909">Discussion on Hacker News</a></p>


guid: https://drewdevault.com/2016/04/11/Please-use-text-plain-for-emails.html
title: Please use text/plain for email
url: https://drewdevault.com/2016/04/11/Please-use-text-plain-for-emails.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>A lot of people have come to hate email, and not without good reason. I don&rsquo;t
hate using email, and I attribute this to better email habits. Unfortunately,
most email clients these days lead users into bad habits that probably
contribute to the sad state of email in 2016. The biggest problem with email is
the widespread use of HTML email.</p>
<p>Compare email to snail mail. You probably throw out most of the mail you get -
it&rsquo;s all junk, ads. Think about the difference between snail mail you read and
snail mail you throw out. Chances are, the mail you throw out is flashy flyers
and spam that&rsquo;s carefully laid out by a designer and full of eye candy (kind of
like many HTML emails). However, if you receive a letter from a friend it&rsquo;s
probably going to be a lot less flashy - just text on a page. Reading letters
like this is pleasant and welcome. Emails should be more like this.</p>
<p>I consider this the basic argument for plaintext emails - they make email
better. There are more specific problems with HTML emails that I can give,
though (not to mention the fact that I read emails <a href="https://drewdevault.com/2016/03/22/Integrating-a-VT220-into-my-life.html">on
this</a>
now).</p>
<h2 id="whats-wrong-with-html-email">What&rsquo;s wrong with HTML email</h2>
<p><strong>Tracking images</strong> are images that are included in HTML emails with &lt;img
/&gt; tags. These images have URLs with unique IDs in them that hit remote
servers and let them know that you opened the email, along with various details
about your mail client and such. This is a form of tracking, which many people
go to great lengths to prevent with tools like
<a href="https://github.com/gorhill/uBlock">uBlock</a>. Most email clients recognize this,
and actually <em>block</em> images from being shown without explicit user consent. If
your images aren&rsquo;t even being shown, then why include them? Tracking users is
evil.</p>
<p>Many <strong>vulnerabilities</strong> in mail clients also stem from rendering HTML email.
Luckily, no mail clients have JavaScript enabled on their HTML email renderers.
However, security issues related to HTML emails are still found quite often in
mail clients. I don&rsquo;t want to view this crap (and I don&rsquo;t).</p>
<p>HTML email also makes <strong>phishing</strong> much easier. I&rsquo;ve often received HTML emails
with links that hide their true intent by using a different href than their text
would suggest (and almost always with a tracking code added, ugh). They are also
<strong>incompatible</strong> with many email-based workflows, such as inline quoting,
mailing list participation, and sending &amp; working with source code patches.</p>
<h2 id="good-habits-for-plaintext-emails">Good habits for plaintext emails</h2>
<p>Some nice things are possible when you choose to use plaintext emails. Remember
before when I was comparing emails to snail mail letters? Well, let&rsquo;s continue
those comparisons. Popular email clients of 2016 have thoroughly bastardized
email, but here&rsquo;s what it once was and perhaps what it could be today.</p>
<p>The common mail client today uses the abhorrent &ldquo;<a href="https://en.wikipedia.org/wiki/Posting_style#Top-posting">top
posting</a>&rdquo; format, where
the entire previous message is dumped underneath your reply. As the usual quote
goes:</p>
<blockquote>
<p>A: Because it messes up the order in which people normally read text.</p>
<p>Q: Why is top-posting such a bad thing?</p>
<p>A: Top-posting.</p>
<p>Q: What is the most annoying thing in e-mail?</p>
</blockquote>
<p>A better way to write emails is the same way you write a letter to send via
snail mail. Would you photocopy the entire history of your correspondence and
staple it to your response? After a while you would start paying more for the
weight! Though bandwidth seems cheap now, the habit is still silly. Instead of
copying the entire conversation into your email, quote only the relevant parts
and respond to them inline. For example, let&rsquo;s say I receive this email:</p>
<pre><code>Hi Drew!

Could you take a look at the server this afternoon? I think it's having some
issues with nginx.

I also took care of the upgrades you asked for last night. Sorry it took so
long!

--
John Doe
</code></pre>
<p>The best way to respond to this would be:</p>
<pre><code>Hi John!

&gt;Could you take a look at the server this afternoon? I think it's having some
&gt;issues with nginx.

No problem. I just had a quick look now and nginx was busted. Should be
working now.

&gt;I also took care of the upgrades you asked for last night. Sorry it took so
&gt;long!

Thanks!

--
Drew DeVault
</code></pre>
<p>John might follow up with this:</p>
<pre><code>&gt;Should be working now.

Yep, seems to be up. Thanks!

--
John Doe
</code></pre>
<p>Much better if you ask me. This works particularly well on mailing lists for
open source projects, where you send a patch and reviewers will respond by
quoting specific parts of your patch and leaving feedback. Just treat emails
like letters!</p>
<h2 id="multipart-emails">Multipart emails</h2>
<p>I think there are nothing but negatives to HTML email. I use
<a href="http://www.mutt.org/">mutt</a> to read email, which doesn&rsquo;t even render HTML
emails and allows me to compose emails with Vim. But if you absolutely insist on
using HTML emails, please use <a href="https://en.wikipedia.org/wiki/MIME#Multipart_messages">multipart
emails</a>. If you&rsquo;re
sending automated emails, your programming language likely contains a mechanism
to facilitate this. The idea is that you send an alternative text/plain body for
your email. Be sure that this body contains all of the information of the HTML
version. If you do this, I will at least be willing to read your emails.</p>
<h2 id="how-do-i-use-plaintext-emails">How do I use plaintext emails?</h2>
<p>Your mail client should have an option for composing emails with plaintext. Look
through your settings for it and it&rsquo;ll change the default. Then you&rsquo;re free!
Tell your friends to do the same, and your email life will be happier.</p>


guid: https://drewdevault.com/2016/04/12/How-to-write-a-better-bloom-filter-in-C.html
title: How to write a better bloom filter in C
url: https://drewdevault.com/2016/04/12/How-to-write-a-better-bloom-filter-in-C.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>This is in response to
<a href="http://blog.michaelschmatz.com/2016/04/11/how-to-write-a-bloom-filter-cpp/">How to write a bloom filter in C++</a>,
which has good intentions, but is ultimately a less than ideal bloom filter
implementation. I put together a better one in C in a few minutes, and I&rsquo;ll
explain the advantages of it.</p>
<p>The important differences are:</p>
<ul>
<li>You bring your own hashing functions</li>
<li>You can add arbitrary data types, not just bytes</li>
<li>It uses bits directly instead of relying on the <code>std::vector&lt;bool&gt;</code>
being space effecient</li>
</ul>
<p>I chose C because (1) I prefer it over C++ and (2) I just think it&rsquo;s a better
choice for implementing low level data types, and C++ is better used in high
level code.</p>
<p>I&rsquo;m not going to explain the mechanics of a bloom filter or most of the details
of why the code looks this way, since I think the original post did a fine job
of that. I&rsquo;ll just present my alternate implementation:</p>
<h2 id="header">Header</h2>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="cp">#ifndef _BLOOM_H
</span><span class="cp">#define _BLOOM_H
</span><span class="cp">#include</span> <span class="cpf">&lt;stddef.h&gt;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;stdbool.h&gt;</span><span class="cp">
</span><span class="cp"></span>
<span class="k">typedef</span> <span class="kt">unsigned</span> <span class="nf">int</span> <span class="p">(</span><span class="o">*</span><span class="n">hash_function</span><span class="p">)(</span><span class="k">const</span> <span class="kt">void</span> <span class="o">*</span><span class="n">data</span><span class="p">);</span>
<span class="k">typedef</span> <span class="k">struct</span> <span class="n">bloom_filter</span> <span class="o">*</span> <span class="n">bloom_t</span><span class="p">;</span>

<span class="cm">/* Creates a new bloom filter with no hash functions and size * 8 bits. */</span>
<span class="n">bloom_t</span> <span class="nf">bloom_create</span><span class="p">(</span><span class="n">size_t</span> <span class="n">size</span><span class="p">);</span>
<span class="cm">/* Frees a bloom filter. */</span>
<span class="kt">void</span> <span class="nf">bloom_free</span><span class="p">(</span><span class="n">bloom_t</span> <span class="n">filter</span><span class="p">);</span>
<span class="cm">/* Adds a hashing function to the bloom filter. You should add all of the
</span><span class="cm"> * functions you intend to use before you add any items. */</span>
<span class="kt">void</span> <span class="nf">bloom_add_hash</span><span class="p">(</span><span class="n">bloom_t</span> <span class="n">filter</span><span class="p">,</span> <span class="n">hash_function</span> <span class="n">func</span><span class="p">);</span>
<span class="cm">/* Adds an item to the bloom filter. */</span>
<span class="kt">void</span> <span class="nf">bloom_add</span><span class="p">(</span><span class="n">bloom_t</span> <span class="n">filter</span><span class="p">,</span> <span class="k">const</span> <span class="kt">void</span> <span class="o">*</span><span class="n">item</span><span class="p">);</span>
<span class="cm">/* Tests if an item is in the bloom filter.
</span><span class="cm"> *
</span><span class="cm"> * Returns false if the item has definitely not been added before. Returns true
</span><span class="cm"> * if the item was probably added before. */</span>
<span class="kt">bool</span> <span class="nf">bloom_test</span><span class="p">(</span><span class="n">bloom_t</span> <span class="n">filter</span><span class="p">,</span> <span class="k">const</span> <span class="kt">void</span> <span class="o">*</span><span class="n">item</span><span class="p">);</span>

<span class="cp">#endif
</span></code></pre></div><h2 id="implementation">Implementation</h2>
<p>The implementation of this is pretty straightfoward. First, here&rsquo;s the actual
structs behind the opaque bloom_t type:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">struct</span> <span class="n">bloom_hash</span> <span class="p">{</span>
    <span class="n">hash_function</span> <span class="n">func</span><span class="p">;</span>
    <span class="k">struct</span> <span class="n">bloom_hash</span> <span class="o">*</span><span class="n">next</span><span class="p">;</span>
<span class="p">};</span>

<span class="k">struct</span> <span class="n">bloom_filter</span> <span class="p">{</span>
    <span class="k">struct</span> <span class="n">bloom_hash</span> <span class="o">*</span><span class="n">func</span><span class="p">;</span>
    <span class="kt">void</span> <span class="o">*</span><span class="n">bits</span><span class="p">;</span>
    <span class="n">size_t</span> <span class="n">size</span><span class="p">;</span>
<span class="p">};</span>
</code></pre></div><p>The hash functions are a linked list, but this isn&rsquo;t important. You can make
that anything you want. Otherwise we have a bit of memory called &ldquo;bits&rdquo; and the
size of it. Now, for the easy functions:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">bloom_t</span> <span class="nf">bloom_create</span><span class="p">(</span><span class="n">size_t</span> <span class="n">size</span><span class="p">)</span> <span class="p">{</span>
	<span class="n">bloom_t</span> <span class="n">res</span> <span class="o">=</span> <span class="n">calloc</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="k">struct</span> <span class="n">bloom_filter</span><span class="p">));</span>
	<span class="n">res</span><span class="o">-&gt;</span><span class="n">size</span> <span class="o">=</span> <span class="n">size</span><span class="p">;</span>
	<span class="n">res</span><span class="o">-&gt;</span><span class="n">bits</span> <span class="o">=</span> <span class="n">malloc</span><span class="p">(</span><span class="n">size</span><span class="p">);</span>
	<span class="k">return</span> <span class="n">res</span><span class="p">;</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">bloom_free</span><span class="p">(</span><span class="n">bloom_t</span> <span class="n">filter</span><span class="p">)</span> <span class="p">{</span>
	<span class="k">if</span> <span class="p">(</span><span class="n">filter</span><span class="p">)</span> <span class="p">{</span>
		<span class="k">while</span> <span class="p">(</span><span class="n">filter</span><span class="o">-&gt;</span><span class="n">func</span><span class="p">)</span> <span class="p">{</span>
			<span class="k">struct</span> <span class="n">bloom_hash</span> <span class="o">*</span><span class="n">h</span><span class="p">;</span>
			<span class="n">filter</span><span class="o">-&gt;</span><span class="n">func</span> <span class="o">=</span> <span class="n">h</span><span class="o">-&gt;</span><span class="n">next</span><span class="p">;</span>
			<span class="n">free</span><span class="p">(</span><span class="n">h</span><span class="p">);</span>
		<span class="p">}</span>
		<span class="n">free</span><span class="p">(</span><span class="n">filter</span><span class="o">-&gt;</span><span class="n">bits</span><span class="p">);</span>
		<span class="n">free</span><span class="p">(</span><span class="n">filter</span><span class="p">);</span>
	<span class="p">}</span>
<span class="p">}</span>
</code></pre></div><p>These should be fairly self explanatory. The first interesting function is here:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">void</span> <span class="nf">bloom_add_hash</span><span class="p">(</span><span class="n">bloom_t</span> <span class="n">filter</span><span class="p">,</span> <span class="n">hash_function</span> <span class="n">func</span><span class="p">)</span> <span class="p">{</span>
	<span class="k">struct</span> <span class="n">bloom_hash</span> <span class="o">*</span><span class="n">h</span> <span class="o">=</span> <span class="n">calloc</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="k">struct</span> <span class="n">bloom_hash</span><span class="p">));</span>
	<span class="n">h</span><span class="o">-&gt;</span><span class="n">func</span> <span class="o">=</span> <span class="n">func</span><span class="p">;</span>
	<span class="k">struct</span> <span class="n">bloom_hash</span> <span class="o">*</span><span class="n">last</span> <span class="o">=</span> <span class="n">filter</span><span class="o">-&gt;</span><span class="n">func</span><span class="p">;</span>
	<span class="k">while</span> <span class="p">(</span><span class="n">last</span> <span class="o">&amp;&amp;</span> <span class="n">last</span><span class="o">-&gt;</span><span class="n">next</span><span class="p">)</span> <span class="p">{</span>
		<span class="n">last</span> <span class="o">=</span> <span class="n">last</span><span class="o">-&gt;</span><span class="n">next</span><span class="p">;</span>
	<span class="p">}</span>
	<span class="k">if</span> <span class="p">(</span><span class="n">last</span><span class="p">)</span> <span class="p">{</span>
		<span class="n">last</span><span class="o">-&gt;</span><span class="n">next</span> <span class="o">=</span> <span class="n">h</span><span class="p">;</span>
	<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
		<span class="n">filter</span><span class="o">-&gt;</span><span class="n">func</span> <span class="o">=</span> <span class="n">h</span><span class="p">;</span>
	<span class="p">}</span>
<span class="p">}</span>
</code></pre></div><p>Given a hashing function from the user, this just adds it to our linked list of
hash functions. There&rsquo;s a slightly different code path if we&rsquo;re adding the first
function. The functions so far don&rsquo;t really do anything specific to bloom
filters. The first one that does is this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">void</span> <span class="nf">bloom_add</span><span class="p">(</span><span class="n">bloom_t</span> <span class="n">filter</span><span class="p">,</span> <span class="k">const</span> <span class="kt">void</span> <span class="o">*</span><span class="n">item</span><span class="p">)</span> <span class="p">{</span>
	<span class="k">struct</span> <span class="n">bloom_hash</span> <span class="o">*</span><span class="n">h</span> <span class="o">=</span> <span class="n">filter</span><span class="o">-&gt;</span><span class="n">func</span><span class="p">;</span>
	<span class="n">uint8_t</span> <span class="o">*</span><span class="n">bits</span> <span class="o">=</span> <span class="n">filter</span><span class="o">-&gt;</span><span class="n">bits</span><span class="p">;</span>
	<span class="k">while</span> <span class="p">(</span><span class="n">h</span><span class="p">)</span> <span class="p">{</span>
		<span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">hash</span> <span class="o">=</span> <span class="n">h</span><span class="o">-&gt;</span><span class="n">func</span><span class="p">(</span><span class="n">item</span><span class="p">);</span>
		<span class="n">hash</span> <span class="o">%=</span> <span class="n">filter</span><span class="o">-&gt;</span><span class="n">size</span> <span class="o">*</span> <span class="mi">8</span><span class="p">;</span>
		<span class="n">bits</span><span class="p">[</span><span class="n">hash</span> <span class="o">/</span> <span class="mi">8</span><span class="p">]</span> <span class="o">|=</span> <span class="mi">1</span> <span class="o">&lt;&lt;</span> <span class="n">hash</span> <span class="o">%</span> <span class="mi">8</span><span class="p">;</span>
		<span class="n">h</span> <span class="o">=</span> <span class="n">h</span><span class="o">-&gt;</span><span class="n">next</span><span class="p">;</span>
	<span class="p">}</span>
<span class="p">}</span>
</code></pre></div><p>This iterates over each of the hash functions the user has provided and computes
the hash of the data for that function (modulo the size of our bloom filter),
then it adds this to the bloom filter with this line:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">bits</span><span class="p">[</span><span class="n">hash</span> <span class="o">/</span> <span class="mi">8</span><span class="p">]</span> <span class="o">|=</span> <span class="mi">1</span> <span class="o">&lt;&lt;</span> <span class="n">hash</span> <span class="o">%</span> <span class="mi">8</span><span class="p">;</span>
</code></pre></div><p>This just sets the nth bit of the filter where n is the hash. Finally, we have
the test function:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">bool</span> <span class="nf">bloom_test</span><span class="p">(</span><span class="n">bloom_t</span> <span class="n">filter</span><span class="p">,</span> <span class="k">const</span> <span class="kt">void</span> <span class="o">*</span><span class="n">item</span><span class="p">)</span> <span class="p">{</span>
	<span class="k">struct</span> <span class="n">bloom_hash</span> <span class="o">*</span><span class="n">h</span> <span class="o">=</span> <span class="n">filter</span><span class="o">-&gt;</span><span class="n">func</span><span class="p">;</span>
	<span class="n">uint8_t</span> <span class="o">*</span><span class="n">bits</span> <span class="o">=</span> <span class="n">filter</span><span class="o">-&gt;</span><span class="n">bits</span><span class="p">;</span>
	<span class="k">while</span> <span class="p">(</span><span class="n">h</span><span class="p">)</span> <span class="p">{</span>
		<span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">hash</span> <span class="o">=</span> <span class="n">h</span><span class="o">-&gt;</span><span class="n">func</span><span class="p">(</span><span class="n">item</span><span class="p">);</span>
		<span class="n">hash</span> <span class="o">%=</span> <span class="n">filter</span><span class="o">-&gt;</span><span class="n">size</span> <span class="o">*</span> <span class="mi">8</span><span class="p">;</span>
		<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="p">(</span><span class="n">bits</span><span class="p">[</span><span class="n">hash</span> <span class="o">/</span> <span class="mi">8</span><span class="p">]</span> <span class="o">&amp;</span> <span class="mi">1</span> <span class="o">&lt;&lt;</span> <span class="n">hash</span> <span class="o">%</span> <span class="mi">8</span><span class="p">))</span> <span class="p">{</span>
			<span class="k">return</span> <span class="nb">false</span><span class="p">;</span>
		<span class="p">}</span>
		<span class="n">h</span> <span class="o">=</span> <span class="n">h</span><span class="o">-&gt;</span><span class="n">next</span><span class="p">;</span>
	<span class="p">}</span>
	<span class="k">return</span> <span class="nb">true</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div><p>This function is extremely similar, but instead of setting the nth bit, it
checks the nth bit and returns if it&rsquo;s 0:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="p">(</span><span class="n">bits</span><span class="p">[</span><span class="n">hash</span> <span class="o">/</span> <span class="mi">8</span><span class="p">]</span> <span class="o">&amp;</span> <span class="mi">1</span> <span class="o">&lt;&lt;</span> <span class="n">hash</span> <span class="o">%</span> <span class="mi">8</span><span class="p">))</span> <span class="p">{</span>
</code></pre></div><p>That&rsquo;s it! You have a bloom filter with arbitrary data types for insert and
user-supplied hash functions. I wrote up some simple test code to demonstrate
this, after googling for a couple of random hash functions:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="cp">#include</span> <span class="cpf">&#34;bloom.h&#34;</span><span class="cp">
</span><span class="cp">#include</span> <span class="cpf">&lt;stdio.h&gt;</span><span class="cp">
</span><span class="cp"></span>
<span class="kt">unsigned</span> <span class="kt">int</span> <span class="nf">djb2</span><span class="p">(</span><span class="k">const</span> <span class="kt">void</span> <span class="o">*</span><span class="n">_str</span><span class="p">)</span> <span class="p">{</span>
	<span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">str</span> <span class="o">=</span> <span class="n">_str</span><span class="p">;</span>
	<span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">hash</span> <span class="o">=</span> <span class="mi">5381</span><span class="p">;</span>
	<span class="kt">char</span> <span class="n">c</span><span class="p">;</span>
	<span class="k">while</span> <span class="p">((</span><span class="n">c</span> <span class="o">=</span> <span class="o">*</span><span class="n">str</span><span class="o">++</span><span class="p">))</span> <span class="p">{</span>
		<span class="n">hash</span> <span class="o">=</span> <span class="p">((</span><span class="n">hash</span> <span class="o">&lt;&lt;</span> <span class="mi">5</span><span class="p">)</span> <span class="o">+</span> <span class="n">hash</span><span class="p">)</span> <span class="o">+</span> <span class="n">c</span><span class="p">;</span>
	<span class="p">}</span>
	<span class="k">return</span> <span class="n">hash</span><span class="p">;</span>
<span class="p">}</span>

<span class="kt">unsigned</span> <span class="kt">int</span> <span class="nf">jenkins</span><span class="p">(</span><span class="k">const</span> <span class="kt">void</span> <span class="o">*</span><span class="n">_str</span><span class="p">)</span> <span class="p">{</span>
	<span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">key</span> <span class="o">=</span> <span class="n">_str</span><span class="p">;</span>
	<span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">hash</span><span class="p">,</span> <span class="n">i</span><span class="p">;</span>
	<span class="k">while</span> <span class="p">(</span><span class="o">*</span><span class="n">key</span><span class="p">)</span> <span class="p">{</span>
		<span class="n">hash</span> <span class="o">+=</span> <span class="o">*</span><span class="n">key</span><span class="p">;</span>
		<span class="n">hash</span> <span class="o">+=</span> <span class="p">(</span><span class="n">hash</span> <span class="o">&lt;&lt;</span> <span class="mi">10</span><span class="p">);</span>
		<span class="n">hash</span> <span class="o">^=</span> <span class="p">(</span><span class="n">hash</span> <span class="o">&gt;&gt;</span> <span class="mi">6</span><span class="p">);</span>
		<span class="n">key</span><span class="o">++</span><span class="p">;</span>
	<span class="p">}</span>
	<span class="n">hash</span> <span class="o">+=</span> <span class="p">(</span><span class="n">hash</span> <span class="o">&lt;&lt;</span> <span class="mi">3</span><span class="p">);</span>
	<span class="n">hash</span> <span class="o">^=</span> <span class="p">(</span><span class="n">hash</span> <span class="o">&gt;&gt;</span> <span class="mi">11</span><span class="p">);</span>
	<span class="n">hash</span> <span class="o">+=</span> <span class="p">(</span><span class="n">hash</span> <span class="o">&lt;&lt;</span> <span class="mi">15</span><span class="p">);</span>
	<span class="k">return</span> <span class="n">hash</span><span class="p">;</span>
<span class="p">}</span>

<span class="kt">int</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
	<span class="n">bloom_t</span> <span class="n">bloom</span> <span class="o">=</span> <span class="n">bloom_create</span><span class="p">(</span><span class="mi">8</span><span class="p">);</span>
	<span class="n">bloom_add_hash</span><span class="p">(</span><span class="n">bloom</span><span class="p">,</span> <span class="n">djb2</span><span class="p">);</span>
	<span class="n">bloom_add_hash</span><span class="p">(</span><span class="n">bloom</span><span class="p">,</span> <span class="n">jenkins</span><span class="p">);</span>
	<span class="n">printf</span><span class="p">(</span><span class="s">&#34;Should be 0: %d</span><span class="se">\n</span><span class="s">&#34;</span><span class="p">,</span> <span class="n">bloom_test</span><span class="p">(</span><span class="n">bloom</span><span class="p">,</span> <span class="s">&#34;hello world&#34;</span><span class="p">));</span>
	<span class="n">bloom_add</span><span class="p">(</span><span class="n">bloom</span><span class="p">,</span> <span class="s">&#34;hello world&#34;</span><span class="p">);</span>
	<span class="n">printf</span><span class="p">(</span><span class="s">&#34;Should be 1: %d</span><span class="se">\n</span><span class="s">&#34;</span><span class="p">,</span> <span class="n">bloom_test</span><span class="p">(</span><span class="n">bloom</span><span class="p">,</span> <span class="s">&#34;hello world&#34;</span><span class="p">));</span>
	<span class="n">printf</span><span class="p">(</span><span class="s">&#34;Should (probably) be 0: %d</span><span class="se">\n</span><span class="s">&#34;</span><span class="p">,</span> <span class="n">bloom_test</span><span class="p">(</span><span class="n">bloom</span><span class="p">,</span> <span class="s">&#34;world hello&#34;</span><span class="p">));</span>
	<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div><p>The full code is available <a href="https://git.sr.ht/~sircmpwn/bloom/">here</a>.</p>


guid: https://drewdevault.com/2016/04/20/State-of-sway-April-2016.html
title: State of Sway - April 2016
url: https://drewdevault.com/2016/04/20/State-of-sway-April-2016.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Since the previous <a href="https://drewdevault.com/2015/12/20/State-of-sway.html">State of Sway</a>, we have
accomplished quite a bit. We are now shipping versioned releases of sway, which
include support for window borders, input device configuration, more new
features, and many bug fixes and stability improvements. I&rsquo;m also happy to say
that Sway 0.5 has landed in the Arch Linux community repository and I&rsquo;m starting
to hear rumors of it landing in other Linux distros as well. Here&rsquo;s a quick
rundown of what&rsquo;s happened in the past four months:</p>
<ul>
<li>Window borders work now</li>
<li>Input devices are configurable</li>
<li>swaybar is much more mature, including support for i3status and i3blocks</li>
<li>swaylock has reached a similar level of maturity</li>
<li>New <code>include</code> config command to include sub-configs</li>
<li>We have a <a href="https://github.com/SirCmpwn/sway/blob/master/assets/Sway_Wallpaper_Blue_1920x1080.png">default wallpaper</a> and a logo now</li>
<li>musl libc support has been added</li>
<li>More features of the i3 IPC protocol have been implemented</li>
<li>18 more i3 commands have been implemented</li>
<li>Many improvements to documentation</li>
<li>Hundreds of bug fixes and small improvements</li>
</ul>
<p>I&rsquo;m a particularly big fan of the new include command, which allows me to add
this to my config file:</p>
<pre><code>include ~/.config/sway/config.d/`hostname`/*
</code></pre>
<p>The net of this is that it includes a set of configs specific to each machine I
run Sway on, which each have a unique output device &amp; input device configuration
and several other details, but I can include them all under
<a href="https://gogs.sr.ht/SirCmpwn/dotfiles">version control</a> to keep my dotfiles
synced between computers.</p>
<p>Today, sway looks like this:</p>
<p><a href="https://sr.ht/me1j.png"><img src="https://sr.ht/me1j.png" alt=""></a></p>
<p>We&rsquo;re now making our way towards Sway 1.0. I have put together a roadmap of the
things we have done and the things that remain to do for Sway 1.0, which is
available on the improved website <a href="http://swaywm.org/roadmap">here</a>. We are
still now moving forward on many of these features, including the most asked for
feature: the stacked &amp; tabbed window layouts, which is under development from
Mikkel Oscar Lyderik. He&rsquo;s given me this screenshot to tease you with:</p>
<p><img src="https://sr.ht/0CkR.png" alt=""></p>
<p>All of this is only possible thanks to the hard work of dozens of contributors.
Here&rsquo;s the breakdown of <strong>lines of code per author</strong> for the top ten authors
(with the difference from the previous State of Sway in parenthesis):</p>
<table class="table">
    <tbody>
        <tr><td>4307 (+3180)</td><td>Mikkel Oscar Lyderik</td></tr>
        <tr><td>3059 (-457)</td><td>Drew DeVault</td></tr>
        <tr><td>2285 (+115)</td><td>taiyu</td></tr>
        <tr><td>1826 (+40)</td><td>S. Christoffer Eliesen</td></tr>
        <tr><td>682 (-38)</td><td>Luminarys</td></tr>
        <tr><td>544 (+544)</td><td>Cole Mickens</td></tr>
        <tr><td>515 (-19)</td><td>minus</td></tr>
        <tr><td>385 (+185)</td><td>Christoph Gysin</td></tr>
        <tr><td>345 (+266)</td><td>Kevin Hamacher</td></tr>
        <tr><td>166 (+45)</td><td>crondog</td></tr>
    </tbody>
</table>
<p>Once again, I&rsquo;m no longer the author of the most lines of code. Sway now
has a grand total of 15,422 lines of C and 2,787 lines of headers. Here&rsquo;s the
total <strong>number of commits per author</strong> for each of the top 10
committers:</p>
<table class="table">
    <tbody>
        <tr><td>688</td><td> Drew DeVault</td></tr>
        <tr><td>212</td><td> Mikkel Oscar Lyderik</td></tr>
        <tr><td>191</td><td> taiyu</td></tr>
        <tr><td>109</td><td> S. Christoffer Eliesen</td></tr>
        <tr><td>97</td><td> Luminarys</td></tr>
        <tr><td>58</td><td> Christoph Gysin</td></tr>
        <tr><td>34</td><td> minus</td></tr>
        <tr><td>18</td><td> crondog</td></tr>
        <tr><td>13</td><td> Yacine Hmito</td></tr>
        <tr><td>12</td><td> progandy</td></tr>
    </tbody>
</table>
<p>As the maintainer of sway, <em>a lot</em> of what I do is reviewing and merging
contributions from others. So these statistics change a bit if we use <strong>number
of commits per author, excluding merge commits</strong>:</p>
<table class="table">
    <tbody>
        <tr><td>343</td><td> Drew DeVault</td></tr>
        <tr><td>201</td><td> Mikkel Oscar Lyderik</td></tr>
        <tr><td>175</td><td> taiyu</td></tr>
        <tr><td>109</td><td> S. Christoffer Eliesen</td></tr>
        <tr><td>96</td><td> Luminarys</td></tr>
        <tr><td>58</td><td> Christoph Gysin</td></tr>
        <tr><td>34</td><td> minus</td></tr>
        <tr><td>18</td><td> crondog</td></tr>
        <tr><td>13</td><td> Yacine Hmito</td></tr>
        <tr><td>12</td><td> progandy</td></tr>
    </tbody>
</table>
<p>These stats only cover the top ten in each, but there are more - check out the
<a href="https://github.com/SirCmpwn/sway/graphs/contributors">full list</a>. Hopefully
next time I write a blog post like this, we&rsquo;ll be well into the lifetime of Sway
1.0!</p>


guid: https://drewdevault.com/2016/05/11/In-Memoriam-Mozilla.html
title: In Memoriam - Mozilla
url: https://drewdevault.com/2016/05/11/In-Memoriam-Mozilla.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Today we look back to the life of Mozilla, a company that was best known for
creating the Firefox web browser. I remember a company that made the web better
and more open by providing a browser that was faster and more customizable than
anyone had ever seen, and by making that browser free and open source.</p>
<p>I expect many of my readers will be older than I am, but my first memories of
Firefox are back in high school with Firefox 3. I fondly remember my discovery
of it. Mozilla gave us a faster and more powerful web browser to use on school
computers. The other choice was Internet Explorer 6 - but with a flash drive we
could run a &ldquo;portable&rdquo; version of Firefox instead. Using tabbed web browsing was
a clear improvement for usability and I loved installing all sorts of cool
add-ons and I&rsquo;m sure I&rsquo;ve spent at least a few hours of my life browsing persona
themes.</p>
<p>Mozilla continued to improve their web browser, and I loved it. As I grew up and
learned more about techology and started making my way into programming I loved
it even more. I remember a time when I would tell my friends that I&rsquo;d gladly
appoint Mozilla as the steward of the open internet over the W3C. Firefox
continued to evolve and allow for even more customiziability. Firefox truly
became a <a href="http://www.catb.org/~esr/jargon/html/H/hacker.html">hacker</a>&rsquo;s web
browser.</p>
<p>Eventually a new player called Chrome arrived on the scene. It was slick and new
and very, very fast. Firefox, on the other hand, appeared to become stagnant.
I made the switch to Chrome for a few years. However, to my eventual delight,
Mozilla didn&rsquo;t quit. They kept making Firefox better and faster and continued to
win on customizability and continued to fight for the best internet possible.
One day I tried Firefox again and I found it to be just as friendly and hackable
as it once was, only now it was a speed demon on par with Chrome. I returned to
Firefox for several happy years.</p>
<p>Chrome adopted a versioning scheme that made Mozilla nervous. They didn&rsquo;t like
being Firefox 4 next to Chrome 11. They made the first of many compromises when
they switched to bumping the major version with each release. Mozilla died in
April of 2011.</p>
<p>In Mozilla&rsquo;s place, a new company appeared and started to build a new browser.
This new company had good intentions, but has completely lost the spirit of
Mozilla. This new browser is a stain on Mozilla&rsquo;s legacy - it ships with
unremovable nonfree add-ons, removes huge swaths of the original add-on API,
includes a cryptographically walled garden for add-ons, and apparently now
includes an instant messaging and video conferencing platform.</p>
<p>The new company has been suffering as well. They have sunk enormous time and
effort into projects that are doomed from the start. They tried to make a mobile
phone OS whose UI was powered by technology that&rsquo;s been proven to produce an
inferior mobile experience (HTML+CSS+JS) using the slowest rendering engine on
the market (gecko) on the lowest powered phones on the market. When this
predictably failed, they turned their sights towards running it on even lower
powered IoT devices. This new company has also announced several times that they
are killing off another well established and well loved project (Thunderbird)
from the old Mozilla. They also recently struck a deal with another dying
company, Yahoo, to make their search engine the default for this &ldquo;neo-Firefox&rdquo;.</p>
<p>To the new company that calls itself Mozilla: you do an injustice to the memory
of Mozilla. I hope that one day we&rsquo;ll see the Mozilla of the past return.</p>


guid: https://drewdevault.com/2016/05/28/Understanding-pointers.html
title: Understanding pointers
url: https://drewdevault.com/2016/05/28/Understanding-pointers.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I was recently chatting with a new contributor to Sway who is using the project
as a means of learning C, and he had some questions about what <code>void**</code> meant
when he found some in the code. It became apparent that this guy only has a
basic grasp on pointers at this point in his learning curve, and I figured it
was time for another blog post - so today, I&rsquo;ll explain pointers.</p>
<p>To understand pointers, you must first understand how memory works. Your RAM is
basically a flat array of
<a href="https://en.wikipedia.org/wiki/Octet_(computing)">octets</a>. Your compiler
describes every data structure you use as a series of octets. For the context of
this article, let&rsquo;s consider the following memory:</p>
<p>{:.table}</p>
<table>
<thead>
<tr>
<th style="text-align:left">0x0000</th>
<th style="text-align:left">0x0001</th>
<th style="text-align:left">0x0002</th>
<th style="text-align:left">0x0003</th>
<th style="text-align:left">0x0004</th>
<th style="text-align:left">0x0005</th>
<th style="text-align:left">0x0006</th>
<th style="text-align:left">0x0007</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x08</td>
<td style="text-align:left">0x42</td>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x00</td>
</tr>
<tr>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
</tr>
</tbody>
</table>
<p>We can refer to each element of this array by its index, or address. For
example, the value at address 0x0004 is 0x08. On this system, we&rsquo;re using 16-bit
addresses to refer to 8-bit values. On an i686 (32-bit) system, we use 32-bit
addresses to refer to 8-bit values. On an amd64 (64-bit) system, we use 64-bit
addresses to refer to 8-bit values. On Notch&rsquo;s imaginary DCPU-16 system, we use
16-bit addresses to refer to 16-bit values.</p>
<p>To refer to the value at 0x0004, we can use a pointer. Let&rsquo;s declare it like so:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">uint8_t</span> <span class="o">*</span><span class="n">value</span> <span class="o">=</span> <span class="p">(</span><span class="n">uint8_t</span> <span class="o">*</span><span class="p">)</span><span class="mh">0x0004</span><span class="p">;</span>
</code></pre></div><p>Here we&rsquo;re declaring a variable named value, whose type is <code>uint8_t*</code>. The *
indicates that it&rsquo;s a pointer. Now, because this is a 16-bit system, the size of
a pointer is 16 bits. If we do this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">printf</span><span class="p">(</span><span class="s">&#34;%d</span><span class="se">\n</span><span class="s">&#34;</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">value</span><span class="p">));</span>
</code></pre></div><p>It will print 2, because it takes 16-bits (or 2 bytes) to refer to an address on
this system, even though the value there is 8 bits. On your system it would
probably print 8, or maybe 4 if you&rsquo;re on a 32-bit system. We could also do this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">uint16_t</span> <span class="n">address</span> <span class="o">=</span> <span class="mh">0x0004</span><span class="p">;</span>
<span class="n">uint8_t</span> <span class="o">*</span><span class="n">ptr</span> <span class="o">=</span> <span class="p">(</span><span class="n">uint8_t</span> <span class="o">*</span><span class="p">)</span><span class="n">address</span><span class="p">;</span>
</code></pre></div><p>In this case we&rsquo;re not casting the <code>uint16_t</code> value 0x0004 to a <code>uint8_t</code>, which
would truncate the integer. No, instead, we&rsquo;re casting it to a <code>uint8_t*</code>, which
is the size required to represent a pointer on this system. All pointers are the
same size.</p>
<h2 id="dereferencing-pointers">Dereferencing pointers</h2>
<p>We can refer to the value at the other end of this pointer by <em>dereferencing</em> it.
The pointer is said to contain a <em>reference</em> to a value in memory. By
<em>dereferencing</em> it, we can obtain that value. For example:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">uint8_t</span> <span class="o">*</span><span class="n">value</span> <span class="o">=</span> <span class="p">(</span><span class="n">uint8_t</span> <span class="o">*</span><span class="p">)</span><span class="mh">0x0004</span><span class="p">;</span>
<span class="n">printf</span><span class="p">(</span><span class="s">&#34;%d</span><span class="se">\n</span><span class="s">&#34;</span><span class="p">,</span> <span class="o">*</span><span class="n">value</span><span class="p">);</span> <span class="c1">// prints 8
</span></code></pre></div><h2 id="working-with-multi-byte-values">Working with multi-byte values</h2>
<p>Even though memory is basically a big array of <code>uint8_t</code>, thankfully we can work
with other kinds of data structures inside of it. For example, say we wanted to
store the value 0x1234 in memory. This doesn&rsquo;t fit in 8 bits, so we need to
store it at two different addresses. For example, we could store it at 0x0006
and 0x0007:</p>
<p>{:.table}</p>
<table>
<thead>
<tr>
<th style="text-align:left">0x0000</th>
<th style="text-align:left">0x0001</th>
<th style="text-align:left">0x0002</th>
<th style="text-align:left">0x0003</th>
<th style="text-align:left">0x0004</th>
<th style="text-align:left">0x0005</th>
<th style="text-align:left">0x0006</th>
<th style="text-align:left">0x0007</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x08</td>
<td style="text-align:left">0x42</td>
<td style="text-align:left">0x34</td>
<td style="text-align:left">0x12</td>
</tr>
<tr>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
</tr>
</tbody>
</table>
<p>*0x0007 makes up the first byte of the value, and *0x0006 makes up the second
byte of the value.</p>
<div class="well">
    Why not the other way around? Well, most systems these days use the "little
    endian" notation for storing multi-byte integers in memory, which stores the
    least significant byte first. The least significant byte is the one with the
    smallest order of magnitude (in base sixteen). To get the final number, we
    use (0x12 * 0x100) + (0x34 * 0x1), which gives us 0x1234. Read more about
    endianness <a href="https://en.wikipedia.org/wiki/Endianness">here</a>.
</div>
<p>C allows us to use pointers that refer to these sorts of composite values, like
so:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">uint16_t</span> <span class="o">*</span><span class="n">value</span> <span class="o">=</span> <span class="p">(</span><span class="n">uint16_t</span> <span class="o">*</span><span class="p">)</span><span class="mh">0x0006</span><span class="p">;</span>
<span class="n">printf</span><span class="p">(</span><span class="s">&#34;0x%X</span><span class="se">\n</span><span class="s">&#34;</span><span class="p">,</span> <span class="o">*</span><span class="n">value</span><span class="p">);</span> <span class="c1">// Prints 0x1234
</span></code></pre></div><p>Here, we&rsquo;ve declared a pointer to a value whose type is <code>uint16_t</code>. Note that the
size of this pointer is the same size of the <code>uint8_t*</code> pointer - 16 bits, or
two bytes. The value it <em>references</em>, though, is a different type than
<code>uint8_t*</code> references.</p>
<h2 id="indirect-pointers">Indirect pointers</h2>
<p>Here comes the crazy part - you can work with pointers to pointers. The address
of the <code>uint16_t</code> pointer we&rsquo;ve been talking about is 0x0006, right? Well, we
can store that number in memory as well. If we store it at 0x0002, our memory
looks like this:</p>
<p>{:.table}</p>
<table>
<thead>
<tr>
<th style="text-align:left">0x0000</th>
<th style="text-align:left">0x0001</th>
<th style="text-align:left">0x0002</th>
<th style="text-align:left">0x0003</th>
<th style="text-align:left">0x0004</th>
<th style="text-align:left">0x0005</th>
<th style="text-align:left">0x0006</th>
<th style="text-align:left">0x0007</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x06</td>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x08</td>
<td style="text-align:left">0x42</td>
<td style="text-align:left">0x34</td>
<td style="text-align:left">0x12</td>
</tr>
<tr>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
</tr>
</tbody>
</table>
<p>The question might then become, how do we get it out again? Well, we can use a
pointer <em>to that pointer</em>! Check out this code:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">uint16_t</span> <span class="o">**</span><span class="n">pointer_to_a_pointer</span> <span class="o">=</span> <span class="p">(</span><span class="n">uint16_t</span><span class="o">**</span><span class="p">)</span><span class="mh">0x0002</span><span class="p">;</span>
</code></pre></div><p>This code just declared a variable whose type is <code>uint16_t**</code>, which a pointer
whose value is a <code>uint16_t*</code>, which itself points to a value that is a
<code>uint16_t</code>. Pretty cool, huh?  We can dereference this too:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">uint16_t</span> <span class="o">**</span><span class="n">pointer_to_a_pointer</span> <span class="o">=</span> <span class="p">(</span><span class="n">uint16_t</span><span class="o">**</span><span class="p">)</span><span class="mh">0x0002</span><span class="p">;</span>
<span class="n">uint16_t</span> <span class="o">*</span><span class="n">pointer</span> <span class="o">=</span> <span class="o">*</span><span class="n">pointer_to_a_pointer</span><span class="p">;</span>
<span class="n">printf</span><span class="p">(</span><span class="s">&#34;0x%X</span><span class="se">\n</span><span class="s">&#34;</span><span class="p">,</span> <span class="o">*</span><span class="n">pointer</span><span class="p">);</span> <span class="c1">// Prints 0x1234
</span></code></pre></div><p>We don&rsquo;t actually even need the intermediate variable. This works too:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">uint16_t</span> <span class="o">**</span><span class="n">pointer_to_a_pointer</span> <span class="o">=</span> <span class="p">(</span><span class="n">uint16_t</span><span class="o">**</span><span class="p">)</span><span class="mh">0x0002</span><span class="p">;</span>
<span class="n">printf</span><span class="p">(</span><span class="s">&#34;0x%X</span><span class="se">\n</span><span class="s">&#34;</span><span class="p">,</span> <span class="o">**</span><span class="n">pointer_to_a_pointer</span><span class="p">);</span> <span class="c1">// Prints 0x1234
</span></code></pre></div><h2 id="void-pointers">Void pointers</h2>
<p>The next question that would come up to your average C programmer would be,
&ldquo;well, what is a <code>void*</code>?&rdquo; Well, remember earlier when I said that all pointers,
regardless of the type of value they reference, are just fixed size integers?
In the imaginary system we&rsquo;ve been talking about, pointers are 16-bit addresses,
or indexes, that refer to places in RAM. On the system you&rsquo;re reading this
article on, it&rsquo;s probably a 64-bit integer. Well, we don&rsquo;t actually need to
specify the type to be able to manipulate pointers if they&rsquo;re just a fixed size
integer - so we don&rsquo;t have to. A <code>void*</code> stores an arbitrary address without
bringing along any type information. You can later <em>cast</em> this variable to a
specific kind of pointer to dereference it. For example:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">void</span> <span class="o">*</span><span class="n">pointer</span> <span class="o">=</span> <span class="p">(</span><span class="kt">void</span><span class="o">*</span><span class="p">)</span><span class="mh">0x0006</span><span class="p">;</span>
<span class="n">uint8_t</span> <span class="o">*</span><span class="n">uintptr</span> <span class="o">=</span> <span class="p">(</span><span class="n">uint8_t</span><span class="o">*</span><span class="p">)</span><span class="n">pointer</span><span class="p">;</span>
<span class="n">printf</span><span class="p">(</span><span class="s">&#34;0x%X&#34;</span><span class="p">,</span> <span class="o">*</span><span class="n">uintptr</span><span class="p">);</span> <span class="c1">// prints 0x34
</span></code></pre></div><p>Take a closer look at this code, and recall that 0x0006 refers to a 16-bit value
from the previous section. Here, though, we&rsquo;re treating it as an 8-bit value -
the <code>void*</code> contains no assumptions about what kind of data is there. The result
is that we end up treating it like an 8-bit integer, which ends up being the
least significant byte of 0x1234;</p>
<h2 id="dereferencing-structures">Dereferencing structures</h2>
<p>In C, we often work with structs. Let&rsquo;s describe one to play with:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">struct</span> <span class="n">coordinates</span> <span class="p">{</span>
    <span class="n">uint16_t</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">;</span>
    <span class="k">struct</span> <span class="n">coordinates</span> <span class="o">*</span><span class="n">next</span><span class="p">;</span>
<span class="p">};</span>
</code></pre></div><p>Our structure describes a linked list of coordinates. X and Y are the
coordinates, and next is a pointer to the next set of coordinates in our list.
I&rsquo;m going to drop two of these in memory:</p>
<p>{:.table}</p>
<table>
<thead>
<tr>
<th style="text-align:left">0x0000</th>
<th style="text-align:left">0x0001</th>
<th style="text-align:left">0x0002</th>
<th style="text-align:left">0x0003</th>
<th style="text-align:left">0x0004</th>
<th style="text-align:left">0x0005</th>
<th style="text-align:left">0x0006</th>
<th style="text-align:left">0x0007</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">0xAD</td>
<td style="text-align:left">0xDE</td>
<td style="text-align:left">0xEF</td>
<td style="text-align:left">0xBE</td>
<td style="text-align:left">0x06</td>
<td style="text-align:left">0x00</td>
<td style="text-align:left">0x34</td>
<td style="text-align:left">0x12</td>
</tr>
<tr>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
<td style="text-align:left">========</td>
</tr>
</tbody>
</table>
<p>Let&rsquo;s write some C code to reason about this memory with:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">struct</span> <span class="n">coordinates</span> <span class="o">*</span><span class="n">coords</span><span class="p">;</span>
<span class="n">coords</span> <span class="o">=</span> <span class="p">(</span><span class="k">struct</span> <span class="n">coordinates</span><span class="o">*</span><span class="p">)</span><span class="mh">0x0000</span><span class="p">;</span>
</code></pre></div><p>If we look at this structure in memory, you might already be able to pick out
the values. C is going to store the fields of this struct in order. So, we can
expect the following:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">printf</span><span class="p">(</span><span class="s">&#34;0x%X, 0x%X&#34;</span><span class="p">,</span> <span class="n">coords</span><span class="o">-&gt;</span><span class="n">x</span><span class="p">,</span> <span class="n">coords</span><span class="o">-&gt;</span><span class="n">y</span><span class="p">);</span>
</code></pre></div><p>To print out &ldquo;0xDEAD, 0xBEEF&rdquo;. Note that we&rsquo;re using the structure dereferencing
operator here, <code>-&gt;</code>. This allows us to dereference values <em>inside</em> of a
structure we have a pointer to. The other case is this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">printf</span><span class="p">(</span><span class="s">&#34;0x%X, 0x-X&#34;</span><span class="p">,</span> <span class="n">coords</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">coords</span><span class="p">.</span><span class="n">y</span><span class="p">);</span>
</code></pre></div><p>Which only works if <code>coords</code> is not a pointer. We also have a pointer within
this structure named next. You can see in the memory I included above that its
address is 0x0004 and its value is 0x0006 - meaning that there&rsquo;s another <code>struct coordinates</code> that lives at 0x0006 in memory. If you look there, you can see the
first part of it. It&rsquo;s X coordinate is 0x1234.</p>
<h2 id="pointer-arithmetic">Pointer arithmetic</h2>
<p>In C, we can use math on pointers. For example, we can do this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">uint8_t</span> <span class="o">*</span><span class="n">addr</span> <span class="o">=</span> <span class="p">(</span><span class="n">uint8_t</span><span class="o">*</span><span class="p">)</span><span class="mh">0x1000</span><span class="p">;</span>
<span class="n">addr</span><span class="o">++</span><span class="p">;</span>
</code></pre></div><p>Which would make the value of <code>addr</code> 0x1001. But this is only true for pointers
whose type is 1 byte in size. Consider this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">uint16_t</span> <span class="o">*</span><span class="n">addr</span> <span class="o">=</span> <span class="p">(</span><span class="n">uint16_t</span><span class="o">*</span><span class="p">)</span><span class="mh">0x1000</span><span class="p">;</span>
<span class="n">addr</span><span class="o">++</span><span class="p">;</span>
</code></pre></div><p>Here, <code>addr</code> becomes 0x1002! This is because ++ on a pointer actually adds
<code>sizeof(type)</code> to the actual address stored. The idea is that if we only added
one, we&rsquo;d be referring to an address that is <em>in the middle</em> of a uint16_t,
rather than the next uint16_t in memory that we meant to refer to. This is also
how arrays work. The following two code snippets are equivalent:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">uint16_t</span> <span class="o">*</span><span class="n">addr</span> <span class="o">=</span> <span class="p">(</span><span class="n">uint16_t</span><span class="o">*</span><span class="p">)</span><span class="mh">0x1000</span><span class="p">;</span>
<span class="n">printf</span><span class="p">(</span><span class="s">&#34;%d</span><span class="se">\n</span><span class="s">&#34;</span><span class="p">,</span> <span class="o">*</span><span class="p">(</span><span class="n">addr</span> <span class="o">+</span> <span class="mi">1</span><span class="p">));</span>
</code></pre></div><div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="n">uint16_t</span> <span class="o">*</span><span class="n">addr</span> <span class="o">=</span> <span class="p">(</span><span class="n">uint16_t</span><span class="o">*</span><span class="p">)</span><span class="mh">0x1000</span><span class="p">;</span>
<span class="n">printf</span><span class="p">(</span><span class="s">&#34;%d</span><span class="se">\n</span><span class="s">&#34;</span><span class="p">,</span> <span class="n">addr</span><span class="p">[</span><span class="mi">1</span><span class="p">]);</span>
</code></pre></div><h2 id="null-pointers">NULL pointers</h2>
<p>Sometimes you need to work with a pointer that points to something that may not
exist yet, or a resource that has been freed. In this case, we use a NULL
pointer. In the examples you&rsquo;ve seen so far, 0x0000 is a valid address. This is
just for simplicity&rsquo;s sake. In practice, pretty much no modern computer has
any reason to refer to the value at address 0. For that reason, we use NULL to
refer to an uninitialized pointer. Dereferencing a NULL pointer is generally a
Bad Thing and will lead to segfaults. As a fun side effect, since NULL is 0, we
can use it in an if statement:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">void</span> <span class="o">*</span><span class="n">ptr</span> <span class="o">=</span> <span class="p">...;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">ptr</span><span class="p">)</span> <span class="p">{</span>
    <span class="c1">// ptr is valid
</span><span class="c1"></span><span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
    <span class="c1">// ptr is not valid
</span><span class="c1"></span><span class="p">}</span>
</code></pre></div><p>I hope you found this article useful! If you&rsquo;d
like something fun to read next, read about <a href="http://c2.com/cgi/wiki?ThreeStarProgrammer">&ldquo;three star
programmers&rdquo;</a>, or programmers who
have variables like <code>void***</code>.</p>


guid: https://drewdevault.com/2016/06/29/Privacy-as-a-hobby.html
title: Life, liberty, and the pursuit of privacy
url: https://drewdevault.com/2016/06/29/Privacy-as-a-hobby.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Privacy is my hobby, and should be a hobby of every technically competent
American. Within the eyes of the law I have a right to secure the privacy of my
information. At least that&rsquo;s the current law - many officials are <a href="http://www.apple.com/customer-letter/">trying to
subvert that right</a>. I figure that we&rsquo;d
better exercise that right while we have it, so that we know how to keep
exercising it once it&rsquo;s illegal and all the information about it dries up.</p>
<p>One particularly annoying coworker often brings up, &ldquo;what do you have to hide?&rdquo;
Though it would defeat the purpose to explain what I&rsquo;m hiding, let&rsquo;s assume that
what I&rsquo;m hiding is benign, at least legally speaking. I&rsquo;m sure you can
understand why I don&rsquo;t want <code>~/Porn</code> to be public information should my
equipment be seized after I publish this blog post and an incompetent (or angry)
investigator leaks it. Building secure facilities for housing secrets is fun!
That&rsquo;s true even if there aren&rsquo;t a lot of interesting secrets to hide there.</p>
<p>But the porn folder brings up an interesting point. I&rsquo;m not ashamed to admit I
have one, but I would be uncomfortable with everyone being able to see it. Or
maybe I&rsquo;m having an affair (a scandalous proposition for a single guy, I know)
and there are relevant texts are on my cell phone. Perhaps I suck at managing my
finances and the spreadsheets in my documents would tell you so. Maybe I have
embarrassing home videos of bedroom activities on my hard drive<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. Maybe
there&rsquo;s evidence that I&rsquo;m a recovering alcoholic in my files. Maybe I&rsquo;m a
closeted homosexual and my files prove it, and 10 years from now the homophobes
win and suddenly the country is more hostile to that. Maybe all of this is true
at once!</p>
<p>Keeping these things secret is an important right, and one I intend to exercise.
I don&rsquo;t want to be accused of some crime and have my equipment seized and then
mishandled by incompetent officials and made public. I don&rsquo;t want a jury chosen
to decide if I really stole that pack of gum when I was 8 and then have
unfavorable secrets leaked. Human nature might lead them to look on my case
unfavorably if they found out about all the tentacle porn or erotic Harry
Potter fanfics I&rsquo;ve been secretly writing. Maybe an investigator finds something
they don&rsquo;t understand, like a private key, and it ends up being exposed through
the proceedings. Maybe this private key proves that I&rsquo;m Satoshi Nakamoto<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> and
my life is threatened when the case is closed because of it.</p>
<p>To the government: <strong>stay the fuck out of my right to encrypt</strong>, or, as I
like to think of it, my right to use math. They will try, again and again, to
take it from us. They must never win.</p>
<p>The second act of this blog post is advice on how to go about securing your
privacy. The crucial bit of advice is that you must strive to understand the
systems you use for privacy and security. Look for their weak spots and be aware
of them. Don&rsquo;t deceive yourself about how secure your systems are.</p>
<p>I try to identify pain points in my security model. Some of them will be hard
to swallow. The first one was Facebook - delete your account<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> <sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>. I did
this years ago. The second one was harder still - Google. I use an Android
phone running CyanogenMod without Google Play Services. I also don&rsquo;t use GMail
or any Google services (I search with DuckDuckGo and add !sp to use StartPage if
necessary). Another one was not using Windows or OS X. This is easy for me but a
lot of people will bitch and moan about it. A valid privacy &amp; security model
does not include Windows. OS X is an improvement but you&rsquo;d be better off on
Linux. Even your non-technical family can surely figure out how to use Xubuntu
to surf the web.</p>
<p>I also use browser extensions to subvert tracking and ads. Ad networks have
severely fucked themselves by this point - I absolutely never trust any ads on
the web, and never will, period. Use software like
<a href="https://github.com/gorhill/uBlock">uBlock</a> to get rid of trackers (and speed
up the web, bonus!). I also block lots of trackers in my /etc/hosts file -
<a href="https://github.com/StevenBlack/hosts">check this out</a>. Also check out
<a href="https://free-software-for-android.github.io/AdAway/">AdAway</a> for Android.</p>
<p>These changes help to remove your need to trust that corporate interests will
be good stewards of your private information. This is very important - no amount
of encryption will help you if you give Google a GPS map of your every move<sup id="fnref:5"><a href="#fn:5" class="footnote-ref" role="doc-noteref">5</a></sup>
and your search history<sup id="fnref:6"><a href="#fn:6" class="footnote-ref" role="doc-noteref">6</a></sup> and information about basically every page on the
internet you visit<sup id="fnref:7"><a href="#fn:7" class="footnote-ref" role="doc-noteref">7</a></sup>. And all of your emails and contacts and appointments on
your calendar. Google can be subpoenaed or subverted<sup id="fnref:8"><a href="#fn:8" class="footnote-ref" role="doc-noteref">8</a></sup> and many other
companies won&rsquo;t even try<sup id="fnref:9"><a href="#fn:9" class="footnote-ref" role="doc-noteref">9</a></sup> to keep your data secret even when they aren&rsquo;t
legally compelled to. I like this image from Maciej Cegłowski&rsquo;s excellent
talk<sup id="fnref:10"><a href="#fn:10" class="footnote-ref" role="doc-noteref">10</a></sup> on website obesity about the state of most websites:</p>
<p><img src="https://sr.ht/ks75.jpg" alt=""></p>
<p>When you give all of this information to Google, Facebook, and others, you&rsquo;re
basically waiving your fifth amendment<sup id="fnref:11"><a href="#fn:11" class="footnote-ref" role="doc-noteref">11</a></sup> rights.</p>
<p>Once you do have control of your information, there are steps you should take to
keep it secure. The answer is encryption. I use
<a href="https://wiki.archlinux.org/index.php/Dm-crypt">dm-crypt</a> which allows me to
encrypt my entire hard drive on Linux. I&rsquo;m prompted for a password on boot and
then everything proceeds (and I&rsquo;ve never noticed any performance issues, for the
record).</p>
<p>I also do most of my mobile computing on a laptop running libreboot<sup id="fnref:12"><a href="#fn:12" class="footnote-ref" role="doc-noteref">12</a></sup> with
100% open source software. The weak point here is that if your hardware is
compromised and you don&rsquo;t know it, they could steal your password. One possible
solution is keeping your boot partition and perhaps another key on a flash
drive, but this doesn&rsquo;t fully solve the problem. I suggest looking into things
like case intrusion detection and working on being aware of it when your
hardware is messed with.</p>
<p>I mentioned earlier that my phone is running CyanogenMod without any of the
Google apps. The weak point here is the radio, which is very insecure and likely
riddled with vulnerabilities. I intend to build my own phone soon with a
Raspberry Pi, where I can have more control over this - things like being able
to disconnect power to the radio or disconnect the microphone when not in use
will help.</p>
<p>I also self host my email, which was a huge pain in the ass to set up, but is
lovely now that I have it. At some point I intend to write a better mail server
to make this easier. I use opportunistic PGP encryption for my emails, but I
send depressingly few encrypted emails like this due to poor adoption (follow me
on <a href="https://keybase.io/sircmpwn">keybase</a>? I&rsquo;ll give you an invitation if you
send me an encrypted email asking for one!)</p>
<p>If you have any questions about how to implement any of this, help identifying
the weaknesses in your setup, or anything else, please feel free to reach out to
me via email (<a href="mailto:sir@cmpwn.com">sir@cmpwn.com</a>+<a href="https://drewdevault.com/publickey.txt">F4EA1B88</a>)
or <a href="https://twitter.com/sircmpwn">Twitter</a> or whatever. Good luck sticking it to
the man!</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p><a href="https://en.wikipedia.org/wiki/ICloud_leaks_of_celebrity_photos">ICloud leaks of celebrity photos</a> <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>The secretive inventor of Bitcoin. I&rsquo;m not Satoshi, if you were wondering. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p><a href="https://www.facebook.com/help/delete_account?rdrhc">Click this</a> to do so <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>&ldquo;But I liiiiike Facebook and it let&rsquo;s me keep up with my frieeeends&hellip;&rdquo; There&rsquo;s no privacy model that includes Facebook and works. Give up. <a href="https://stallman.org/facebook.html">Read this</a> and try to ignore the childish language and see the tangible evidence instead. <a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>If you have location services enabled on your phone, <a href="https://maps.google.com/locationhistory/">here&rsquo;s a map of everywhere you&rsquo;ve been</a>. Enjoy! <a href="#fnref:5" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p><a href="https://myactivity.google.com/myactivity">Here&rsquo;s all of your searches</a>. You can delete the history here, supposedly. I bet it doesn&rsquo;t unfeed that history to your personal advertising neural network at Google. <a href="#fnref:6" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>Google Adsense and Google Analytics are present on basically every website. I&rsquo;m positive they&rsquo;re writing it down somewhere when you hit a page with those on it. Facebook certainly is, too. <a href="#fnref:7" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>Remember <a href="https://en.wikipedia.org/wiki/PRISM">PRISM</a>? <a href="#fnref:8" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p><a href="http://www.pbs.org/newshour/rundown/report-att-cooperated-extensively-nsa-sharing-billions-phone-email-records/">Like AT&amp;T, for example</a> <a href="#fnref:9" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p><a href="http://idlewords.com/talks/website_obesity.htm">The Website Obesity Crisis</a> <a href="#fnref:10" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:11" role="doc-endnote">
<p>That&rsquo;s the right to remain silent. Come on, you should know this. <a href="#fnref:11" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:12" role="doc-endnote">
<p><a href="https://libreboot.org/">libreboot</a> is an open source BIOS. I got my laptop from <a href="https://minifree.org/">minifree</a>, which directly supports the libreboot project with their profits. <a href="#fnref:12" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2016/07/19/Using-Wl-wrap-for-mocking-in-C.html
title: Using -Wl,--wrap for mocking in C
url: https://drewdevault.com/2016/07/19/Using-Wl-wrap-for-mocking-in-C.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>One of the comforts I&rsquo;ve grown used to in higher level languages when testing
my code is mocking. The idea is that in order to test some code in isolation,
you should &ldquo;mock&rdquo; the behavior of things it depends on. Let&rsquo;s see a (contrived)
example:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">int</span> <span class="nf">read_to_end</span><span class="p">(</span><span class="n">FILE</span> <span class="o">*</span><span class="n">f</span><span class="p">,</span> <span class="kt">char</span> <span class="o">*</span><span class="n">buf</span><span class="p">)</span> <span class="p">{</span>
    <span class="kt">int</span> <span class="n">r</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">l</span><span class="p">;</span>
    <span class="k">while</span> <span class="p">(</span><span class="o">!</span><span class="n">feof</span><span class="p">(</span><span class="n">f</span><span class="p">))</span> <span class="p">{</span>
        <span class="n">l</span> <span class="o">=</span> <span class="n">fread</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">256</span><span class="p">,</span> <span class="n">f</span><span class="p">);</span>
        <span class="n">r</span> <span class="o">+=</span> <span class="n">l</span><span class="p">;</span>
        <span class="n">buf</span> <span class="o">+=</span> <span class="n">l</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="k">return</span> <span class="n">r</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div><p>If we want to test this function without mocking, we would need to actually open
a specially crafted file and provide a <code>FILE*</code> to the function. However, with
the linker <code>--wrap</code> flag, we can define a wrapper function. Using <code>-Wl,[flag]</code>
in your C compiler command line will pass <code>[flag]</code> to the linker. Gold (GNU) and
lld (LLVM) both support the wrap flag, which specifies a function to be
&ldquo;wrapped&rdquo;. If I use <code>-Wl,--wrap=fread</code>, then the code above will be compiled
like so:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">int</span> <span class="nf">read_to_end</span><span class="p">(</span><span class="n">FILE</span> <span class="o">*</span><span class="n">f</span><span class="p">,</span> <span class="kt">char</span> <span class="o">*</span><span class="n">buf</span><span class="p">)</span> <span class="p">{</span>
    <span class="kt">int</span> <span class="n">r</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">l</span><span class="p">;</span>
    <span class="k">while</span> <span class="p">(</span><span class="o">!</span><span class="n">feof</span><span class="p">(</span><span class="n">f</span><span class="p">))</span> <span class="p">{</span>
        <span class="n">l</span> <span class="o">=</span> <span class="n">__wrap_fread</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">256</span><span class="p">,</span> <span class="n">f</span><span class="p">);</span>
        <span class="n">r</span> <span class="o">+=</span> <span class="n">l</span><span class="p">;</span>
        <span class="n">buf</span> <span class="o">+=</span> <span class="n">l</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="k">return</span> <span class="n">r</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div><p>And if I add <code>-Wl,--wrap=feof</code> we&rsquo;ll get this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">int</span> <span class="nf">read_to_end</span><span class="p">(</span><span class="n">FILE</span> <span class="o">*</span><span class="n">f</span><span class="p">,</span> <span class="kt">char</span> <span class="o">*</span><span class="n">buf</span><span class="p">)</span> <span class="p">{</span>
    <span class="kt">int</span> <span class="n">r</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">l</span><span class="p">;</span>
    <span class="k">while</span> <span class="p">(</span><span class="o">!</span><span class="n">__wrap_feof</span><span class="p">(</span><span class="n">f</span><span class="p">))</span> <span class="p">{</span>
        <span class="n">l</span> <span class="o">=</span> <span class="n">__wrap_fread</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">256</span><span class="p">,</span> <span class="n">f</span><span class="p">);</span>
        <span class="n">r</span> <span class="o">+=</span> <span class="n">l</span><span class="p">;</span>
        <span class="n">buf</span> <span class="o">+=</span> <span class="n">l</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="k">return</span> <span class="n">r</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div><p>Now, we can define some functions that do the behavior we need to test instead
of invoking fread directly:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">int</span> <span class="n">feof_return_value</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>

<span class="kt">int</span> <span class="nf">__wrap_feof</span><span class="p">(</span><span class="n">FILE</span> <span class="o">*</span><span class="n">f</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">assert</span><span class="p">(</span><span class="n">f</span> <span class="o">==</span> <span class="p">(</span><span class="n">FILE</span> <span class="o">*</span><span class="p">)</span><span class="mh">0x1234</span><span class="p">);</span>
    <span class="k">return</span> <span class="n">feof_return_value</span><span class="p">;</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">test_read_to_end_eof</span><span class="p">()</span> <span class="p">{</span>
    <span class="c1">// ...
</span><span class="c1"></span>    <span class="n">feof_return_value</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
    <span class="n">read_to_end</span><span class="p">((</span><span class="n">FILE</span> <span class="o">*</span><span class="p">)</span><span class="mh">0x1234</span><span class="p">,</span> <span class="n">buf</span><span class="p">);</span>
    <span class="c1">// ...
</span><span class="c1"></span><span class="p">}</span>
</code></pre></div><p>Using <code>--wrap</code> also conveniently defines <code>__real_feof</code> and <code>__real_fread</code> if we
need them.</p>
<p>Unfortunately, you can&rsquo;t have two different wrappers for the same function in
an executable. This could lead to having to write several executables for each,
or making your wrapper function smart enough to have several configurable
outcomes.</p>
<p>Eventually I intend to write my own test framework for C, which will use
wrappers to support mocking. I want wrappers to be done automatically and have
it behave something like this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">static</span> <span class="kt">int</span> <span class="nf">fake_fread</span><span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="n">ptr</span><span class="p">,</span> <span class="n">size_t</span> <span class="n">size</span><span class="p">,</span>
        <span class="n">size_t</span> <span class="n">nmemb</span><span class="p">,</span> <span class="n">FILE</span> <span class="o">*</span><span class="n">stream</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">hello</span> <span class="o">=</span> <span class="s">&#34;Hello world!&#34;</span><span class="p">;</span>
    <span class="n">memcpy</span><span class="p">(</span><span class="n">ptr</span><span class="p">,</span> <span class="n">hello</span><span class="p">,</span> <span class="n">strlen</span><span class="p">(</span><span class="n">hello</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">);</span>
    <span class="k">return</span> <span class="n">strlen</span><span class="p">(</span><span class="n">hello</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">test_read_to_end</span><span class="p">()</span> <span class="p">{</span>
    <span class="n">FILE</span> <span class="o">*</span><span class="n">test</span> <span class="o">=</span> <span class="p">(</span><span class="n">FILE</span> <span class="o">*</span><span class="p">)</span><span class="mh">0x1234</span><span class="p">;</span>
    <span class="kt">char</span> <span class="o">*</span><span class="n">buffer</span> <span class="o">=</span> <span class="kt">char</span><span class="p">[</span><span class="mi">1024</span><span class="p">];</span>

    <span class="n">mock_t</span> <span class="o">*</span><span class="n">mock_feof</span> <span class="o">=</span> <span class="n">configure_mock</span><span class="p">(</span><span class="n">feof</span><span class="p">,</span> <span class="s">&#34;p&#34;</span><span class="p">);</span>
    <span class="n">mock_feof</span><span class="o">-&gt;</span><span class="n">call</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">returns</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span>
    <span class="n">mock_feof</span><span class="o">-&gt;</span><span class="n">returns</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>

    <span class="c1">// pzzp is pointer, size_t, size_t, pointer
</span><span class="c1"></span>    <span class="c1">// Tells us what the fread arguments look like
</span><span class="c1"></span>    <span class="n">mock_t</span> <span class="o">*</span><span class="n">mock_fread</span> <span class="o">=</span> <span class="n">configure_mock</span><span class="p">(</span><span class="n">fread</span><span class="p">,</span> <span class="s">&#34;pzzp&#34;</span><span class="p">);</span>
    <span class="n">mock_fread</span><span class="o">-&gt;</span><span class="n">exec</span><span class="p">(</span><span class="n">fake_fread</span><span class="p">);</span>

    <span class="n">read_to_end</span><span class="p">(</span><span class="n">test</span><span class="p">,</span> <span class="n">buffer</span><span class="p">);</span>

    <span class="n">assert</span><span class="p">(</span><span class="n">mock_feof</span><span class="o">-&gt;</span><span class="n">call_count</span> <span class="o">==</span> <span class="mi">2</span><span class="p">);</span>
    <span class="n">assert</span><span class="p">((</span><span class="n">FILE</span><span class="o">*</span><span class="p">)</span><span class="n">mock_feof</span><span class="o">-&gt;</span><span class="n">call</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">args</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="n">test</span><span class="p">);</span>

    <span class="n">assert</span><span class="p">(</span><span class="n">mock_fread</span><span class="o">-&gt;</span><span class="n">call_count</span> <span class="o">==</span> <span class="mi">1</span><span class="p">);</span>
    <span class="n">assert</span><span class="p">((</span><span class="n">FILE</span><span class="o">*</span><span class="p">)</span><span class="n">mock_fread</span><span class="o">-&gt;</span><span class="n">call</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">args</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="n">buffer</span><span class="p">);</span>
    <span class="n">assert</span><span class="p">((</span><span class="n">FILE</span><span class="o">*</span><span class="p">)</span><span class="n">mock_fread</span><span class="o">-&gt;</span><span class="n">call</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">args</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="n">test</span><span class="p">);</span>

    <span class="n">assert</span><span class="p">(</span><span class="n">strcmp</span><span class="p">(</span><span class="n">buffer</span><span class="p">,</span> <span class="s">&#34;Hello world!&#34;</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div>


guid: https://drewdevault.com/2016/08/02/Sway-0.9-in-retro.html
title: Sway 0.9 & One year of Sway
url: https://drewdevault.com/2016/08/02/Sway-0.9-in-retro.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Today marks one year since the <a href="https://github.com/SirCmpwn/sway/commit/6a33e1e3cddac31b762e4376e29c03ccf8f92107">initial
commit</a>
of Sway. Over the year since, we&rsquo;ve written 1,823 commits by 54 authors,
totalling 16,601 lines of C (and 1,866 lines of header files). This was written
over the course of 515 pull requests and 300 issues. Today, most i3 features are
supported. In fact, as of last week, all of the features from the i3
configuration I used before I started working on Sway are now supported by Sway.
Today, Sway looks like this (click to expand):</p>
<p><a href="https://sr.ht/ICd5.png"><img src="https://sr.ht/ICd5.png" alt=""></a></p>
<p>For those who are new to the project, <a href="http://swaywm.org">Sway</a> is an
i3-compatible Wayland compositor. That is, your existing <a href="http://i3wm.org/">i3</a>
configuration file will work as-is on Sway, and your keybindings and colors and
fonts and for_window rules and so on will all be the same. It&rsquo;s i3, but for
Wayland, plus it&rsquo;s got some bonus features. Here&rsquo;s a quick rundown of what&rsquo;s
happened since the <a href="https://drewdevault.com/2016/04/20/State-of-sway.html">previous state of Sway</a>:</p>
<ul>
<li>Stacked &amp; tabbed layouts</li>
<li>Customizable input acceleration</li>
<li>Mouse support for swaybar</li>
<li>Experimental HiDPI support</li>
<li>New features for swaylock and swaybg</li>
<li>Support for more i3 IPC features</li>
<li>Tracking of the workspace new windows should arrive on</li>
<li>Improved compatibility with i3</li>
<li>Many improvements to the documentation</li>
<li>Hundreds of bug fixes and small improvements</li>
</ul>
<p>Since the last State of Sway, we&rsquo;ve also seen packages land in the official
repositories of Gentoo, OpenSUSE Tumbleweed, and NixOS (though the last group
warn me that it&rsquo;s experimental). And now for some updated stats. Here&rsquo;s the
breakdown of <strong>lines of code per author</strong> for the top ten authors (with the
change from the previous state of Sway in parens):</p>
<table class="table">
    <tbody>
        <tr><td>4659 (+352)</td><td>Mikkel Oscar Lyderik</td></tr>
        <tr><td>3024 (-35)</td><td>Drew DeVault</td></tr>
        <tr><td>2232 (+53)</td><td>taiyu</td></tr>
        <tr><td>1786 (-40)</td><td>S. Christoffer Eliesen</td></tr>
        <tr><td>1090 (+1090)</td><td>Zandr Martin</td></tr>
        <tr><td>619 (-63)</td><td>Luminarys</td></tr>
        <tr><td>525 (-19)</td><td>Cole Mickens</td></tr>
        <tr><td>461 (-54)</td><td>minus</td></tr>
        <tr><td>365 (-20)</td><td>Christoph Gysin</td></tr>
        <tr><td>334 (-11)</td><td>Kevin Hamacher</td></tr>
    </tbody>
</table>
<p>Notably, Zandr Martin has started regular contributions to Sway and brought
himself right up to 5th place in a short time, and while he&rsquo;s still learning C to
boot. Not included here are his recent forays into contributing to our
dependencies as well. Thanks man! This time around, I also lost a much more
respectable line count - only 35 compared to 457 from the last update.</p>
<p>Here&rsquo;s the total <strong>number of commits per author</strong> for each of the top ten
committers:</p>
<table class="table">
    <tbody>
        <tr><td>842</td><td> Drew DeVault</td></tr>
        <tr><td>239</td><td> Mikkel Oscar Lyderik</td></tr>
        <tr><td>186</td><td> taiyu</td></tr>
        <tr><td>97</td><td> Luminarys</td></tr>
        <tr><td>91</td><td> S. Christoffer Eliesen</td></tr>
        <tr><td>58</td><td> Christoph Gysin</td></tr>
        <tr><td>48</td><td> Zandr Martin</td></tr>
        <tr><td>30</td><td> minus</td></tr>
        <tr><td>25</td><td> David Eklov</td></tr>
        <tr><td>24</td><td> Mykyta Holubakha</td></tr>
    </tbody>
</table>
<p>Most of what I do for Sway personally is reviewing and merging pull requests.
Here&rsquo;s the same figures using <strong>number of commits per author, excluding merge
commits</strong>, which changes my stats considerably:</p>
<table class="table">
    <tbody>
        <tr><td>383</td><td> Drew DeVault</td></tr>
        <tr><td>224</td><td> Mikkel Oscar Lyderik</td></tr>
        <tr><td>170</td><td> taiyu</td></tr>
        <tr><td>96</td><td> Luminarys</td></tr>
        <tr><td>91</td><td> S. Christoffer Eliesen</td></tr>
        <tr><td>58</td><td> Christoph Cysin</td></tr>
        <tr><td>38</td><td> Zandr Martin</td></tr>
        <tr><td>30</td><td> minus</td></tr>
        <tr><td>25</td><td> David Eklov</td></tr>
        <tr><td>24</td><td> Mykyta Holubakha</td></tr>
    </tbody>
</table>
<p>These stats only cover the top ten in each, but there are more - check out the
<a href="https://github.com/SirCmpwn/sway/graphs/contributors">full list</a>.</p>
<p>Sway is still going very strong, and continues developing at a fast pace. I&rsquo;ve
updated <a href="http://swaywm.org/roadmap">the roadmap</a> with our plans for Sway 1.0.
You might notice a few features have been reprioritized here, which increases
the scope of Sway 1.0. It&rsquo;ll be worth it, though, to make sure we have a solid
1.0 release. Hopefully we&rsquo;ll see that and more within the year ahead!</p>


guid: https://drewdevault.com/2016/08/18/Arch-Linux-with-full-disk-encryption-in-15-minutes.html
title: [VIDEO] Arch Linux with full disk encryption in (about) 15 minutes
url: https://drewdevault.com/2016/08/18/Arch-Linux-with-full-disk-encryption-in-15-minutes.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<link rel="stylesheet" href="https://drewdevault.com/css/video-js.css">
<script>
window.HELP_IMPROVE_VIDEOJS = false;
</script>
<script src="https://drewdevault.com/js/video.js"></script>
<p>After my <a href="https://drewdevault.com/2016/06/29/Privacy-as-a-hobby.html">blog post</a> emphasizing the
importance of taking control of your privacy, I&rsquo;ve decided to make a few more
posts going over detailed instructions on how to actually do so. Today we have a
video that goes over the process of installing Arch Linux with full disk
encryption.</p>
<p>This is my first go at publishing videos on my blog, so please provide some
feedback in the comments of this article. I&rsquo;d prefer to use my blog instead of
YouTube for publishing technical videos, since it&rsquo;s all open source, ad-free,
and DRM-free. Let me know if you&rsquo;d like to see more content like this on my
blog and which topics you&rsquo;d like covered - I intend to at least release another
video going over this process for Ubuntu as well.</p>
<video class="video-js vjs-16-9" data-setup="{}" controls>
  <source src="https://sr.ht/archlinux.webm" type="video/webm">
  <p>Your browser does not support HTML5 video.</p>
</video>
<p><a class="pull-right" href="https://sr.ht/archlinux.webm">Download video (WEBM)</a></p>
<div class="clearfix"></div>
<p>The video goes into detail on each of these steps, but here&rsquo;s the high level
overview of how to do this. Always check the latest version of the <a href="https://wiki.archlinux.org/index.php/Installation_guide">Install
Guide</a> and the
<a href="https://wiki.archlinux.org/index.php/Dm-crypt">dm-crypt</a> page on the Arch Wiki
for the latest procedure.</p>
<ol>
<li>Partition your disks with gdisk and be sure to set aside a partition for
/boot</li>
<li>Create a filesystem on /boot</li>
<li>(optional) Securely erase all of the existing data on your disks with <code>dd if=/dev/zero of=/dev/sdXY bs=4096</code> - <em>note: this is a correction from the
command mentioned in the video</em></li>
<li>Set up encryption for your encrypted partitions with <code>cryptsetup luksFormat /dev/sdXX</code></li>
<li>Open the encrypted volumes with <code>cryptsetup open /dev/sdXX [name]</code></li>
<li>Create filesystems on /dev/mapper/[names]</li>
<li>Mount all of the filesystems on /mnt</li>
<li>Perform the base install with <code>pacstrap /mnt base [extra packages...]</code></li>
<li><code>genfstab -p /mnt &gt;&gt; /mnt/etc/fstab</code></li>
<li><code>arch-chroot /mnt /usr/bin/bash</code></li>
<li><code>ln -s /usr/share/zoneinfo/[region]/[zone] /etc/localtime</code></li>
<li><code>hwclock --systohc --utc</code></li>
<li>Edit /etc/locale.gen to your liking and run <code>locale-gen</code></li>
<li><code>locale &gt; /etc/locale.conf</code> - note this only works for en_US users, adjust if
necessary</li>
<li>Edit /etc/hostname to your liking</li>
<li>Reconfigure the network</li>
<li>Edit /etc/mkinitcpio.conf and ensure that the <code>keyboard</code> and <code>encrypt</code> hooks
run before the <code>filesystems</code> hook</li>
<li><code>mkinitcpio -p linux</code></li>
<li>Set the root password with <code>passwd</code></li>
<li>Configure /etc/crypttab with any non-root encrypted disks you need. You can
get partition UUIDs with <code>ls -l /dev/disk/by-partuuid</code></li>
<li>Configure your kernel command line to include
<code>cryptdevice=PARTUUID=[...]:[name] root=/dev/mapper/[name] rw</code></li>
<li>Install your bootloader and reboot!</li>
</ol>


guid: https://drewdevault.com/2016/09/09/What-motivates-the-authors-of-the-software-you-use.html
title: What motivates the authors of the software you use?
url: https://drewdevault.com/2016/09/09/What-motivates-the-authors-of-the-software-you-use.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>We face an important choice in our lives as technophiles, hackers, geeks: the
choice between proprietary software and free/open source software. What
platforms we choose to use are important. We have a choice between Windows, OS
X, and Linux (not to mention the several less popular choices). We choose
between Android or iOS. We choose hardware that requires nonfree drivers or ones
that don&rsquo;t. We choose to store our data in someone else&rsquo;s cloud or in our own.
How do we make the right choice?</p>
<p>I think it&rsquo;s important to consider the basic motivations behind the software you
choose to use. Why did the author write it? What are their goals? How might that
influence the future (or present) direction of this software?</p>
<p>In the case of most proprietary software, the motivations are to make money.
They make decisions that benefit the company rather than the user. If you&rsquo;re
paying for the software, they might use vendor lock-in strategies to prevent you
from having ownership of your data. If you don&rsquo;t pay for the software, they
might place ads on it, sell your personal information, etc. When Cloud Storage
Incorporated is sold to Somewhat Less Trustworthy Business, who&rsquo;s to say that
your data is in good hands?</p>
<p>In the case of most open source<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> software, however, things are different.
The decisions the developers make are generally working in the interests of the
user. In open source, people work as people, not as companies. You can find the
name and email address of the person who wrote a particular feature and send
them bugs and questions.</p>
<p>An open source Facebook wouldn&rsquo;t be rearranging and filtering your timeline to
best suit their advertisers interests. An open source iCloud would include
import and export tools so you can take your data elsewhere if you so choose. An
open source phone wouldn&rsquo;t be loaded with unremovable crapware, and even if it
was, you could patch it.</p>
<p>When you install software on Linux, you get cryptographically verified packages
from individuals you can trust. You can look up who packaged your software and
get to know them personally, or even help them out! You can download the files
necessary to build the package from scratch and do so, adding any tweaks and
customizations as you wish. You don&rsquo;t have a human point of contact for Facebook
or GMail.</p>
<p>Yes, there is a usability tradeoff. It is often more difficult to use open
source software. However, it&rsquo;s also often more powerful, tweakable, flexible,
and hackable.</p>
<p>Next time you decide what software <em>you</em> should use, ask yourself: does this
software serve my interests or someone else&rsquo;s?</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>I&rsquo;m certain some readers will take offense at my language choice in this article with respect to free/libre/open source software - I chose my words intentionally. I&rsquo;ll talk more about my opinions on the free software movement in a later post. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2016/09/17/Use-the-right-tool.html
title: Using the right tool for the job
url: https://drewdevault.com/2016/09/17/Use-the-right-tool.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>One of the most important choices you&rsquo;ll make for the software you write is what
you write it in, what frameworks you use, the design methodologies to subscribe
to, and so on. This choice doesn&rsquo;t seem to get the respect it&rsquo;s due. These are
some of the only choices you&rsquo;ll make that <em>you cannot change</em>. Or, at least,
these choices are among the most difficult ones to change.</p>
<p>People often question why TrueCraft is written in C# next to projects like Sway
in C, alongside KnightOS in Assembly or sr.ht in Python. It would certainly be
easier from the outset if I made every project in a language I&rsquo;m comfortable
with, using tools and libraries I&rsquo;m comfortable with, and there&rsquo;s certainly
something to be had for that. That&rsquo;s far from being the only concern, though.</p>
<p>A new project is a <em>great</em> means of learning a new language or framework - the
only effective means, in fact. However, the inspiration and drive for new
projects doesn&rsquo;t come often. I think that the opportunity for learning is more
important than the short term results of producing working code more quickly.
Making a choice that&rsquo;s more well suited to the problem at the expense of comfort
will also help your codebase in the long run. Why squander the opportunity to
choose something unfamiliar when you have the rare opportunity to start working
on a new project?</p>
<p>I&rsquo;m not advocating for you to use something new for every project, though. I&rsquo;m
suggesting that you detatch your familiarity with your tools from the
decision-making process. I often reach for old tools when starting a new
project, but I have learned enough about new tools that I can judge what
projects are a good use-case for them. Sometimes this doesn&rsquo;t work out, too - I
just threw away and rewrote a prototype in C after deciding that it wasn&rsquo;t a
good candidate for Rust.</p>
<p>Often it does work out, though. I&rsquo;m glad I chose to learn Python for MediaCrush
despite having no experience with it (thanks again for the help with that,
Jose!). Today I still know it was the correct choice and knowing it has hugely
expanded my programming skills, and without that choice there probably wouldn&rsquo;t
have been a Kerbal Stuff or a sr.ht or likely even the new API we&rsquo;re working on
at Linode. I&rsquo;m glad I chose to learn C for z80e, though I had previously written
emulators in C#. Without it there wouldn&rsquo;t be many other great tools in the
KnightOS ecosystem written in C, and there wouldn&rsquo;t be a Sway or an aerc. I&rsquo;m
glad I learned ES6 and React instead of falling back on the familiar Knockout.js
when building prototypes for the new Linode manager as well.</p>
<p>Today, I have a mental model of the benefits and drawbacks of a lot of
languages, frameworks, libraries, and platforms I don&rsquo;t know how to use. I&rsquo;m
sort of waiting for projects that would be well suited to things like Rust or
Django or Lisp or even Plan 9. Remember, the skills you already know make for a
great hammer, but you shouldn&rsquo;t nail screws to the wall.</p>


guid: https://drewdevault.com/2016/11/05/Losing-faith-in-America.html
title: I'm losing faith in America
url: https://drewdevault.com/2016/11/05/Losing-faith-in-America.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I recently quit my job at Linode and started looking for something else to do.
For the first time in my career, I&rsquo;m seriously considering opportunities abroad.
Sorry for the politically charged post - I promise to get back to tech stuff
right away.</p>
<p><img src="https://imgs.xkcd.com/comics/canada.png" alt=""></p>
<p>On November 8th, I&rsquo;m going to step into the voting booth and will be presented
with the following options:</p>
<ul>
<li>A criminal who cheated her way into a spot on the ballot</li>
<li>An egotistical racist maniac</li>
</ul>
<p>The next president of the United States will probably be Hillary Clinton. I&rsquo;m
sure I don&rsquo;t have to tell you how ridiculous this is. This is a person who has
pulled all of the stops to get her name on the ballot, including <em>voter fraud</em>
and disturbing amounts of corruption within the Democratic party.  Not to
mention that she&rsquo;s probably going to start a war with Syria, mess with the
already fragile geopolitical relationship we have with Russia, and likely
deserves to be incarcerated for mishandling classified information. Say what you
will about the Republican party - at least Trump won his nomination fair and
square. Bonus: not voting for Hillary is sexist.</p>
<p>Not that I&rsquo;d prefer it if Trump wins. I have a free sandwich waiting for me at
the deli nearby if he doesn&rsquo;t win. He got his nomination fairly, but that
doesn&rsquo;t mean he deserves it. This is a guy with little political clout who is
incapable of handling international relations or commanding our military. He
staunchly advocates committing war crimes to deal with ISIS. He makes racist,
sweeping generalizations about anyone different from him. He&rsquo;s a misogynist.
Even worse, he&rsquo;s all of these things and seems to actually represent a fair
portion of his supporters.</p>
<p>Neither of the independents are serious contenders, so I won&rsquo;t bother with why I
don&rsquo;t like them. They haven&rsquo;t earned my vote, either.</p>
<p>Congress is composed of many of the same sort of people. Corrupt politicians who
answer to the checkbooks of lobbyists who work against the interests of the
American people for the sake of their own. We&rsquo;re facing climate change and our
politicians are taking money from rich fossil fuel lobbyists and damning our
species to extinction. The wealth gap between the rich and the poor grows deeper
and deeper as absurdly rich people get absurdly richer at the expense of the
poor and middle class - through the support of the politicians whose pockets
they&rsquo;ve greased. Their excess wealth could pay for programs to improve our
failing infrastructure and provide hundreds of thousands of jobs in doing so. We
could provide free healthcare for all Americans too, if it wasn&rsquo;t for the
ongoing debate about whether or not being alive and healthy is a fundamental
human right - many thanks to the pharmaceutical interests for shaping this
debate to maximize their profits. It&rsquo;d be less of a problem if many companies
weren&rsquo;t getting rich off of the ever widening waistlines of Americans, too.</p>
<p>Mass surveillance remains in full effect even years after Snowden&rsquo;s revelations.
The ridiculous war on drugs keeps putting people behind bars for lifetimes for
victimless crimes to support the financial needs of private prisons and local
police departments, who themselves are now better armed than most militaries,
based on drug policies that have no basis in reality. 97% of trails end in plea
bargains instead of justice, and minimum sentences ensure these people spend
ridiculous amounts of time in prisons that punish them rather than rehabilitate
them into productive citizens. A judge will hold a defendant indefinitely in
prison without a conviction for refusing to disclose their disk encryption
password in accordance with their 5th amendment rights - though if many
political players had their way, encryption would be illegal anyway.</p>
<p>There&rsquo;s a word for what America is: <strong>corrupt</strong>. What the fuck is going on in
this country? We aren&rsquo;t a representative democracy by any stretch of the
imagination. We have become an oligarchy. We are ruled by money.</p>
<p>I love America, honestly. My whole family is here and I connect most with the
American people. We have an incredibly rich land and great cities full of great
innovators and interesting people. I hate that it&rsquo;s become what it is today. I
don&rsquo;t expect anywhere else to be perfect, but we should be ashamed of how we
look next to some other countries out there.</p>


guid: https://drewdevault.com/2016/11/16/Getting-on-without-Google.html
title: Getting on without Google
url: https://drewdevault.com/2016/11/16/Getting-on-without-Google.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p><img src="https://sr.ht/d718.png" alt=""></p>
<p>I used Google for a long time, but have waned myself off of it over the past
few years, and I finally deleted my account a little over a month ago. I feel so
much better about my privacy now that I&rsquo;ve removed Google from the equation, and
self hosting my things affords me a lot of flexibility and useful customizations.</p>
<h2 id="mailcmpwncom">mail.cmpwn.com</h2>
<p>This one was the most difficult and time consuming to set up, but it was <em>very</em>
worth it. I&rsquo;ve intended for a while to make a new mail server software suite
that&rsquo;s less terrible to set up, so hopefully that situation will improve in the
future. I want to flesh out <a href="https://github.com/SirCmpwn/aerc">aerc</a> some more
first. A personal mail server was one of the earliest things I set up in my
post-Google life - I&rsquo;ve operated it for about two years now.</p>
<ul>
<li>Postfix to handle incoming and outgoing mail</li>
<li>Dovecot to handle mail delivery, filtering, and IMAP</li>
<li>Postfixadmin to provide a nice interface for managing accounts</li>
<li>mutt to read and compose my emails on the desktop</li>
<li>K9 to read and compose my emails on Android</li>
<li>Roundcube for when it&rsquo;s occasionally necessary to read an HTML email</li>
</ul>
<p>With my mail server provides a lot of side benefits, too. For one, all of my
email-sending software now uses it. Once Mandrill went kaput, it was easy to
switch everything over to it. I can be sending and receiving email from a new
domain in less than 5 minutes now. Using sieve scripts for filtering emails is
also a lot more flexible than what Google offered - I now have filtering set up
to organize several mailing lists, alerts and notifications sent by my software
and servers, RSS feeds, and more.</p>
<p>My strategy for defeating spam is to use a combination of the spamhaus
blocklist, greylisting, and blacklisting with sieve. I see about 3-5 spam emails
per week on average with this setup. To ensure my own emails get delivered, I&rsquo;ve
set up SPF and DKIM, reverse DNS, and appealed to have my IP address removed
from blocklists. A great tool in figuring all this out has been
<a href="http://mail-tester.com">mail-tester.com</a>.</p>
<h2 id="youtube">YouTube</h2>
<p>For YouTube, I &ldquo;subscribe&rdquo; to channels by adding their RSS feeds to
<a href="http://www.allthingsrss.com/rss2email/">rss2email</a>, combined with sieve scripts
that filter them into a specific folder. I then have a keybinding in mutt that,
when pressed, pulls the YouTube URL out of an email and feeds it to mpv, a
desktop video player. It&rsquo;s so much easier to access YouTube this way than
through the web browser - no ads, familiar keybindings, remote control support,
and a no-nonsense feed of your videos.</p>
<h2 id="music">Music</h2>
<p>Instead of Google Music, Spotify, or anything else, I run an internet radio
with my friends. We all keep our music collections (mostly lossless) on NFS
servers, and we mounted these servers on a streaming server that shuffles the
entire thing and keeps a searchable database of music. We have an API that I
pull from to integrate desktop keybindings and a status line on my taskbar, and
an IRC bot for searching the database and requesting songs. I can also stream to
my phone with VLC, as well as use scripts to maintain an offline archive of my
favorite songs. This setup is <em>way</em> nicer than any commercial service I&rsquo;ve used
in the past. We&rsquo;ll be open sourcing version 2 to provide a turnkey solution for
this type of self-hosted music service.</p>
<h2 id="web-search">Web search</h2>
<p><a href="https://duckduckgo.com/">DuckDuckGo</a>. Even if you think the search results
aren&rsquo;t up to snuff (you get used to just being a bit more specific anyway), the
bangs feature is absolutely indispensable. I recently patched Chromium for
Android to support DuckDuckGo as a search engine as well:
<a href="https://sr.ht/h4bZ.patch">here&rsquo;s the patch</a>.</p>
<h2 id="file-hosting">File hosting</h2>
<p>Instead of using Google Drive, I&rsquo;m using a number of different solutions
depending on what&rsquo;s most convenient at the time. I operate
<a href="https://sr.ht">sr.ht</a> for me and my friends, which allows me to just have a
place to drop a file and get a link to share. I have scripts and keybindings set
up to make uploading files here second nature, as well as an Android app someone
wrote. I also keep a 128G flash drive on my keychain now that comes in handy all
the time, and a big-ass file server on OVH that I keep mounted with NFS or sshfs
depending on the scenario, and sometimes I just stash files on a random server
with rsync. sr.ht is <a href="https://gogs.sr.ht/SirCmpwn/sr.ht">open source</a>, by the
way.</p>
<h2 id="cyanogenmod">CyanogenMod</h2>
<p>On Android, I use CyanogenMod without Google Play Services, and I use F-Droid to
get apps. When I used Google Now, I found that I most often just asked it for
reminders, which I now do via an open source app called Notable Plus. I also
have open source apps for reading HN, downloading torrents, blocking ads,
connecting to IRC, two factor authentication, YouTube, password management,
Twitter, and more.</p>
<h2 id="notably-missing-docs">Notably missing: Docs</h2>
<p>Hopefully the new LibreOffice thing will do the trick once it&rsquo;s ready. I&rsquo;m
looking forward to that.</p>
<h2 id="things-i-self-host-that-google-doesnt-offer">Things I self host that Google doesn&rsquo;t offer</h2>
<p>I use ZNC to operate an IRC bouncer, which is great because I use IRC <em>a lot</em>.
It keeps logs for me, keeps me always connected, and gives me a number of nice
features to work with. I also host a number of simple websites related to IRC to
do things like channel stats and rules.</p>
<p>To all sr.ht users I offer access to <a href="https://gogs.sr.ht">gogs.sr.ht</a>, which I
personally use to host many private repositories as well as a number of small
projects, and as a kind of staging area for repositories that aren&rsquo;t quite ready
for GitHub yet.</p>
<p>For passwords, I use a tool called <a href="https://www.passwordstore.org/">pass</a>, which
encrypts passwords with my PGP key and stores them in a git repository I keep on
gogs.sr.ht, with desktop keybindings to make grabbing them convenient.</p>
<h2 id="help-me-do-this">Help me do this!</h2>
<p>Well, that covers most of my major self hosted services. If you&rsquo;re interested in
more detail about how any of this works so you might set something up yourself,
feel free to reach out to me by <a href="mailto:sir@cmpwn.com">email</a>,
<a href="https://cmpwn.com/@sir">Mastodon</a>, or IRC (SirCmpwn on any network). I&rsquo;d
be happy to help!</p>


guid: https://drewdevault.com/2016/11/24/Electron-considered-harmful.html
title: Electron considered harmful
url: https://drewdevault.com/2016/11/24/Electron-considered-harmful.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Yeah, I know that &ldquo;considered harmful&rdquo; essays are allegedly <a href="http://meyerweb.com/eric/comment/chech.html">considered
harmful</a>. If it surprises you that
I&rsquo;m writing one, though, you must be a new reader. Welcome! Let&rsquo;s get started.
If you&rsquo;re unfamiliar with Electron, it&rsquo;s some hot new tech that lets you make
desktop applications with HTML+CSS+JavaScript. It&rsquo;s basically a chromeless web
browser with a Node.js backend and a Chromium-based frontend. What follows is
the rant of a pissed off Unix hacker, you&rsquo;ve been warned.</p>
<p>As software engineers we have a responsibility to pick the <em>right</em> tools for the
job. In fact, that&rsquo;s the <em>most important</em> choice we have to make when we start a
project. When you choose Electron you get:</p>
<ul>
<li>An entire copy of Chromium you&rsquo;ll be shipping with your app</li>
<li>An interface that looks and feels nothing like the rest of the user&rsquo;s OS</li>
<li>One of the slowest, least memory efficient, and most inelegant GUI application
platforms out there (remember, we <em>tolerate</em> frontend web development because
we have no choice, not because it is by any means <em>good</em>)</li>
</ul>
<p>Let&rsquo;s go over some case studies.</p>
<p><strong><a href="https://github.com/mifi/lossless-cut">lossless-cut</a></strong> is an Electron app that
gives you a graphical UI for <em>two ffmpeg flags</em>. Seriously, the flags in
question are -ss and -t. No really, that&rsquo;s <em><a href="https://github.com/mifi/lossless-cut/blob/master/src/ffmpeg.js#L46">literally all it
does</a></em>. It
doesn&rsquo;t even use ffmpeg to decode the video preview in the app, it&rsquo;s limited to
the codecs chromium supports. It also ships its own ffmpeg, so it has the
industry standard video decoding tool <em>right there</em> and doesn&rsquo;t use it to render
video. For the price of 200 extra MiB of disk space and an entire Chromium process
in RAM and on your CPU, you get a less capable GUI that saves you from having to
type the -ss and -t flags yourself.</p>
<p><strong><a href="http://1clipboard.io/">1Clipboard</a></strong> is a clipboard manager. In Electron. A
<em>clipboard manager</em>. In order to show you <em>a list of things you&rsquo;ve copied</em>, it
uses <em>an entire bundled copy of Chromium</em>. Also note that despite the promises
of Electron making cross platform development easy, it doesn&rsquo;t support Linux.</p>
<p><strong><a href="https://getcollectie.com/">Collectie</a></strong> is a&hellip; fancy bookmark manager, I
guess? Another one that fails to get the cross platform value add from Electron,
this only supports OS X (or is it macOS). For only $10 bucks you get to organize
your shit into folders. Or you could just open the Finder for free and get a
native UX to boot.</p>
<p>This is a <a href="https://hyper.is/">terminal</a> written with Electron. On the landing
page they say &ldquo;# A terminal emulator 100% based on JavaScript, HTML, and CSS&rdquo;
like they&rsquo;re proud of it. They&rsquo;ve taken one of the most lightweight and
essential tools on your computer and bloated it by orders of magnitude. Why the
fuck would you want to render Google in your god damn terminal emulator? Bonus:
also not cross platform.</p>
<p>This is not to mention the dozens of companies that have taken their websites
and crammed them into a shitty electron app and called it their desktop app.
Come on guys!</p>
<p>By the way, if you&rsquo;re the guy who&rsquo;s going to leave a comment about how this blog
post introduced you to a bunch of interesting apps you&rsquo;re going to install now,
I hate you.</p>
<h2 id="electron-enables-lazy-developers-to-write-garbage">Electron enables lazy developers to write garbage</h2>
<p>Let me be clear about this: JavaScript sucks. It&rsquo;s not the worst, but it&rsquo;s also
not by any means good. ES6 is a really great step forward and I&rsquo;m thrilled about
how much easier it&rsquo;s going to be to write JavaScript, but it&rsquo;s still JavaScript
underneath the syntactic sugar. We use it because we have no choice (people who
know more than just JavaScript know this). The object model is whack and the
loose typing is whack and the DOM is super whack.</p>
<p>When Node.js happened, a bunch of developers who never bothered to learn more
than JavaScript for their frontend work suddenly could write their crappy code
on the backend, too. Now this is happening to desktop applications. The reason
people choose Electron is because they are <em>too lazy</em> to learn the right tools
for the job. This is the <em>worst</em> quality a developer can have. You&rsquo;re an
engineer, for the love of God! Fucking act like one! Do they build square
airplanes so they don&rsquo;t have to learn about aerodynamics, then just throw on an
extra ten engines to make up for it? NO!</p>
<p>For the love of God, learn something else. Learn how to use GTK or Qt. Maybe Xwt
is more up your alley. How about GNOME&rsquo;s Vala thing? <em>Learn another programming
language</em>. Learn Python or C/C++ or C#. Fun fact: it&rsquo;ll make your JavaScript
better, and once you have it in your toolbox you can make more educated
decisions on the appropriate tool to use when you face your next problem. Hint:
it&rsquo;s not Electron.</p>
<h2 id="some-electron-apps-dont-suck">Some Electron apps don&rsquo;t suck</h2>
<p>For some use-cases Electron is a reasonable choice.</p>
<ul>
<li><a href="https://code.visualstudio.com/">Visual Studio Code</a>, because it&rsquo;s a full
blown IDE with a debugger and plugins and more. It&rsquo;s already gonna be
bloated.</li>
<li><a href="http://www.soundnodeapp.com/">Soundnode</a>, because it&rsquo;s not like any other
music service&rsquo;s app obeys your OS&rsquo;s UI conventions</li>
</ul>
<p>Uh, that&rsquo;s it. That&rsquo;s the entire list.</p>


guid: https://drewdevault.com/2016/12/06/A-broad-intro-to-networking.html
title: A broad intro to networking
url: https://drewdevault.com/2016/12/06/A-broad-intro-to-networking.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Disclaimer: I am not a network engineer. That&rsquo;s the point of this blog post,
though - I want to share with non-networking people enough information about
networking to get by. Hopefully by the end of this post you&rsquo;ll know enough about
networking to keep up with a conversation on networking, or know what to search
for when something breaks, or know what tech to research more in-depth when you
are putting together something new.</p>
<h2 id="layers">Layers</h2>
<p>The <strong>OSI model</strong> is the standard model we describe networks with. There are 7
<strong>layers</strong>:</p>
<p>Layer 1, the physical layer, is the electrical engineering stuff.</p>
<p>Layer 2, the link layer, is how devices talk to each other.</p>
<p>Layer 3, the network layer, is what they talk about.</p>
<p>Layer 4, the transport layer, is where things like TCP and UDP live.</p>
<p>Layers 5 and 6 aren&rsquo;t very important.</p>
<p>Layer 7, the application layer, is where Minecraft lives.</p>
<p>When you hear some security guy talking about a &ldquo;layer 7 attack&rdquo;, he&rsquo;s
talking about a attack that focuses on flaws in the application layer. In
practice that means i.e. flooding the server with HTTP requests.</p>
<h2 id="1-physical-layer">1: Physical Layer</h2>
<p><em>Generally implemented by matter</em></p>
<p>Layer 1 is the hardware of a network. Commonly you&rsquo;ll find things here like your
computer&rsquo;s <strong>NIC</strong> (network interface controller), aka the network interface or
just the interface, which is the bit of silicon in your PC that you plug network
cables or WiFi signals into.</p>
<p>On Linux, network interfaces are assigned names like <em>eth0</em> or <em>eno1</em>. eth0 is
the traditional name for the 0th wired network interface. eno1 is the newer
&ldquo;consistent network device naming&rdquo; format popularized by tools like udev (which
manages hardware on many Linux systems) - this is a deterministic name based on
your network hardware, and won&rsquo;t change if you add more interfaces. You can
manage your interfaces with the <em>ip</em> command (<code>man 8 ip</code>), or the now-deprecated
<em>ifconfig</em> command. Some non-Linux Unix systems have not deprecated ifconfig.</p>
<p>This layer also has ownership over <strong>MAC addresses</strong>, in theory. A MAC address
is an allegedly unique identifier for a network device. In practice, software
at higher layers can use whatever MAC address they want. You can change your MAC
address with the ip command, which is often useful for dealing with annoying
public WiFi resource limits or for frustrating someone else on the network.</p>
<p>Other things you find at layer 1 include <strong>switches</strong>, which do network
multiplexing (they generally can be thought of as networking&rsquo;s version of a
power strip - they turn one Ethernet port into many). Also common are
<strong>routers</strong>, whose behaviors are better explained in other layers. You also have
hardware like <strong>firewalls</strong>, which filter network traffic, and <strong>load
balancers</strong>, which distribute a load among several nodes. Both firewalls and
load balancers can be done in software, depending on your needs.</p>
<h2 id="2-data-link-layer">2: Data link layer</h2>
<p><em>Generally implemented by network hardware</em></p>
<p>At this layer you have protocols that cover how nodes talk to one another. Here
the <strong>ethernet</strong> protocol is almost certainly the most common - the protocol
that goes over your network cables. Said network cables are probably <strong>Cat 5</strong>
cables, or &ldquo;category 5&rdquo; cables.</p>
<p>Other protocols here include tunnels, which allow you to indirectly access a
network. A common example is a <strong>VPN</strong>, or virtual private network, which allows
you to participate in another network remotely. Tunnels can also be useful for
getting around firewalls, or for setting up a secure means to access resources
on another network.</p>
<h2 id="3-network-layer">3: Network layer</h2>
<p><em>Generally implemented by the kernel</em></p>
<p>As a software guy, this is where the fun really starts. The other layers are how
computers talk to each other - this layer is what they talk about. Computers are
often connected via a <strong>LAN</strong>, or local area network - a <em>local</em> network of
computers. Computers are also often connected to a <strong>WAN</strong>, or wide area
network - the internet is one such network.</p>
<p>The most common protocol at this layer is IP, or Internet Protocol. There are
two versions that matter: IPv4, and IPv6. Both of them use <strong>IP addresses</strong> to
identify nodes on their networks, and they carry <strong>packets</strong> between them. The
major difference between IPv4 and IPv6 is the size of their respective <strong>address
spaces</strong>. IPv4 uses 32 bit addresses, supporting a total of 4.3 billion possible
addresses, which on the public internet are quickly becoming a sparse resource.
IPv6 uses 128-bit addresses, which allows for a zillion unique addresses.</p>
<p>Ranges of IP addresses can be described with a <strong>subnet mask</strong>. Such a range of
IP addresses constitutes a <strong>subnetwork</strong>, or subnet. Though you&rsquo;re probably
used to seeing an IPv4 address encoded like <code>10.20.30.40</code>, remember that it can
also just be represented as one 32-bit number - in this case 169090600, or
0xA141E28, and you can do bitwise math against these numbers. You generally
represent a subnet with CIDR notation, such as <code>192.168.1.0/24</code>. In this case, the
first 24 bits are meaningful, and all possible values for the remaining 8 bits
constitute the range of addresses represented by this mask.</p>
<p>IPv4 has several subnets reserved for this and that. Some important ones are:</p>
<ul>
<li><code>0.0.0.0/8</code> - current network. On many systems, you can treat <code>0.0.0.0</code> as all
IP addresses assigned to your device</li>
<li><code>127.0.0.0/8</code> - loopback network. These addresses refer to yourself.</li>
<li><code>10.0.0.0/8</code>, <code>172.16.0.0/12</code>, and <code>192.168.0.0/16</code> are reserved for private
networks - you can allocate these addresses on a LAN.</li>
</ul>
<p>An IPv4 packet includes, among other things: a <strong>time to live</strong>, or TTL, which
limits how long the packet can live for; the <strong>protocol</strong>, such as TCP; the
<strong>source</strong> and <strong>destination</strong> addresses; a header checksum; and the
<strong>payload</strong>, which is specific to the higher level protocol in use.</p>
<p>Given the limited size of the IPv4 space, most networks are designed with an
isolated LAN that uses <strong>NAT</strong>, or network address translation, to translate IP
addresses from the WAN. Basically, a router or similar component will translate
internal IP addresses (allocated from the private subnets) to its own external
IP address, and vice versa, when passing communications along to the WAN. With
IPv6 there are so many IP addresses that you don&rsquo;t need to use NAT. If you&rsquo;re
wondering whether or not we&rsquo;ll ever run out of IPv6 addresses - leave that to
someone else to solve tens of millions of years from now.</p>
<p>IPv6 addresses are 128-bits long and are described with strings like
<code>2001:0db8:0000:0000:0000:ff00:0042:8329</code>. Luckily the people who designed it
were kind enough to realize people don&rsquo;t want to write that, so it can be
shortened to <code>2001:db8::ff00:42:8329</code> by removing leading zeros and removing
sections entirely composed of zeros. Where colons are reserved for another
purpose, you&rsquo;ll typically add brackets around the IPv6 address, such as
<code>http://[2607:f8b0:400d:c03::64]</code>. The IPv6 loopback address (localhost) is
<code>::1</code>, and IPv6 subnets are written the same way as in IPv4. Given how many
IPv6 addresses there are, it&rsquo;s common to be allocated lots of them in cases when
you might have expected to only receive one IPv4 address. Typically these blocks
will be anywhere from /48 to /56 - which contains more addresses than the entire
IPv4 space.</p>
<p>IP addresses are often <strong>static</strong>, which means the node connecting to the
network already knows its IP address and starts using it right away. They may
also be <strong>dynamic</strong>, and are allocated by some computer on the network with the
<strong>DHCP</strong> protocol.</p>
<p>IPsec also lives in layer 3.</p>
<h2 id="4-transport-layer">4: Transport Layer</h2>
<p><em>Generally implemented by the kernel</em></p>
<p>The transport layer is where you have higher level protocols, through which much
of the work gets done. Protocols here include TCP, UDP, ICMP (used for ping),
and others. These protocols are used to power application-layer protocols.</p>
<p><strong>TCP</strong>, or the transmission control protocol, is probably the most popular
transport layer protocol out there. It turns the unreliable internet protocol
into a reliable byte stream. TCP (tries to) make four major guarantees: data
will arrive, will arrive exactly once, will arrive in the correct order, and
will be the correct data.</p>
<p>TCP takes a stream of bytes and breaks it up into <strong>segments</strong>. Each segment is
then stuck into an IP packet and sent on its way. A TCP segment includes the
source and destination <strong>ports</strong>, which are used to distinguish between
different application-layer protocols in use and to distinguish between
different applications using the protocol on the same host; a <strong>sequence
number</strong>, which is used to order the packet; an <strong>ACK number</strong>, which is used to
inform the other end that it has received some packet and it can stop retrying;
a checksum; and the data itself. The protocol also includes a handshake process
and other housekeeping processes that the application needn&rsquo;t be aware of.
Generally speaking, the overhead of TCP is significant for real-time
applications.</p>
<p>Most TCP servers will <strong>bind</strong> to a certain port to <strong>listen</strong> for incoming
connections, via the operating system&rsquo;s <strong>socket</strong> implementation. Many TCP
<strong>clients</strong> can connect to one server.</p>
<p>Ports are a 16 bit unsigned integer. Most applications have a default port
they&rsquo;re known to use, such as 80 for HTTP. Originally these numbers were
allocated by the internet police, but this has fallen out of practice. On most
systems, ports less than 1024 require elevated permissions to listen to.</p>
<p><strong>UDP</strong>, or the user datagram protocol, is the second most popular transport
layer protocol, and is the lighter of the two. UDP is a paper thin layer on top
of IP. A UDP packet contains a source port, destination port, checksum, and a
payload. This protocol is fast and lightweight, but makes none of the promises
TCP makes - UDP &ldquo;<strong>datagrams</strong>&rdquo; may arrive multiple or zero times, in a
different order than they were sent, and possibly with data errors. Many people
who use UDP will implement these guarantees themselves in a some lighter-weight
fashion than TCP. Importantly, UDP source IPs can be spoofed and the destination
has no means of knowing where it really came from - TCP avoids this by doing a
handshake before exchanging any data.</p>
<p>UDP can also issue broadcasts, which are datagrams that are sent to every node
on the network. Such datagrams should be addressed to <code>255.255.255.255</code>. There&rsquo;s
also multicast, which specifies a subset of all nodes to send the datagram to.
Note that both of these have limited support in real-world networks.</p>
<h2 id="5--6-session-and-presentation">5 &amp; 6: Session and presentation</h2>
<p>Think of these as extensions of layer 7, the application layer. Technically
things like SSL, compression, etc are done here, but in practice it doesn&rsquo;t
have any important technical implications.</p>
<h2 id="7-application-layer">7: Application layer</h2>
<p><em>Generally implemented by end-user software</em></p>
<p>The application layer is the uppermost layer of the network and it&rsquo;s what all
the other layers are there for. At this layer you have all of the hundreds of
thousands of application-specific protocols out there.</p>
<p><strong>DNS</strong>, or the domain name system, is a protocol for mapping domain names (i.e.
google.com) to IP addresses (i.e. 209.85.201.100), among other features. DNS
servers keep track of DNS records, which associate names with records of various
types. Common records include A, which maps a name to an IPv4 address, AAAA for
IPv6, CNAME for aliases, and MX for email records. The most popular DNS server
is bind, which you can run on your own network to operate a private name system.</p>
<p>Some other UDP protocols: NTP, the network time protocol; DHCP, which assigns
dynamic IP addresses on networks; and nearly all real-time video and audio
streaming protocols (like VoIP). Many video games also use UDP for their
multiplayer networking.</p>
<p>TCP is more popular than UDP and powers many, many, many applications, due
largely to the fact that it simplifies the complex intricacies of networking.
You&rsquo;re probably familiar with HTTP, which is used by web browsers use to fetch
resources. Email applications often communicate over TCP with IMAP to retrieve
the contents of your inbox, and SMTP to send emails to other servers. SSH (the
secure shell), FTP (file transfer protocol), IRC (internet relay chat), and
countless other protocols also use TCP.</p>
<hr>
<p>Hopefully this article helps you gain a general understanding of how computers
talk to each other. In my own experience, I&rsquo;ve used a broad understanding of the
entire stack and a deep understanding of levels 3 and up. I expect most
programmers today need a broad understanding of the entire stack and a deep
understanding of level 7, and I hope that most programmers would seek a deep
understanding of level 4 as well.</p>
<p>Please leave some feedback if you appreciated this article - I may do more
similar articles in the future, giving a broad introduction to other topics. The
next topics I have in mind are security and encryption (as separate posts).</p>


guid: https://drewdevault.com/2016/12/27/State-of-sway.html
title: State of Sway December 2016 - secure your Wayland desktop, get paid to work on Sway
url: https://drewdevault.com/2016/12/27/State-of-sway.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Earlier today I released <a href="https://github.com/SirCmpwn/sway/releases/tag/0.11">sway
0.11</a>, which (along with
lots of the usual new features and bug fixes) introduces support for security
policies that can help realize the promise of a secure Wayland desktop. We also
just started a bounty program that lets you sponsor the things you want done and
rewards contributors for working on them.</p>
<p>Today sway has 19,371 lines of C (and 3,761 lines of header files) written by 70
authors across 2,067 commits. These were written through 589 pull requests and
425 issues. Sway packages are available today in the official repos of Arch,
Gentoo, Fedora, NixOS, openSUSE, Void Linux, and more. Sway looks like this:</p>
<p><a href="https://sr.ht/ICd5.png"><img src="https://sr.ht/ICd5.png" alt=""></a></p>
<p>Side note: please add pretty screenshots of sway to <a href="https://github.com/SirCmpwn/sway/wiki/Screenshots-of-Sway">this wiki
page</a>. Thanks!</p>
<p>For those who are new to the project, <a href="http://swaywm.org">Sway</a> is an
i3-compatible Wayland compositor. That is, your existing <a href="http://i3wm.org/">i3</a>
configuration file will work as-is on Sway, and your keybindings and colors and
fonts and for_window rules and so on will all be the same. It&rsquo;s i3, but for
Wayland, plus it&rsquo;s got some bonus features. Here&rsquo;s a quick rundown of what&rsquo;s
new since the <a href="https://drewdevault.com/2016/08/02/Sway-0.9-in-retro.html">previous state of Sway</a>:</p>
<ul>
<li>Security policy configuration (man sway-security)</li>
<li>FreeBSD support</li>
<li>Initial support for HiDPI among sway clients (swaybar et al)</li>
<li>Support for new i3 features</li>
<li>Clicky title bars</li>
<li>Lots of i3 compatability improvements</li>
<li>Lots of documentation improvements</li>
<li>Lots of bugfixes</li>
</ul>
<p>Today it seems that most of the features sway needs are implemented. Work hasn&rsquo;t
slowed down - there&rsquo;s been lots of work fixing small bugs, improving
documentation, fixing subtle incompatabilities with i3, and so on. However, to
encourage the development of new features, I&rsquo;ve officially put into action the
new bounty program today. Here&rsquo;s how it works - you can donate to the features
you want to see, and you can claim the donations by implementing the features
and sending a pull request. To date I&rsquo;ve received about $200 in donations
towards sway, and I&rsquo;ve matched that with a donation of my own to bring it up to
$400. I&rsquo;ve distributed these donations into various buckets of features. Not
every feature is for sway - anything that improves the sway experience is
eligible for a bounty, and in fact over half of the initial bounties are for
features in other parts of the ecosystem. For details on the program, check out
<a href="https://github.com/SirCmpwn/sway/issues/986">this link</a>.</p>
<p>Here&rsquo;s the updated stats. First, <strong>lines of code per author</strong>:</p>
<table class="table">
    <tbody>
        <tr><td>3799 (+775)</td><td>Drew DeVault</td></tr>
        <tr><td>3489 (-1170)</td><td>Mikkel Oscar Lyderik</td></tr>
        <tr><td>1705 (-527)</td><td>taiyu</td></tr>
        <tr><td>1236 (-550)</td><td>S. Christoffer Eliesen</td></tr>
        <tr><td>1160 (+70)</td><td>Zandr Martin</td></tr>
        <tr><td>449 (-12)</td><td>minus</td></tr>
        <tr><td>311 (-54)</td><td>Christoph Gysin</td></tr>
        <tr><td>285 (+285)</td><td>D.B</td></tr>
        <tr><td>247 (-87)</td><td>Kevin Hamacher</td></tr>
        <tr><td>227 (-298)</td><td>Cole Mickens</td></tr>
        <tr><td>219 (+219)</td><td>David Eklov</td></tr>
    </tbody>
</table>
<p>Finally, I&rsquo;m the top contributor! I haven&rsquo;t been on top for over a year. Lots of
the top contributors are slowly having their lines of code reduced as lots of
new contributors are coming in and displacing them with refactorings and bug
fixes.</p>
<p>Here&rsquo;s the total <strong>number of commits per author</strong> for each of the top ten
committers:</p>
<table class="table">
    <tbody>
        <tr><td>1009</td><td> Drew DeVault</td></tr>
        <tr><td>245</td><td> Mikkel Oscar Lyderik</td></tr>
        <tr><td>153</td><td> taiyu</td></tr>
        <tr><td>97</td><td> Luminarys</td></tr>
        <tr><td>91</td><td> S. Christoffer Eliesen</td></tr>
        <tr><td>68</td><td> Zandr Martin</td></tr>
        <tr><td>58</td><td> Christoph Gysin</td></tr>
        <tr><td>45</td><td> D.B</td></tr>
        <tr><td>33</td><td> Taiyu</td></tr>
        <tr><td>32</td><td> minus</td></tr>
    </tbody>
</table>
<p>Most of what I do for Sway personally is reviewing and merging pull requests.
Here&rsquo;s the same figures using <strong>number of commits per author, excluding merge
commits</strong>, which changes my stats considerably:</p>
<table class="table">
    <tbody>
        <tr><td>479</td><td> Drew DeVault</td></tr>
        <tr><td>229</td><td> Mikkel Oscar Lyderik</td></tr>
        <tr><td>138</td><td> taiyu</td></tr>
        <tr><td>96</td><td> Luminarys</td></tr>
        <tr><td>91</td><td> S. Christoffer Eliesen</td></tr>
        <tr><td>58</td><td> Christoph Gysin</td></tr>
        <tr><td>56</td><td> Zandr Martin</td></tr>
        <tr><td>45</td><td> D.B</td></tr>
        <tr><td>32</td><td> Taiyu</td></tr>
        <tr><td>32</td><td> minus</td></tr>
    </tbody>
</table>
<p>These stats only cover the top ten in each, but there are more - check out the
<a href="https://github.com/SirCmpwn/sway/graphs/contributors">full list</a>.</p>
<p>Here&rsquo;s looking forward to sway 1.0 in 2017!</p>


guid: https://drewdevault.com/2017/01/06/Actually-you-CAN-do-it.html
title: Actually, you CAN do it
url: https://drewdevault.com/2017/01/06/Actually-you-CAN-do-it.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I maintain a <em>lot</em> of open source projects. In order to do so, I have to
effectively manage my time. Most of my projects follow this philosophy: if you
want something changed, send a patch. If you are running into an annoying bug,
fix it and send a patch. If you want a new feature, implement it and send a
patch. It&rsquo;s definitely a good idea to talk about it beforehand on the issue
tracker or IRC, but don&rsquo;t make the mistake of thinking this processes ends with
someone else doing it for you.</p>
<p>Every developer who contributes to a project I maintain is self-directed. They
work on what they&rsquo;d like. They scratch their own itches. Sometimes what they&rsquo;d
like to work on is non-specific, and in that case I&rsquo;ll help them find something
to do based on what users are asking for lately or based on my own goals for the
project. I often maintain a list of &ldquo;low hanging fruit&rdquo; issues on Github, and
I am generally willing to offer some suggestions if someone asks for such a
task. However, for more complex, non-&ldquo;low hanging fruit&rdquo; tasks, they generally
only get worked on when someone with the know-how wants it done and does it.</p>
<p>So what does this mean for you, user whose problem no developer is interested
in? Well, it&rsquo;s time for you to step up and work on it yourself. I don&rsquo;t really
care if your problem is &ldquo;a showstopper&rdquo; or &ldquo;the only thing preventing you from
switching to my software&rdquo;, or any of a number of other excuses you may have
lined up for getting someone else to do it for you. None of the other regular
contributors really care about your interpretation of what their priorities
should be, either. We aren&rsquo;t a business. We aren&rsquo;t making a sale. We&rsquo;re just
making cool software that works for us and publishing it in the hopes that
you&rsquo;ll find it useful, too.</p>
<p>Generally by this point in the conversation with Joe User, they tell me they
<em>can&rsquo;t</em> do it. Well, Joe User, I beg to differ. It doesn&rsquo;t matter that you don&rsquo;t
know <em>[insert programming language]</em>, or haven&rsquo;t used <em>[insert relevant
library]</em> before. You don&rsquo;t learn new things by hanging out in your comfort
zone. Many of the regulars you&rsquo;re bugging to do your work for you were once in
your shoes.</p>
<p>Everything is setting you up for success. You literally have hundreds of
resources at your disposal. The internet is was made by developers, you know,
and we built tons of resources to support ourselves with it. You have
documentation, Q&amp;A sites, chat rooms, and more waiting to help you when you get
stuck. We&rsquo;re here to answer your questions with the codebase, too. I pride
myself on making the code accessible and easy to get into, and I&rsquo;ll help you
learn to do the same when you integrate your with our project.</p>
<p>We would much rather give you advice on how to fix the problem yourself than to
fix the problem for you. Even if it takes more of our attention to do so, we get
the added benefit of a new person who is qualified to help out the next guy. A
person who is now fixing their own bugs and improving the software for everyone.
That&rsquo;s a much better outcome than having to waste our own time on a task we
aren&rsquo;t interested in.</p>
<p>It might be hard, but hey, it&rsquo;d be hard for us too. You&rsquo;ll learn and be better
for it. Wouldn&rsquo;t it be nice to add <em>[language you don&rsquo;t know]</em> or <em>[library you
don&rsquo;t know]</em> to your resume, anyway? If you&rsquo;re concerned about the scope of your
problem, how about asking about the low hanging fruit so you have easier tasks
to learn with?</p>
<p>The cards are stacked in your favor. The only problem is your defeatist
attitude. Just do it!</p>


guid: https://drewdevault.com/2017/01/13/The-problem-with-Python-3.html
title: The only problem with Python 3's str is that you don't grok it
url: https://drewdevault.com/2017/01/13/The-problem-with-Python-3.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I&rsquo;ve found myself explaining Python 3&rsquo;s str to people online more and more often
lately. There&rsquo;s this ridiculous claim about that Python 3&rsquo;s string handling is
broken or somehow worse than Python 2, and today I intend to put that myth to
rest.  Python 2 strings are broken, and Python 3 strings are sane. The only
problem is that you don&rsquo;t grok strings.</p>
<p>The basic problem many people seem to have with Python 3&rsquo;s strings arises when
they write code that treats bytes like a string, because that&rsquo;s how it was in
Python 2. Let me make this as clear as possible:</p>
<div class="loud">a bytes is not a string</div>
<style>
.loud {
    font-size: 14pt;
    font-weight: bold;
    text-align: center;
    margin-bottom: 1rem;
}
</style>
<p>I want you to read that, over and over again, until it sinks in. A string is
basically an array of characters (characters being Unicode codepoints), whereas
bytes is an array of bytes, aka octets, aka unsigned 8 bit integers. That&rsquo;s
right - bytes is an array of unsigned 8 bit integers, or as the name would
imply, bytes.  If you <em>ever</em> do string operations against bytes, you are Doing
It Wrong because bytes are not strings.</p>
<div class="loud">a bytes is not a string</div>
<p>It&rsquo;s entirely possible that your bytes contains an <em>encoded representation</em> of a
string. That encoding could be ASCII, UTF-8, UTF-32, etc. These encodings are
means of representing strings as bytes, aka unsigned 8 bit integers. In order to
treat it like a string, you first must <em>decode</em> it. Luckily Python 3 makes this
painless: <code>bytes.decode()</code>. This defaults to UTF-8, but you can specify any
encoding you want: <code>bytes.decode('latin-1')</code>. If you want bytes again, use
<code>str.encode()</code>, which again defaults to UTF-8 but accepts any encoding. If you
have a bytes that contains an encoded string, your first order of business is
decoding it.</p>
<div class="loud">a bytes is not a string</div>
<p>Let&rsquo;s look at some examples of why this matters in practice:</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="n">Python</span> <span class="mf">3.6</span><span class="o">.</span><span class="mi">0</span> <span class="p">(</span><span class="n">default</span><span class="p">,</span> <span class="n">Dec</span> <span class="mi">24</span> <span class="mi">2016</span><span class="p">,</span> <span class="mi">08</span><span class="p">:</span><span class="mo">03</span><span class="p">:</span><span class="mi">08</span><span class="p">)</span> 
<span class="p">[</span><span class="n">GCC</span> <span class="mf">6.2</span><span class="o">.</span><span class="mi">1</span> <span class="mi">20160830</span><span class="p">]</span> <span class="n">on</span> <span class="n">linux</span>
<span class="n">Type</span> <span class="s2">&#34;help&#34;</span><span class="p">,</span> <span class="s2">&#34;copyright&#34;</span><span class="p">,</span> <span class="s2">&#34;credits&#34;</span> <span class="ow">or</span> <span class="s2">&#34;license&#34;</span> <span class="k">for</span> <span class="n">more</span> <span class="n">information</span><span class="o">.</span>
<span class="o">&gt;&gt;&gt;</span> <span class="s1">&#39;おはようございます&#39;</span>
<span class="s1">&#39;おはようございます&#39;</span>
<span class="o">&gt;&gt;&gt;</span> <span class="s1">&#39;おはようございます&#39;</span><span class="p">[::</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="s1">&#39;すまいざごうよはお&#39;</span>
<span class="o">&gt;&gt;&gt;</span> <span class="s1">&#39;おはようございます&#39;</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="s1">&#39;お&#39;</span>
<span class="o">&gt;&gt;&gt;</span> <span class="s1">&#39;おはようございます&#39;</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="s1">&#39;は&#39;</span>
</code></pre></div><p>Or in Python 2:</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="n">Python</span> <span class="mf">2.7</span><span class="o">.</span><span class="mi">13</span> <span class="p">(</span><span class="n">default</span><span class="p">,</span> <span class="n">Dec</span> <span class="mi">21</span> <span class="mi">2016</span><span class="p">,</span> <span class="mo">07</span><span class="p">:</span><span class="mi">16</span><span class="p">:</span><span class="mi">46</span><span class="p">)</span> 
<span class="p">[</span><span class="n">GCC</span> <span class="mf">6.2</span><span class="o">.</span><span class="mi">1</span> <span class="mi">20160830</span><span class="p">]</span> <span class="n">on</span> <span class="n">linux2</span>
<span class="n">Type</span> <span class="s2">&#34;help&#34;</span><span class="p">,</span> <span class="s2">&#34;copyright&#34;</span><span class="p">,</span> <span class="s2">&#34;credits&#34;</span> <span class="ow">or</span> <span class="s2">&#34;license&#34;</span> <span class="k">for</span> <span class="n">more</span> <span class="n">information</span><span class="o">.</span>
<span class="o">&gt;&gt;&gt;</span> <span class="s1">&#39;おはようございます&#39;</span>
<span class="s1">&#39;</span><span class="se">\xe3\x81\x8a\xe3\x81\xaf\xe3\x82\x88\xe3\x81\x86\xe3\x81\x94\xe3\x81\x96\xe3\x81\x84\xe3\x81\xbe\xe3\x81\x99</span><span class="s1">&#39;</span>
<span class="o">&gt;&gt;&gt;</span> <span class="s1">&#39;おはようございます&#39;</span><span class="p">[::</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="s1">&#39;</span><span class="se">\x99\x81\xe3\xbe\x81\xe3\x84\x81\xe3\x96\x81\xe3\x94\x81\xe3\x86\x81\xe3\x88\x82\xe3\xaf\x81\xe3\x8a\x81\xe3</span><span class="s1">&#39;</span>
<span class="o">&gt;&gt;&gt;</span> <span class="k">print</span><span class="p">(</span><span class="s1">&#39;おはようございます&#39;</span><span class="p">[::</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="err">㾁㄁㖁㔁ㆁ㈂㯁㊁ã</span>
<span class="o">&gt;&gt;&gt;</span> <span class="s1">&#39;おはようございます&#39;</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="s1">&#39;</span><span class="se">\xe3</span><span class="s1">&#39;</span>
<span class="o">&gt;&gt;&gt;</span> <span class="s1">&#39;おはようございます&#39;</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="s1">&#39;</span><span class="se">\x81</span><span class="s1">&#39;</span>
</code></pre></div><p>For anything other than ASCII, Python 2 &ldquo;strings&rdquo; are broken. Python 3&rsquo;s string
handling is superb. The problem with it has only ever been that you don&rsquo;t
actually know how strings work. Instead of starting ignorant flamewars about it,
learn how it works.</p>
<h2 id="actual-examples-people-have-given-me">Actual examples people have given me</h2>
<p><strong>&ldquo;Python 3 can&rsquo;t handle bytes as file names&rdquo;</strong></p>
<p>Yes it can. Just stop treating them like strings:</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="o">&gt;&gt;&gt;</span><span class="nb">open</span><span class="p">(</span><span class="sa">b</span><span class="s1">&#39;test-</span><span class="se">\xd8\x01</span><span class="s1">.txt&#39;</span><span class="p">,</span> <span class="s1">&#39;w&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</code></pre></div><p>Note the use of bytes as the file name, not str. \xd8\x01 is unrepresentable as
UTF-8.</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="o">&gt;&gt;&gt;</span> <span class="p">[</span><span class="nb">open</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="sa">b</span><span class="s1">&#39;.&#39;</span><span class="p">)]</span>
<span class="p">[</span><span class="bp">None</span><span class="p">]</span>
</code></pre></div><p>Note the use of bytes as the path to os.listdir (the documentation says that if
you want bytes back as file names, pass bytes as the path. The docs are helpful
like that). Also note the lack of crashes or broken behavior.</p>
<p><strong>&ldquo;Python 3&rsquo;s csv module writes b&rsquo;Hello',b&rsquo;World' into CSV files&rdquo;</strong></p>
<p>CSV files are &ldquo;comma seperated values&rdquo;. Is each value an array of unsigned 8 bit
integers? No, of course not. They&rsquo;re strings. So why would you pass an array of
unsigned 8 bit integers to it?</p>
<p><strong>&ldquo;Python 3 doesn&rsquo;t support writing files as latin-1&rdquo;</strong></p>
<p>Sure it does.</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">&#39;some latin-1 file&#39;</span><span class="p">,</span> <span class="s1">&#39;rb&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
  <span class="n">text</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;latin-1&#39;</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">&#39;some utf8 file&#39;</span><span class="p">,</span> <span class="s1">&#39;wb&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
  <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">text</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">))</span>
</code></pre></div><div class="loud">a bytes is not a string</div>
<div class="loud">a bytes is not a string</div>
<div class="loud">a bytes is not a string</div>
<p>Python 2&rsquo;s shitty design has broken your mindset. Unlearn it.</p>
<h2 id="python-2-is-dead-long-live-python-3">Python 2 is dead, long live Python 3</h2>
<p>Listen. It&rsquo;s time you moved to Python 3. You&rsquo;re missing out on a lot of really
great improvements to the language and are stuck with a lot of problems. Python
2 is really being EoL&rsquo;d, and closing your eyes and covering your ears singing
&ldquo;la la la&rdquo; doesn&rsquo;t change that. The transition is really not that difficult or
time consuming, and well worth it. Some people say only new projects should be
written in Python 3. I say that&rsquo;s bollocks - all projects should be written in
Python 3 and you need to migrate, <em>now</em>.</p>
<p>Python 3 is better. Much, much better. For every legitimate criticism of Python
3 I&rsquo;ve seen, I&rsquo;ve seen 10 that are bullshit. Come join us in the wonderful world
of sane string handling, type decorations, async/await, and more awesome
features. Every library supports it now. Let go of your biases and evaluate the
language honestly.</p>


guid: https://drewdevault.com/2017/01/30/Lessons-to-learn-from-C.html
title: Lessons to learn from C
url: https://drewdevault.com/2017/01/30/Lessons-to-learn-from-C.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>C is my favorite language, though I acknowledge that it has its warts. I&rsquo;ve
tried looking at languages people hope will replace C (Rust, Go, etc), and
though they&rsquo;ve improved on some things they won&rsquo;t be supplanting C in my life
any time soon. I&rsquo;ll share with you what makes C a great language to me. Take
some of these things as inspiration for the next C replacement you write.</p>
<p>First of all, it&rsquo;s important to note that I&rsquo;m talking about the language, not
its standard library. The C standard library isn&rsquo;t <em>awful</em>, but it certainly
leaves a lot to be desired. I also want to place a few limitations on the kind
of C we&rsquo;re talking about - you can write bad code in any language, and C is no
different. For the purpose of argument, let&rsquo;s assume the following:</p>
<ul>
<li>C99 minimum</li>
<li>Absolutely no code in headers - just type definitions and function prototypes</li>
<li>Minimal use of typedefs</li>
<li>No macros</li>
<li>No compiler extensions</li>
</ul>
<p>I hold myself to these guidelines when writing C, and it is from this basis that
I compare other languages with C. It&rsquo;s not useful to compare bad C to another
language, because I wouldn&rsquo;t want to write bad C either.</p>
<p>Much of what I like about C boils down to this: <strong>C is simple</strong>. The ultimate
goal of any system should be to attain the simplest solution for the problems it
faces. C prefers to be conservative with new features. The lifetime of a feature
in Rust, for example, from proposal to shipping is generally 0 to 6 months. The
same process in C can take up to 10 years. C is a venerable language, and has
already long since finished adding core features. It is stable, simple, and
reliable.</p>
<p>To this end, language features map closely to behaviors common to most CPUs. C
strikes a nearly perfect balance of usability versus simplicity, which
results in a small set of features that are easy to reason about. A C expert
could roughly predict the assembly code produced by their compiler (assuming
<code>-O0</code>) for any given C function. It follows that C compilers are easy to write
and reason about.</p>
<p>The same person would also be able to give you a rough idea of the
performance characteristics of that function, pointing out things like cache
misses and memory accesses that are draining on speed, or giving you a precise
understanding of how the function handles memory. If I look at a function in
other languages, it&rsquo;s much more difficult to discern these things with any
degree of precision without actually compiling the code and looking at the
output.</p>
<p>The compiler also integrates very comfortably with the other tools near it, like
the assembler and linker. Symbols in C map 1:1 to symbols in the object files,
which means linking objects together is simple and easily reasoned about. It
also makes interop with other languages and tools straightforward - there&rsquo;s a
reason every language has a means of writing C bindings, but not generally C++
bindings. The use of headers to declare external symbols and types is also nicer
than some would have you believe, since it gives you an opportunity to organize
and document your API.</p>
<p>C is also the most portable programming language in the world. Every operating
system on every architecture has a C compiler, and they weren&rsquo;t really
considered a viable platform until it did. Once you have a C compiler you
generally have everything else, because everything else was either written in C
or was written in a language that was implemented in C. I can write C programs
on/for Linux, Windows, BSD, Minix, plan9, and a dozen other niche operating
systems, or even no operating system, on pretty much any CPU architecture I
want. No other language supports nearly as many platforms as C does.</p>
<p>With these benefits acknowledged, there are some things C could do better. The
standard library is one of them, but we can talk about that some other time.
Another is generics; using void* all the time isn&rsquo;t good. Some features from
other languages would be nice - I would take something similar to Rust&rsquo;s match
keyword. Of course, the fragility of memory management in C is a concern that
other languages are wise to address. Undefined behavior is awful.</p>
<p>Even for all of these warts, however, the basic simplicity and elegance of C
keeps me there. I would love to see a language that fixes these problems without
trying to be the kitchen sink, too.</p>
<p>In short, I like C because <strong>C is simple</strong>.</p>


guid: https://drewdevault.com/2017/02/22/cozy-devnotes-machine-specs.html
title: Compiler devnotes: Machine specs
url: https://drewdevault.com/2017/02/22/cozy-devnotes-machine-specs.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I have a number of long-term projects that I plan for on long timelines, on the
order of decades or more. One of these projects is cozy, a C toolchain. I
haven&rsquo;t talked about this project in public before, so I&rsquo;ll start by introducing
you to the project. The main C toolchains in the &ldquo;actually usable&rdquo; category are
GNU and LLVM, but I&rsquo;m satisfied with neither and I want to build my own
toolchain. I see no reason why compilers should be deep magic. Here are my goals
for cozy:</p>
<ul>
<li>Self hosting and written in C</li>
<li>An easy to grok codebase and internal design</li>
<li>Focused on C. No built-in support for other languages</li>
<li>Adding new targets architectures and ports should be straightforward</li>
<li>Modular build pipeline with lots of opportunities for external integrations</li>
<li>Trivially cross-compiles without building another version of the toolchain</li>
<li>Includes a decent optimizer</li>
</ul>
<p>Some other plans include opinionated warnings about code and minimal support for
language extensions. Ambitious goals, right? That&rsquo;s why this project is on my
long-term schedule. I&rsquo;ve found that large projects are entirely feasible, so
long as you (1) start them and (2) keep working on them for a long time. I don&rsquo;t
need to rush this - gcc and clang may not be ideal, but they work today. In
support of these goals, I&rsquo;ll be writing these dev notes to explain my design
choices and gather feedback — please <a href="mailto:sir@cmpwn.com">email me</a> if
you have some!</p>
<p>Since I want to place an emphasis on portability and retargetability, I&rsquo;m
starting by designing the machine spec and its support code, which is used to
add support for new architectures. I don&rsquo;t like gcc&rsquo;s lisp specs, and I <em>really</em>
don&rsquo;t like LLVM&rsquo;s &ldquo;huge pile of C++&rdquo; approach. I think a really good machine
spec meets these goals:</p>
<ul>
<li>Easy to write and human friendly</li>
<li>More about data than code, but</li>
<li>Easily extended with C to support architecture-specific nuances</li>
<li>Provides loads of useful metadata about the target architecture</li>
<li>Exposes information about the speed and side-effects of each instruction</li>
<li>Can also be used to generate an assembler and disassembler</li>
<li>Easily reused to create derivative architectures</li>
</ul>
<p>Adding a new architecture should be a weekend project, and when you&rsquo;re done the
entire toolchain should both support and run on your new architecture. I set out
to come up with a new syntax that could potentially meet these goals. I started
with the Z80 architecture in mind because it&rsquo;s simple, I&rsquo;m intimately familiar
with it, and I want cozy to be able to target 8-bit machines just as easily as
32 or 64 bit.</p>
<p>For reference, here are the gcc and LLVM guides on adding new targets:</p>
<ul>
<li><a href="https://gcc.gnu.org/onlinedocs/gccint/Back-End.html">gcc - Anatomy of a Target Back End</a></li>
<li><a href="http://llvm.org/docs/WritingAnLLVMBackend.html">Writing an LLVM Backend</a></li>
</ul>
<p>The cozy machine spec is a cross between ini files, yaml, and a custom syntax.
The format is somewhat complex, but once understood is intuitive and flexible.
At the top level, it looks like an ini file:</p>
<div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml"><span class="p">[</span><span class="l">metadata]</span><span class="w">
</span><span class="w"></span><span class="c"># ...</span><span class="w">
</span><span class="w">
</span><span class="w"></span><span class="p">[</span><span class="l">registers]</span><span class="w">
</span><span class="w"></span><span class="c"># ...</span><span class="w">
</span><span class="w">
</span><span class="w"></span><span class="p">[</span><span class="l">macros]</span><span class="w">
</span><span class="w"></span><span class="c"># ...</span><span class="w">
</span><span class="w">
</span><span class="w"></span><span class="p">[</span><span class="l">instructions]</span><span class="w">
</span><span class="w"></span><span class="c"># ...</span><span class="w">
</span></code></pre></div><h3 id="metadata">Metadata</h3>
<p>The <strong>metadata</strong> section contains some high-level information about the
architecture design, and is the simplest section to understand. It currently
looks like this for z80:</p>
<div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml"><span class="p">[</span><span class="l">metadata]</span><span class="w">
</span><span class="w"></span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">z80</span><span class="w">
</span><span class="w"></span><span class="nt">bits</span><span class="p">:</span><span class="w"> </span><span class="m">8</span><span class="w">
</span><span class="w"></span><span class="nt">endianness</span><span class="p">:</span><span class="w"> </span><span class="l">little</span><span class="w">
</span><span class="w"></span><span class="nt">signedness</span><span class="p">:</span><span class="w"> </span><span class="l">twos-complement</span><span class="w">
</span><span class="w"></span><span class="nt">cache</span><span class="p">:</span><span class="w"> </span><span class="l">none</span><span class="w">
</span><span class="w"></span><span class="nt">pipeline</span><span class="p">:</span><span class="w"> </span><span class="l">none</span><span class="w">
</span></code></pre></div><p>This isn&rsquo;t comprehensive, and I&rsquo;ll be adding more metadata as it becomes
necessary. On LLVM, this sort of information is encoded into a string that looks
something like this: <code>&quot;e-p:16:8:8-i8:8:8-i16:8:8-n8:16&quot;</code>. This string is passed
into the <code>LLVMTargetMachine</code> base constructor in C++. I think we can do a hell
of a lot better than that!</p>
<h3 id="registers">Registers</h3>
<p>The <strong>registers</strong> section describes the registers on this architecture.</p>
<div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml"><span class="p">[</span><span class="l">registers]</span><span class="w">
</span><span class="w"></span><span class="nt">BC</span><span class="p">:</span><span class="w"> </span><span class="m">16</span><span class="w">
</span><span class="w">    </span><span class="nt">B</span><span class="p">:</span><span class="w"> </span><span class="m">8</span><span class="w">
</span><span class="w">    </span><span class="nt">C</span><span class="p">:</span><span class="w"> </span><span class="m">8</span><span class="l">; offset=8</span><span class="w">
</span><span class="w"></span><span class="nt">DE</span><span class="p">:</span><span class="w"> </span><span class="m">16</span><span class="w">
</span><span class="w">    </span><span class="nt">D</span><span class="p">:</span><span class="w"> </span><span class="m">8</span><span class="w">
</span><span class="w">    </span><span class="nt">E</span><span class="p">:</span><span class="w"> </span><span class="m">8</span><span class="l">; offset=8</span><span class="w">
</span><span class="w"></span><span class="nt">HL</span><span class="p">:</span><span class="w"> </span><span class="m">16</span><span class="w">
</span><span class="w">    </span><span class="nt">H</span><span class="p">:</span><span class="w"> </span><span class="m">8</span><span class="w">
</span><span class="w">    </span><span class="nt">L</span><span class="p">:</span><span class="w"> </span><span class="m">8</span><span class="l">; offset=8</span><span class="w">
</span><span class="w"></span><span class="nt">SP</span><span class="p">:</span><span class="w"> </span><span class="m">16</span><span class="l">; stack</span><span class="w">
</span><span class="w"></span><span class="nt">PC</span><span class="p">:</span><span class="w"> </span><span class="m">16</span><span class="l">; program</span><span class="w">
</span></code></pre></div><p>Here we can start to see some interesting syntax and get an idea of the design
of cozy machine specs. The contents of each section are keys, which have values,
attributes, and children. The format looks like this:</p>
<div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml"><span class="nt">key</span><span class="p">:</span><span class="w"> </span><span class="l">value; attributes, ...</span><span class="w">
</span><span class="w">    </span><span class="l">children...</span><span class="w">
</span></code></pre></div><p>In this example, we&rsquo;ve defined the BC, DE, HL, SP, and PC registers. HL, DE, and
BC are general purpose 16-bit registers, and each can also be used as two
separate 8-bit registers. The attributes for these sub-registers indicates their
offsets in the parent register. We also define the stack and program registers,
SP and PC, which use the stack and program attributes to indicate their special
purposes.</p>
<p>We can also describe CPU flags in this section:</p>
<div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml"><span class="p">[</span><span class="l">registers]</span><span class="w">
</span><span class="w"></span><span class="nt">AF</span><span class="p">:</span><span class="w"> </span><span class="m">16</span><span class="l">; special</span><span class="w">
</span><span class="w">    </span><span class="nt">A</span><span class="p">:</span><span class="w"> </span><span class="m">8</span><span class="l">; accumulator</span><span class="w">
</span><span class="w">    </span><span class="nt">F</span><span class="p">:</span><span class="w"> </span><span class="m">8</span><span class="l">; flags, offset 8;; flag</span><span class="w">
</span><span class="w">        </span><span class="nt">_C</span><span class="p">:</span><span class="w">  </span><span class="m">1</span><span class="w">
</span><span class="w">        </span><span class="nt">_N</span><span class="p">:</span><span class="w">  </span><span class="m">1</span><span class="l">; offset 1</span><span class="w">
</span><span class="w">        </span><span class="nt">_PV</span><span class="p">:</span><span class="w"> </span><span class="m">1</span><span class="l">; offset 2</span><span class="w">
</span><span class="w">        </span><span class="nt">_3</span><span class="p">:</span><span class="w">  </span><span class="m">1</span><span class="l">; offset 3, undocumented</span><span class="w">
</span><span class="w">        </span><span class="nt">_H</span><span class="p">:</span><span class="w">  </span><span class="m">1</span><span class="l">; offset 4</span><span class="w">
</span><span class="w">        </span><span class="nt">_5</span><span class="p">:</span><span class="w">  </span><span class="m">1</span><span class="l">; offset 5, undocumented</span><span class="w">
</span><span class="w">        </span><span class="nt">_Z</span><span class="p">:</span><span class="w">  </span><span class="m">1</span><span class="l">; offset 6</span><span class="w">
</span><span class="w">        </span><span class="nt">_S</span><span class="p">:</span><span class="w">  </span><span class="m">1</span><span class="l">; offset 7</span><span class="w">
</span></code></pre></div><p>Here we introduce another feature of cozy specs with <code>F: 8; flags, offset 8;; flag</code>. Using <code>;;</code> adds those attributes to all children of this key, so each of
_C, _N, etc have the <code>flag</code> attribute.</p>
<p>Take note of the &ldquo;undocumented&rdquo; attribute here. Some of the metadata included
in a spec can be applied to cozy tools. Some of it, however, is there for other
tools to utilize. We have a good opportunity to make a machine-readable
description of the architecture, so I&rsquo;ve opted to include a lot of extra details
in machine specs that third parties could utilize (though there might be a
-fno-undocumented compiler flag some day, I guess).</p>
<h3 id="macros">Macros</h3>
<p>The <strong>macros</strong> section is heavily tied to the instructions section. Most instruction
sets are quite large, and I don&rsquo;t want to burden spec authors with writing out
the entire thing. We can speed up their work by providing macros.</p>
<p>z80 instructions have a few sets of common patterns in their encodings. Register
groups are often represented by the same set of bits, and we can make our
instruction set specification more concise by taking advantage of this. For
example, here&rsquo;s a macro that we can use for instructions that can use either the
BC, DE, HL, or SP registers:</p>
<div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml"><span class="p">[</span><span class="l">macros]</span><span class="w">
</span><span class="w"></span><span class="nt">reg_BCDEHLSP</span><span class="p">:</span><span class="w">
</span><span class="w">    </span><span class="nt">BC</span><span class="p">:</span><span class="w"> </span><span class="m">00</span><span class="w">
</span><span class="w">    </span><span class="nt">DE</span><span class="p">:</span><span class="w"> </span><span class="m">01</span><span class="w">
</span><span class="w">    </span><span class="nt">HL</span><span class="p">:</span><span class="w"> </span><span class="m">10</span><span class="w">
</span><span class="w">    </span><span class="nt">SP</span><span class="p">:</span><span class="w"> </span><span class="m">11</span><span class="w">
</span></code></pre></div><p>We have the name of the macro as the top-level key, in this case <code>reg_BCDEHLSP</code>.
We can later refer to this macro with <code>@reg_BCDEHLSP</code>. Then, we have each of the
cases it can match on, and the binary values these correspond to when encoded in
an instruction.</p>
<h3 id="instructions">Instructions</h3>
<p>The instructions section brings everything together and defines the actual
instructions available on this architecture. Instructions can be organized into
groups at the spec author&rsquo;s pleasure, which can be referenced by derivative
architectures. Here we can take a look at the &ldquo;load&rdquo; group:</p>
<div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml"><span class="p">[</span><span class="l">instructions]</span><span class="w">
</span><span class="w"></span><span class="nt">.load</span><span class="p">:</span><span class="w">
</span><span class="w">    </span><span class="nt">ld</span><span class="p">:</span><span class="w">
</span><span class="w">        </span><span class="nt">@reg_BCDEHLSP, @imm[16]</span><span class="p">:</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="l">$1 0001 $2</span><span class="w">
</span></code></pre></div><p>On z80, the <code>ld</code> instruction is similar to the <code>mov</code> instruction on Intel
architectures. It assigns the second argument to the first. This could be used
to assign registers to each other (e.g. <code>ld a, b</code> to set A = B), to set
registers to constants, and so on. Our example here uses our macro from earlier
to match instructions like this:</p>
<pre><code>ld hl, 0x1234
</code></pre>
<p>The value for this key may reference the arguments with variables. $1 here
equals <code>10</code>, from the macro. The <code>imm</code> built-in is implemented in C to match
constants and provides $2. An assembler could use this information to assemble
our example instruction into this machine code:</p>
<pre><code>00100001 00110100 00010010
</code></pre>
<p>Which will load HL with the value 0x1234 when executed.</p>
<h3 id="lots-more-metadata">Lots more metadata</h3>
<p>Now that we have the basics down, let&rsquo;s dive into some deeper details. Cozy
specs are designed to provide most of the information the <em>entire toolchain</em>
needs to support an architecture. The information we have so far could be used
to generate assemblers and disassemblers, but I want this file to be able to
generate things like optimizers as well. You can add the necessary metadata to
each instruction by utilizing attributes.</p>
<p>Consider the z80 instruction LDIR, which stands for
&ldquo;load/decrement/increment/repeat&rdquo;. This instruction is used for memcpy
operations. To use it, you set the HL register to a source address, the DE
register to a destination address, and BC to a length. This instruction looks
like this in the spec:</p>
<div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml"><span class="nt">ldir</span><span class="p">:</span><span class="w"> </span><span class="m">11101101</span><span class="w"> </span><span class="m">10110000</span><span class="l">; uses[HL, DE, BC], \</span><span class="w">
</span><span class="w">    </span><span class="l">affects[HL[+BC], DE[+BC], BC[0]], \</span><span class="w">
</span><span class="w">    </span><span class="l">flags[_H:0,_N:0,_PV:0], cycles[16 + BC * 5]</span><span class="w">
</span></code></pre></div><p>That&rsquo;s a lot of attributes! The purpose of these attributes are to give the
toolchain insights into the registers this instruction uses, its side effects,
and how fast it is. These attributes can help us compare the efficiency of
different approaches and understand the how the state of registers evolves
during a function, which leads to all sorts of useful optimizations.</p>
<p>The <code>affects</code> attribute, for example, tells us how each register is affected by
this instruction. We can see that after this instruction, HL and DE will have
had BC added to them, and BC will have been set to 0. We can make all sorts of
optimizations based on this knowledge. Here are some examples:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">char</span> <span class="o">*</span><span class="n">dest</span><span class="p">,</span> <span class="o">*</span><span class="n">src</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">len</span> <span class="o">=</span> <span class="mi">10</span><span class="p">;</span>
<span class="n">memcpy</span><span class="p">(</span><span class="n">dest</span><span class="p">,</span> <span class="n">src</span><span class="p">,</span> <span class="n">len</span><span class="p">);</span>
<span class="n">src</span> <span class="o">+=</span> <span class="n">len</span><span class="p">;</span>
</code></pre></div><p>The compiler can assign <code>src</code> to HL, <code>dest</code> to DE, and <code>len</code> to BC. We can then
optimize out the final statement entirely because we know that the LDIR
instruction will have already added BC to HL for us.</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">char</span> <span class="o">*</span><span class="n">dest</span><span class="p">,</span> <span class="o">*</span><span class="n">src</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">len</span> <span class="o">=</span> <span class="mi">10</span><span class="p">;</span>
<span class="n">memcpy</span><span class="p">(</span><span class="n">dest</span><span class="p">,</span> <span class="n">src</span><span class="p">,</span> <span class="n">len</span><span class="p">);</span>
<span class="kt">int</span> <span class="n">foobar</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</code></pre></div><p>In this case, the register allocator can just assign BC to <code>foobar</code> and avoid
initializing it because we know it&rsquo;s already going to be zero. Many other
optimizations are made possible when we are keeping track of the side effects of
each instruction.</p>
<h2 id="next-steps">Next steps</h2>
<p>I&rsquo;ve iterated over this spec design for a while now, and I&rsquo;m pretty happy with
it. I would love to hear your feedback. Assuming that this looks good, my next
step is writing more specs, and a tool that parses and compiles them to C. These
C files are going to be linked into <code>libcozyspec</code>, which will provide an API to
access all of this metadata from C. It will also include an instruction matcher,
which will be utilized by the next step - writing the assembler.</p>
<p>The assembler is going to take a while, because I don&rsquo;t want to go the gas route
of making a half-baked assembler that&rsquo;s more useful for compiling the C
compiler&rsquo;s output than for anything else. I want to make an assembler that
assembly programmers would <em>want</em> to use.</p>
<p>I have not yet designed an intermediate bytecode for the compiler to use, but
one will have to be made. The machine spec will likely change somewhat to
accommodate this. Some of the conversion from internal bytecode to target
assembly can likely be inferred from metadata, but some will have to be done
manually for each architecture.</p>
<p><a href="https://sr.ht/7_Pe.txt">Here&rsquo;s the entire z80 spec</a> I&rsquo;ve been working on, for
your reading pleasure.</p>


guid: https://drewdevault.com/2017/03/15/How-I-learned-to-stop-worrying-and-love-C.html
title: Principles for C programming
url: https://drewdevault.com/2017/03/15/How-I-learned-to-stop-worrying-and-love-C.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>In the words of Doug Gwyn, &ldquo;Unix was not designed to stop you from doing stupid
things, because that would also stop you from doing clever things&rdquo;. C is a very
powerful tool, but it is to be used with care and discipline. Learning this
discipline is well worth the effort, because C is one of the best programming
languages ever made. A disciplined C programmer will&hellip;</p>
<p><strong>Prefer maintainability</strong>. Do not be clever where cleverness is not required.
Instead, seek out the simplest and most understandable solution that meets the
requirements. Most concerns, including performance, are secondary to
maintainability. You should have a performance budget for your code, and you
should be comfortable spending it.</p>
<p>As you become more proficient with the language and learn about more features
you can take advantage of, you should also be learning when not to use them.
It&rsquo;s more important that a novice could understand your code than it is to use
some interesting way of solving the problem. Ideally, a novice will understand
your code <em>and</em> learn something from it. Write code as if the person maintaining
it was you, circa last year.</p>
<p><strong>Avoid magic</strong>. Do not use macros<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. Do not use a typedef to hide a pointer or
avoid writing &ldquo;struct&rdquo;. Avoid writing complex abstractions. Keep your build
system simple and transparent. Don&rsquo;t use stupid hacky crap just because it&rsquo;s a
cool way of solving the problem. The underlying behavior of your code should be
apparent even without context.</p>
<p>One of C&rsquo;s greatest advantages is its transparency and simplicity. This should
be embraced, not subverted. But in the fine C tradition of giving yourself
enough rope to hang yourself with, you can use it for magical purposes. You
must not do this. Be a muggle.</p>
<p><strong>Recognize and avoid dangerous patterns</strong>. Do not use fixed size buffers with
variable sized data - always calculate how much space you&rsquo;ll need and allocate
it. Read the man pages for functions you use and handle their failure modes.
Immediately convert unsafe user input into sanitized C structures. If you later
have to present this data to the user, keep it in C structures until the last
possible moment. Learn of and use extra care around sensitive functions like
strcat.</p>
<p>Writing C is sometimes like handling a gun. Guns are important tools, but
accidents with them can be very bad. You treat guns with care: you don&rsquo;t point
them at anything you love, you exercise good trigger discipline, and you treat
it like it&rsquo;s always loaded. And like guns are useful for making holes in things,
C is useful for writing kernels with.</p>
<p><strong>Take care organizing the code</strong>. Never put code into a header. Never use the
<code>inline</code> keyword. Put separate concerns in separate files. Use static functions
liberally to organize your logic. Use a coding style that gives everything
enough breathing room to be easy on the eyes. Use single letter variable names
when their purpose is self-evident and descriptive names when it&rsquo;s not, and
avoid neither.</p>
<p>I like to organize my code into directories that implement some group of
functions, and give each function its own file. This file will often contain
lots of static functions, but they all serve to organize the behavior this file
is responsible for implementing. Write up a header to give others access to
this module. And use the Linux kernel coding style, god dammit.</p>
<p><strong>Use only standard features</strong>. Do not assume the platform is Linux. Do not
assume the compiler is gcc. Do not assume the libc is glibc. Do not assume the
architecture is x86. Do not assume the coreutils are GNU. Do not define
_GNU_SOURCE.</p>
<p>If you must use platform-specific features, describe an interface for it,
then write platform-specific support code separately. Under no circumstances
should you ever use gcc extensions or glibc extensions. GNU is a blight on this
Earth, do not let it infect your code.</p>
<p><strong>Use a disciplined workflow</strong>. Have a disciplined approach to version control,
too. Write thoughtful commit messages - briefly explain the change in the first
line, and add justification for it in the extended commit message. Work in
feature branches with clearly defined goals, and do not include changes that
don&rsquo;t serve that goal. Do not be afraid to rebase and edit your branch&rsquo;s history
so that it presents your changes clearly.</p>
<p>When you have to return to your code later, you will be thankful for the
detailed commit message you wrote. Others who interact with your code will be
thankful for this as well. When you see some stupid code, it&rsquo;s nice to know what
the bastard was thinking at the time, especially when the bastard in question
was you.</p>
<p><strong>Do strict testing and reviews</strong>. Identify the different possible code paths
that your changes may take. Test each of them for the correct behavior. Give it
incorrect input. Give it inputs that could &ldquo;never happen&rdquo;. Pay special attention
to error-prone patterns. Look for places to simplify the code and make the
processes clearer.</p>
<p>Next, give your changes to another human to review. This human should apply the
same process and sign off on your changes. Review with discipline as well,
taking all of the same steps. Review like it&rsquo;ll be your ass on the line if
there&rsquo;s a problem with this code.</p>
<p><strong>Learn from mistakes</strong>. First, fix the bug. Then, fix the real bug: your
process allowed this mistake to happen. Bring your code reviewer into the
discussion - this is their fault, too. Critically examine the process of
writing, reviewing, and deploying this code, and seek out the root cause.</p>
<p>The solution might be simple, like adding strcat to the list of functions that
should trigger your &ldquo;review this code carefully&rdquo; reflex. It might be employing
static analysis so a computer can detect this problem for you. Perhaps the code
needs to be refactored so it&rsquo;s simpler and easier to spot errors in. Failing to
reflect on how to avoid future fuck-ups would be the real fuck-up here.</p>
<hr>
<p>It&rsquo;s important to remember that rules are made to be broken. There may be cases
where things that are discouraged should be used, and things that are encouraged
disregarded. You should strive to make such cases the exception, not the norm,
and carefully justify them when they happen.</p>
<p>C is the shit. I love it, and I hope more people can learn to see it the way I
do. Good luck!</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Defining constants with them is fine, though <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2017/04/13/MSG_PEEK-is-more-common-than-you-think-CVE-2016-10229.html
title: MSG_PEEK is pretty common, CVE-2016-10229 is worse than you think
url: https://drewdevault.com/2017/04/13/MSG_PEEK-is-more-common-than-you-think-CVE-2016-10229.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I heard about <a href="https://nvd.nist.gov/vuln/detail/CVE-2016-10229">CVE-2016-10229</a>
earlier today. In a nutshell, it allows for arbitrary code execution via UDP
traffic if userspace programs are using <code>MSG_PEEK</code> in their <code>recv</code> calls. I
quickly updated my kernels and rebooted any boxes where necessary, but when I
read the discussions on this matter I saw people downplaying this issue by
claiming <code>MSG_PEEK</code> is an obscure feature.</p>
<p>I don&rsquo;t want to be a fear monger and I&rsquo;m by no means a security expert but I
suspect that this is a deeply incorrect conclusion. If I understand this
vulnerability right you need to drop everything and update any servers running
a kernel &lt;4.5 <em>immediately</em>. <code>MSG_PEEK</code> allows a programmer using UDP to
read from the kernel&rsquo;s UDP buffer without consuming the data (so subsequent
reads will continue to read the same data). This immediately sounds to me like
a pretty useful feature that a lot of software might use, not an obscure one.</p>
<p>I did quick search for software where <code>MSG_PEEK</code> appears in the source code
somewhere. This does not necessarily mean that it&rsquo;s exploitable, but should
certainly raise red flags. Here&rsquo;s a list of some notable software I found:</p>
<ul>
<li>nginx</li>
<li>haproxy</li>
<li>curl</li>
<li>gnutls</li>
<li>jack2</li>
<li>lynx</li>
<li>plex (and kodi/xbmc)</li>
<li>busybox</li>
</ul>
<p>I also found a few things like programming languages and networking libraries
that you might expect to have MSG_PEEK if only to provide that functionality to
programmers leveraging them. I didn&rsquo;t investigate too deeply into whether or not
that was the case or if this software is using the feature in a less apparent
way, but in this category I found Python, Ruby, Node.js, smalltalk, octave,
libnl, and socat. I used searchcode.com to find these - <a href="https://searchcode.com/?q=MSG_PEEK">here&rsquo;s the full search
results</a>.</p>
<p>Again, I&rsquo;m not a security expert, but I&rsquo;m <em>definitely</em> spooked enough to update
my shit and I suggest you do so as well. Red Hat, Debian, and Ubuntu are all
unaffected because of the kernel they ship. Note, however, that many cloud
providers do not let you choose your own kernel. This could mean that you are
affected even if you&rsquo;re running a distribution like Debian. Double check it -
use <code>uname -r</code> and update+reboot if necessary.</p>


guid: https://drewdevault.com/2017/04/29/State-of-sway-April-2017.html
title: State of Sway April 2017
url: https://drewdevault.com/2017/04/29/State-of-sway-April-2017.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Development on Sway continues. I thought we would have slowed down a lot more by
now, but every release still comes with new features - Sway 0.12 added
redshift support and binary space partitioning layouts. Sway 0.13.0 is
coming soon and includes, among other things, nvidia proprietary driver support.
We already have some interesting features slated for Sway 0.14.0, too!</p>
<p>Today Sway has 21,446 lines of C (and 4,261 lines of header files) written by 81
authors across 2,263 commits. These were written through 653 pull requests and
529 issues. Sway packages are available today in the official repos of pretty
much every distribution except for Debian derivatives, and a PPA is available
for those guys.</p>
<p><a href="https://sr.ht/ICd5.png"><img src="https://sr.ht/ICd5.png" alt=""></a></p>
<p>For those who are new to the project, <a href="http://swaywm.org">Sway</a> is an
i3-compatible Wayland compositor. That is, your existing <a href="http://i3wm.org/">i3</a>
configuration file will work as-is on Sway, and your keybindings and colors and
fonts and for_window rules and so on will all be the same. It&rsquo;s i3, but for
Wayland, plus it&rsquo;s got some bonus features. Here&rsquo;s a quick rundown of what&rsquo;s
new since the <a href="https://drewdevault.com/2016/12/27/State-of-sway.html">previous state of Sway</a>:</p>
<ul>
<li>Redshift support</li>
<li>Improved security configuration</li>
<li>Automatic binary space partitioning layouts ala AwesomeWM</li>
<li>Support for more i3 window criterion</li>
<li>Support for i3 marks</li>
<li>xdg_shell v6 support (Wayland thing, makes more native Wayland programs work)</li>
<li>We&rsquo;ve switched from X.Y to X.Y.Z releases, Z releases shipping bugfixes while
the next Y release is under development</li>
<li>Lots of i3 compatibility improvements</li>
<li>Lots of documentation improvements</li>
<li>Lots of bugfixes</li>
</ul>
<p>The new <a href="https://github.com/SirCmpwn/sway/issues/986">bounty program</a> has also
raised $1,200 to support Sway development! Several bounties have been awarded,
including redshift support and i3 marks, but every awardee chose to redonate
their reward to the bounty pool. Thanks to everyone who&rsquo;s donated and everyone
who&rsquo;s worked on new features! Bounties have also been awarded for features in
the Wayland ecosystem beyond Sway - a fact I&rsquo;m especially proud of. If you want
a piece of that $1,200 pot, <a href="http://webchat.freenode.net/?channels=sway&amp;uio=d4">join us on
IRC</a> and we&rsquo;ll help you get started.</p>
<p>Many new developments are in the pipeline for you. 0.13.0 is expected to
ship within the next few weeks - <a href="https://github.com/SirCmpwn/sway/issues/1162#issuecomment-295012255">here&rsquo;s a sneak peek at the
changelog</a>.
In the future releases, development is ongoing for tray icons (encouraged by the
sweet $270 bounty sitting on that feature), and several other features for
0.14.0 have been completed. We&rsquo;ve also started work on a long term project to
replace our compositor plumbling library, wlc, with a new one:
<a href="https://github.com/SirCmpwn/wlroots">wlroots</a>. This should allow us to fix many
of the more difficult bugs in Sway, and opens the doors for <em>many</em> features that
weren&rsquo;t previously possible. It should also give us a platform on which we can
build standard protocols that other compositors can implement, unifying the
Wayland platform a bit more.</p>
<p>Many thanks to <a href="https://github.com/SirCmpwn/sway/graphs/contributors">everyone that&rsquo;s contributed to
sway</a>! There&rsquo;s no way Sway
would have enjoyed its success without your help. That wraps things up for
today, thanks for using Sway and look forward to Sway 1.0!</p>
<hr>
<p>Note: future posts like this will omit some of the stats that were included in
the previous posts. You can use the following commands to find them for
yourself:</p>
<div class="highlight"><pre class="chroma"><code class="language-bash" data-lang="bash"><span class="c1"># Lines of code per author:</span>
git ls-tree -r -z --name-only HEAD -- */*.c <span class="se">\
</span><span class="se"></span>    <span class="p">|</span> xargs -0 -n1 git blame --line-porcelain HEAD <span class="se">\
</span><span class="se"></span>    <span class="p">|</span> grep  <span class="s2">&#34;^author &#34;</span> <span class="p">|</span> sort <span class="p">|</span> uniq -c <span class="p">|</span> sort -nr
<span class="c1"># Commits per author:</span>
git shortlog
</code></pre></div>


guid: https://drewdevault.com/2017/05/05/Building-a-real-Linux-distro.html
title: Building a "real" Linux distro
url: https://drewdevault.com/2017/05/05/Building-a-real-Linux-distro.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I recently saw a post on Hacker News: &ldquo;<a href="https://github.com/MichielDerhaeg/build-linux">Build yourself a
Linux</a>&rdquo;, a cool project
that guides you through building a simple Linux system. It&rsquo;s similar to Linux
from Scratch in that it helps you build a simple Linux system for personal use.
I&rsquo;d like to supplement this with some insight into my experience with a more
difficult task: building a full blown Linux distribution. The result is
<a href="http://agunix.org">agunix</a>, the &ldquo;silver unix&rdquo; system.</p>
<p>For many years I&rsquo;ve been frustrated with every distribution I&rsquo;ve tried. Many of
them have compelling features and design, but there&rsquo;s always a catch. The
popular distros are stable and portable, but cons include bloat, frequent use of
GNU, systemd, and often apt. Some more niche distros generally have good points
but often have some combination of GNU, an init system I don&rsquo;t like, poor docs,
dynamic linking, or an overall amateurish or incomplete design. Many of them are
tolerable, but none have completely aligned with my desires.</p>
<p>I&rsquo;ve also looked at not-Linux - I have plenty of beefs with the Linux kernel. I
like the BSD kernels, but I dislike the userspaces (though NetBSD is pretty good)
I like the microkernel design of Minix, but it&rsquo;s too unstable and has shit
hardware support. plan9/9front has the most elegant kernel and userspace design
ever made, but it&rsquo;s not POSIX and has shit hardware support. Though none of
these userspaces are for me, I intend to attempt a port of the agunix userspace
to all of their kernels at some point (a KFreeBSD port is underway).</p>
<p>After trying a great number of distros and coming away with a kind of
dissatisfaction unique to each one, I resolved to make a distro that embodied my
own principles about userspace design. It turns out this is a ton of work -
here&rsquo;s how it&rsquo;s done.</p>
<p>Let&rsquo;s distinguish a Linux &ldquo;system&rdquo; from a Linux &ldquo;distribution&rdquo;. A Linux system
is anything that boots up from the Linux kernel. A Linux <em>distribution</em>, on the
other hand, is a Linux system that can be <em>distributed</em> to end users.  It&rsquo;s this
sort of system that I wanted to build. In my opinion, there are two core
requirements for a Linux system to become a Linux distribution:</p>
<ol>
<li>It has a package manager (or some other way of staying up to date)</li>
<li>It is self-hosting (it can compile itself and all of the infrastructure runs
on it)</li>
</ol>
<p>The first order of business in creating a Linux distro is to fulfill these two
requirements. Getting to this stage is called <em>bootstrapping</em> your distribution -
everything else can come later. To do this, you&rsquo;ll need to port your package
manager to your current system, and start building the base packages with it.
If your new distro doesn&rsquo;t use the same architecture or libc as your current
system, you also need to build a cross compiler and use it for building your
new packages.</p>
<p>My initial approach was different - I used my cross compiler to fill up a chroot
with software without using my package manager, hoping to later bootstrap from
it. I used this approach on my first 3 attempts before deciding to make
base packages on the host system instead. With this approach, I started by
building packages that weren&rsquo;t necessarily self hosting - they used the
host-specific cross compiler builds and such - but produced working packages for
the new environment. I built packages for:</p>
<ul>
<li>my package manager</li>
<li>musl libc</li>
<li>bash</li>
<li>busybox</li>
<li>autotools</li>
<li>make</li>
<li>gcc (clang can&rsquo;t compile the Linux kernel)</li>
<li>vim</li>
</ul>
<p>I also had to package all of the dependencies for these. Once I had a system
that was reasonably capable of compiling arbitrary software, I transferred my
PKGBUILDs (scripts used to build packages) to my chroot and started tweaking
them to re-build packages from the new distro itself. This process took months to
get completely right - there are <em>tons</em> of edge cases and corner cases. Simply
getting this software to run in a new Linux system is only moderately difficult -
getting a system that can build itself is <em>much harder</em>. I was successful on
my 4th attempt, but threw it out and redid it to get a cleaner distribution with
the benefit of hindsight. This became agunix.</p>
<p>Once you reach this stage you can go ham on making packages for your system. The
next step for me was graduating from a chroot to dedicated hardware. I built out
an init system with runit and <a href="http://git.agunix.org/init/">agunix-init</a> and
various other packages that are useful on a proper install. I also compiled a
kernel without support for loadable modules (on par with the static linking theme
of agunix). If you make your own Linux distribution you will probably have to
figure out modules yourself, likely implicating something like eudev.
Eventually, I was able to get agunix <a href="https://sr.ht/OzCq.jpg">running on my
laptop</a>, which has now become my primary agunix dev
machine (often via SSH from my dev desktop).</p>
<p>The next stage for me was getting agunix.org up and running on agunix. I
deliberately chose not to have a website until it could be hosted on agunix
itself. I deployed agunix to a VPS, then ported nginx and put the website up.
The rest of the infrastructure was a bit more difficult: cgit took me about 10
packages of work, and bugzilla was about 100 packages of work. Haven&rsquo;t started
working on mailman yet.</p>
<p>Then begins the eternal packaging phase. At this point you&rsquo;ve successfully made
a Linux distribution, and now you just need to fill it with packages. This takes
<em>forever</em>. I have made 407 packages to date and I still don&rsquo;t have a desktop to
show for it (I&rsquo;m <em>almost</em> there, just have to make a few dozen more packages
before <a href="https://github.com/SirCmpwn/sway">sway</a> will run). At this point to have
success you need others to buy into your ideas and start contributing - it&rsquo;s
impossible to package everything yourself. Speaking of which, check out
<a href="http://agunix.org">agunix.org</a> and see if you like it! I haven&rsquo;t been doing
much marketing for this distro yet, but I do have a little bit of help. If
you&rsquo;re interested in contributing in a new distro, we have lots of work for you
to do!</p>


guid: https://drewdevault.com/2017/05/11/Rotating-passwords.html
title: Rotating passwords in bulk in the wake of security events
url: https://drewdevault.com/2017/05/11/Rotating-passwords.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I&rsquo;ve been putting this post off for a while. Do you remember the <a href="https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/">CloudFlare
security
problem</a>
that happened a few months ago? This is the one that disclosed huge amounts of
sensitive information for huge numbers websites. When this happened, your
accounts on <a href="https://github.com/pirate/sites-using-cloudflare">thousands of
websites</a> were potentially
compromised.</p>
<p>Updating passwords for all of these services at once was a major source of
frustration for users. Updating a single password can take 5 minutes, and
changing dozens of them might take hours. I decided that I wanted to make this
process easier.</p>
<pre><code>$ ./pass-rotate github.com linode.com news.ycombinator.com twitter.com
Rotating github.com... 
  Enter your two factor (TOTP) code:
OK
Rotating linode.com... 
  Enter your two-factor (TOTP) code:
OK
Rotating news.ycombinator.com... OK
Rotating twitter.com... 
  Enter your SMS authorization code:
OK                                                                       
</code></pre><p>I just changed 4 passwords in about 20 seconds. This is
<a href="https://github.com/SirCmpwn/pass-rotate">pass-rotate</a>, which is basically
youtube-dl for rotating passwords. It integrates with your password manager to
make it easy to change your password. pass-rotate is also provided in the form
of a library that password managers can directly integrate with to provide
first-class support for password rotation with a shared implementation of
various websites. Not only can it help you rotate passwords after security
events, but it can be used for periodic password rotation to keep your accounts
safer in general.</p>
<p>How this was basically done is by reverse engineering the password change flow of
each of the websites it supports. Each provider&rsquo;s backend submits HTTP requests
that simulates logging into the website and interacting with the password reset
form. This is often quite simple, like
<a href="https://github.com/SirCmpwn/pass-rotate/blob/master/passrotate/providers/github.py">github.py</a>,
but can sometimes be quite complex, like
<a href="https://github.com/SirCmpwn/pass-rotate/blob/master/passrotate/providers/namecheap.py">namecheap.py</a>.</p>
<p>The current list of supported services is available
<a href="https://github.com/SirCmpwn/pass-rotate/wiki/Currently-supported-services">here</a>.
There&rsquo;s also an issue to discuss making a standardized mechanism for automated
password rotation <a href="https://github.com/SirCmpwn/pass-rotate/issues/1">here</a>. At
the time of writing, the list of supported services is:</p>
<ul>
<li>Cloudflare <sub>✗ TOTP</sub></li>
<li>Digital Ocean <sub>✗ TOTP</sub></li>
<li>Discord <sub>✓ TOTP</sub></li>
<li>GitHub <sub>✓ TOTP ✗ U2F</sub></li>
<li>Linode <sub>✓ TOTP</sub></li>
<li>NameCheap <sub>✓ SMS</sub></li>
<li>Pixiv</li>
<li>Twitter <sub>✓ SMS ✓ TOTP</sub></li>
<li>YCombinator</li>
</ul>
<p>Adding new services is easy - check out <a href="https://github.com/SirCmpwn/pass-rotate/blob/master/CONTRIBUTING.md">the
guide</a>. I
would be happy to merge your pull requests. Please add websites you use and
websites you maintain!</p>
<p>I also set up a Patreon campaign today. If you&rsquo;d like to contribute to my work,
please visit <a href="https://patreon.com/sircmpwn">the Patreon page</a>. This supports all
of my open source projects, but if you want to support pass-rotate in
particular feel free to let me know when you make your contribution. This kind
of project needs long term maintenance to support countless providers and
keep up with changes to them. Feel free to let me know what service providers
you want me to add support for when you make your pledge!</p>


guid: https://drewdevault.com/2017/06/05/Limited-generics-in-C.html
title: Limited "generics" in C without macros or UB
url: https://drewdevault.com/2017/06/05/Limited-generics-in-C.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I should start this post off by clarifying that what I have to show you today is
not, in fact, generics. However, it&rsquo;s useful in some situations to solve the
same problems that generics might. This is a pattern I&rsquo;ve started using to
reduce the number of <code>void*</code> pointers floating around in my code: multiple
definitions of a struct.</p>
<p><strong>Errata</strong>: we rolled this approach back in wlroots because it causes problems
with LTO. I no longer recommend it.</p>
<p>Let&rsquo;s take a look at a specific example. In
<a href="https://github.com/SirCmpwn/wlroots">wlroots</a>, <code>wlr_output</code> is a generic type
that can be implemented by any number of backends, like DRM (direct rendering
manager), wayland windows, X11 windows, RDP outputs, etc. The <code>wlr/types.h</code>
header includes this structure:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">struct</span> <span class="n">wlr_output_impl</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">wlr_output_state</span><span class="p">;</span>

<span class="k">struct</span> <span class="n">wlr_output</span> <span class="p">{</span>
    <span class="k">const</span> <span class="k">struct</span> <span class="n">wlr_output_impl</span> <span class="o">*</span><span class="n">impl</span><span class="p">;</span>
    <span class="k">struct</span> <span class="n">wlr_output_state</span> <span class="o">*</span><span class="n">state</span><span class="p">;</span>
    <span class="c1">// [...]
</span><span class="c1"></span><span class="p">};</span>

<span class="kt">void</span> <span class="nf">wlr_output_enable</span><span class="p">(</span><span class="k">struct</span> <span class="n">wlr_output</span> <span class="o">*</span><span class="n">output</span><span class="p">,</span> <span class="kt">bool</span> <span class="n">enable</span><span class="p">);</span>
<span class="kt">bool</span> <span class="nf">wlr_output_set_mode</span><span class="p">(</span><span class="k">struct</span> <span class="n">wlr_output</span> <span class="o">*</span><span class="n">output</span><span class="p">,</span>
    <span class="k">struct</span> <span class="n">wlr_output_mode</span> <span class="o">*</span><span class="n">mode</span><span class="p">);</span>
<span class="kt">void</span> <span class="nf">wlr_output_destroy</span><span class="p">(</span><span class="k">struct</span> <span class="n">wlr_output</span> <span class="o">*</span><span class="n">output</span><span class="p">);</span>
</code></pre></div><p><code>wlr_output_impl</code> is defined elsewhere:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">struct</span> <span class="n">wlr_output_impl</span> <span class="p">{</span>
    <span class="kt">void</span> <span class="p">(</span><span class="o">*</span><span class="n">enable</span><span class="p">)(</span><span class="k">struct</span> <span class="n">wlr_output_state</span> <span class="o">*</span><span class="n">state</span><span class="p">,</span> <span class="kt">bool</span> <span class="n">enable</span><span class="p">);</span>
    <span class="kt">bool</span> <span class="p">(</span><span class="o">*</span><span class="n">set_mode</span><span class="p">)(</span><span class="k">struct</span> <span class="n">wlr_output_state</span> <span class="o">*</span><span class="n">state</span><span class="p">,</span>
        <span class="k">struct</span> <span class="n">wlr_output_mode</span> <span class="o">*</span><span class="n">mode</span><span class="p">);</span>
    <span class="kt">void</span> <span class="p">(</span><span class="o">*</span><span class="n">destroy</span><span class="p">)(</span><span class="k">struct</span> <span class="n">wlr_output_state</span> <span class="o">*</span><span class="n">state</span><span class="p">);</span>
<span class="p">};</span>

<span class="k">struct</span> <span class="n">wlr_output</span> <span class="o">*</span><span class="nf">wlr_output_create</span><span class="p">(</span><span class="k">struct</span> <span class="n">wlr_output_impl</span> <span class="o">*</span><span class="n">impl</span><span class="p">,</span>
        <span class="k">struct</span> <span class="n">wlr_output_state</span> <span class="o">*</span><span class="n">state</span><span class="p">);</span>
<span class="kt">void</span> <span class="nf">wlr_output_free</span><span class="p">(</span><span class="k">struct</span> <span class="n">wlr_output</span> <span class="o">*</span><span class="n">output</span><span class="p">);</span>
</code></pre></div><p>Nowhere, however, is <code>wlr_output_state</code> defined. It&rsquo;s left an incomplete type
throughout all of the common <code>wlr_output</code> code. The &ldquo;generic&rdquo; part is that each
output implementation, in its own private headers, defines the
<code>wlr_output_state</code> struct for itself, like the DRM backend:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">struct</span> <span class="n">wlr_output_state</span> <span class="p">{</span>
    <span class="n">uint32_t</span> <span class="n">connector</span><span class="p">;</span>
    <span class="kt">char</span> <span class="n">name</span><span class="p">[</span><span class="mi">16</span><span class="p">];</span>
    <span class="n">uint32_t</span> <span class="n">crtc</span><span class="p">;</span>
    <span class="n">drmModeCrtc</span> <span class="o">*</span><span class="n">old_crtc</span><span class="p">;</span>
    <span class="k">struct</span> <span class="n">wlr_drm_renderer</span> <span class="o">*</span><span class="n">renderer</span><span class="p">;</span>
    <span class="k">struct</span> <span class="n">gbm_surface</span> <span class="o">*</span><span class="n">gbm</span><span class="p">;</span>
    <span class="n">EGLSurface</span> <span class="o">*</span><span class="n">egl</span><span class="p">;</span>
    <span class="kt">bool</span> <span class="n">pageflip_pending</span><span class="p">;</span>
    <span class="k">enum</span> <span class="n">wlr_drm_output_state</span> <span class="n">state</span><span class="p">;</span>
    <span class="c1">// [...]
</span><span class="c1"></span><span class="p">};</span>
</code></pre></div><p>This allows implementations of the <code>enable</code>, <code>set_mode</code>, and <code>destroy</code> functions
to avoid casting a <code>void*</code> to the appropriate type:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">static</span> <span class="k">struct</span> <span class="n">wlr_output_impl</span> <span class="n">output_impl</span> <span class="o">=</span> <span class="p">{</span>
    <span class="p">.</span><span class="n">enable</span> <span class="o">=</span> <span class="n">wlr_drm_output_enable</span><span class="p">,</span>
    <span class="c1">// [...]
</span><span class="c1"></span><span class="p">};</span>

<span class="k">static</span> <span class="kt">void</span> <span class="nf">wlr_drm_output_enable</span><span class="p">(</span><span class="k">struct</span> <span class="n">wlr_output_state</span> <span class="o">*</span><span class="n">output</span><span class="p">,</span>
        <span class="kt">bool</span> <span class="n">enable</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">struct</span> <span class="n">wlr_backend_state</span> <span class="o">*</span><span class="n">state</span> <span class="o">=</span>
        <span class="n">wl_container_of</span><span class="p">(</span><span class="n">output</span><span class="o">-&gt;</span><span class="n">renderer</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">renderer</span><span class="p">);</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">output</span><span class="o">-&gt;</span><span class="n">state</span> <span class="o">!=</span> <span class="n">DRM_OUTPUT_CONNECTED</span><span class="p">)</span> <span class="p">{</span>
        <span class="k">return</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">enable</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">drmModeConnectorSetProperty</span><span class="p">(</span><span class="n">state</span><span class="o">-&gt;</span><span class="n">fd</span><span class="p">,</span>
            <span class="n">output</span><span class="o">-&gt;</span><span class="n">connector</span><span class="p">,</span>
            <span class="n">output</span><span class="o">-&gt;</span><span class="n">props</span><span class="p">.</span><span class="n">dpms</span><span class="p">,</span>
            <span class="n">DRM_MODE_DPMS_ON</span><span class="p">);</span>
        <span class="c1">// [...]
</span><span class="c1"></span>    <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
        <span class="n">drmModeConnectorSetProperty</span><span class="p">(</span><span class="n">state</span><span class="o">-&gt;</span><span class="n">fd</span><span class="p">,</span>
            <span class="n">output</span><span class="o">-&gt;</span><span class="n">connector</span><span class="p">,</span>
            <span class="n">output</span><span class="o">-&gt;</span><span class="n">props</span><span class="p">.</span><span class="n">dpms</span><span class="p">,</span>
            <span class="n">DRM_MODE_DPMS_STANDBY</span><span class="p">);</span>
    <span class="p">}</span>
<span class="p">}</span>

<span class="c1">// [...]
</span><span class="c1"></span><span class="k">struct</span> <span class="n">wlr_output</span> <span class="n">output</span> <span class="o">=</span> <span class="n">wlr_output_create</span><span class="p">(</span><span class="o">&amp;</span><span class="n">output_impl</span><span class="p">,</span> <span class="n">output</span><span class="p">);</span>
</code></pre></div><p>The limitations of this approach are apparent: you cannot work with multiple
definitions of <code>wlr_output_state</code> in the same file. However, you get improved
type safety, have to write less code, and improve readability.</p>


guid: https://drewdevault.com/2017/06/10/Introduction-to-Wayland.html
title: An introduction to Wayland
url: https://drewdevault.com/2017/06/10/Introduction-to-Wayland.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Wayland is the new hotness on the Linux graphics stack. There are plenty of
introductions to Wayland that give you the high level details on how the stack
is laid out how applications talk directly to the kernel with EGL and so on, but
that doesn&rsquo;t give you much practical knowledge. I&rsquo;d like to instead share with
you details about how the protocol actually works and how you can use it.</p>
<p>Let&rsquo;s set aside the idea that Wayland has anything to do with graphics. Instead
we&rsquo;ll treat it like a generic protocol for two parties to share and talk about
resources. These resources are at the heart of the Wayland protocol - resources
like a keyboard or a surface to draw on. Each of these resources exposes an API
for engaging with it, including functions you can call and <em>events</em> you can
listen to.</p>
<p>Some of these resources are <em>globals</em>, which are exactly what they sound like.
These resources include things like <strong>wl_outputs</strong>, which are the displays
connected to your graphics card. Other resources, like <strong>wl_surface</strong>, require
the client to ask the server to allocate new resources when needed. Negotiating
for new resources is generally possible through the API of some global resource.</p>
<p>Your Wayland client gets started by obtaining a reference to the <strong>wl_display</strong>
like so:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">struct</span> <span class="n">wl_display</span> <span class="o">*</span><span class="n">display</span> <span class="o">=</span> <span class="n">wl_display_connect</span><span class="p">(</span><span class="nb">NULL</span><span class="p">);</span>
</code></pre></div><p>This establishes a connection to the Wayland server. The most important role of
the display, from the client perspective, is to provide the <strong>wl_registry</strong>.
The registry enumerates the globals available on the server.</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">struct</span> <span class="n">wl_registry</span> <span class="o">*</span><span class="n">registry</span> <span class="o">=</span> <span class="n">wl_display_get_registry</span><span class="p">(</span><span class="n">display</span><span class="p">);</span>
</code></pre></div><p>The registry emits an <em>event</em> every time the server adds or removes a global.
<em>Listening</em> to these events is done by providing an implementation of a
<strong>wl_registry_listener</strong>, like so:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">void</span> <span class="nf">global_add</span><span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="n">our_data</span><span class="p">,</span>
        <span class="k">struct</span> <span class="n">wl_registry</span> <span class="o">*</span><span class="n">registry</span><span class="p">,</span>
        <span class="n">uint32_t</span> <span class="n">name</span><span class="p">,</span>
        <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">interface</span><span class="p">,</span>
        <span class="n">uint32_t</span> <span class="n">version</span><span class="p">)</span> <span class="p">{</span>
    <span class="c1">// TODO
</span><span class="c1"></span><span class="p">}</span>

<span class="kt">void</span> <span class="nf">global_remove</span><span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="n">our_data</span><span class="p">,</span>
        <span class="k">struct</span> <span class="n">wl_registry</span> <span class="o">*</span><span class="n">registry</span><span class="p">,</span>
        <span class="n">uint32_t</span> <span class="n">name</span><span class="p">)</span> <span class="p">{</span>
    <span class="c1">// TODO
</span><span class="c1"></span><span class="p">}</span>

<span class="k">struct</span> <span class="n">wl_registry_listener</span> <span class="n">registry_listener</span> <span class="o">=</span> <span class="p">{</span>
    <span class="p">.</span><span class="n">global</span> <span class="o">=</span> <span class="n">global_add</span><span class="p">,</span>
    <span class="p">.</span><span class="n">global_remove</span> <span class="o">=</span> <span class="n">global_remove</span>
<span class="p">};</span>
</code></pre></div><p>Interfaces like this are used to listen to events from all kinds of resources.
Attaching the listener to the registry is done like this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">void</span> <span class="o">*</span><span class="n">our_data</span> <span class="o">=</span> <span class="cm">/* arbitrary state you want to keep around */</span><span class="p">;</span>
<span class="n">wl_registry_add_listener</span><span class="p">(</span><span class="n">registry</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">registry_listener</span><span class="p">,</span> <span class="n">our_data</span><span class="p">);</span>
<span class="n">wl_display_dispatch</span><span class="p">(</span><span class="n">display</span><span class="p">);</span>
</code></pre></div><p>During the <code>wl_display_dispatch</code>, the <code>global_add</code> function is called for each
global on the server. Subsequent calls to <code>wl_display_dispatch</code> may call
<code>global_remove</code> when the server destroys globals. The <code>name</code> passed into
<code>global_add</code> is more like an ID, and identifies this resource. The <code>interface</code>
tells you what API the resource implements, and distinguishes things like a
<strong>wl_output</strong> from a <strong>wl_seat</strong>. The API these resources implement are
described with XML files like this:</p>
<div class="highlight"><pre class="chroma"><code class="language-xml" data-lang="xml"><span class="cp">&lt;?xml version=&#34;1.0&#34; encoding=&#34;UTF-8&#34;?&gt;</span>
<span class="c">&lt;!-- For copyright information, see https://git.io/vHyIB --&gt;</span>
<span class="nt">&lt;protocol</span> <span class="na">name=</span><span class="s">&#34;gamma_control&#34;</span><span class="nt">&gt;</span>
    <span class="nt">&lt;interface</span> <span class="na">name=</span><span class="s">&#34;gamma_control_manager&#34;</span> <span class="na">version=</span><span class="s">&#34;1&#34;</span><span class="nt">&gt;</span>
        <span class="nt">&lt;request</span> <span class="na">name=</span><span class="s">&#34;destroy&#34;</span> <span class="na">type=</span><span class="s">&#34;destructor&#34;</span><span class="nt">/&gt;</span>

        <span class="nt">&lt;request</span> <span class="na">name=</span><span class="s">&#34;get_gamma_control&#34;</span><span class="nt">&gt;</span>
            <span class="nt">&lt;arg</span> <span class="na">name=</span><span class="s">&#34;id&#34;</span> <span class="na">type=</span><span class="s">&#34;new_id&#34;</span> <span class="na">interface=</span><span class="s">&#34;gamma_control&#34;</span><span class="nt">/&gt;</span>
            <span class="nt">&lt;arg</span> <span class="na">name=</span><span class="s">&#34;output&#34;</span> <span class="na">type=</span><span class="s">&#34;object&#34;</span> <span class="na">interface=</span><span class="s">&#34;wl_output&#34;</span><span class="nt">/&gt;</span>
        <span class="nt">&lt;/request&gt;</span>
    <span class="nt">&lt;/interface&gt;</span>

    <span class="nt">&lt;interface</span> <span class="na">name=</span><span class="s">&#34;gamma_control&#34;</span> <span class="na">version=</span><span class="s">&#34;1&#34;</span><span class="nt">&gt;</span>
        <span class="nt">&lt;enum</span> <span class="na">name=</span><span class="s">&#34;error&#34;</span><span class="nt">&gt;</span>
            <span class="nt">&lt;entry</span> <span class="na">name=</span><span class="s">&#34;invalid_gamma&#34;</span> <span class="na">value=</span><span class="s">&#34;0&#34;</span><span class="nt">/&gt;</span>
        <span class="nt">&lt;/enum&gt;</span>

        <span class="nt">&lt;request</span> <span class="na">name=</span><span class="s">&#34;destroy&#34;</span> <span class="na">type=</span><span class="s">&#34;destructor&#34;</span><span class="nt">/&gt;</span>

        <span class="nt">&lt;request</span> <span class="na">name=</span><span class="s">&#34;set_gamma&#34;</span><span class="nt">&gt;</span>
            <span class="nt">&lt;arg</span> <span class="na">name=</span><span class="s">&#34;red&#34;</span> <span class="na">type=</span><span class="s">&#34;array&#34;</span><span class="nt">/&gt;</span>
            <span class="nt">&lt;arg</span> <span class="na">name=</span><span class="s">&#34;green&#34;</span> <span class="na">type=</span><span class="s">&#34;array&#34;</span><span class="nt">/&gt;</span>
            <span class="nt">&lt;arg</span> <span class="na">name=</span><span class="s">&#34;blue&#34;</span> <span class="na">type=</span><span class="s">&#34;array&#34;</span><span class="nt">/&gt;</span>
        <span class="nt">&lt;/request&gt;</span>

        <span class="nt">&lt;request</span> <span class="na">name=</span><span class="s">&#34;reset_gamma&#34;</span><span class="nt">/&gt;</span>

        <span class="nt">&lt;event</span> <span class="na">name=</span><span class="s">&#34;gamma_size&#34;</span><span class="nt">&gt;</span>
            <span class="nt">&lt;arg</span> <span class="na">name=</span><span class="s">&#34;size&#34;</span> <span class="na">type=</span><span class="s">&#34;uint&#34;</span><span class="nt">/&gt;</span>
        <span class="nt">&lt;/event&gt;</span>
    <span class="nt">&lt;/interface&gt;</span>
<span class="nt">&lt;/protocol&gt;</span>
</code></pre></div><p>A typical Wayland server implementing this protocol would create a
<code>gamma_control_manager</code> global and add it to the registry. The client then binds
to this interface in our <code>global_add</code> function like so:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="cp">#include</span> <span class="cpf">&#34;wayland-gamma-control-client-protocol.h&#34;</span><span class="cp">
</span><span class="cp"></span><span class="c1">// ...
</span><span class="c1"></span><span class="k">struct</span> <span class="n">wl_output</span> <span class="o">*</span><span class="n">example</span><span class="p">;</span>
<span class="c1">// gamma_control_manager.name is a constant: &#34;gamma_control_manager&#34;
</span><span class="c1"></span><span class="k">if</span> <span class="p">(</span><span class="n">strcmp</span><span class="p">(</span><span class="n">interface</span><span class="p">,</span> <span class="n">gamma_control_manager</span><span class="p">.</span><span class="n">name</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">struct</span> <span class="n">gamma_control_manager</span> <span class="o">*</span><span class="n">mgr</span> <span class="o">=</span>
        <span class="n">wl_registry_bind</span><span class="p">(</span><span class="n">registry</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span>
            <span class="o">&amp;</span><span class="n">gamma_control_manager_interface</span><span class="p">,</span> <span class="n">version</span><span class="p">);</span>
    <span class="k">struct</span> <span class="n">gamma_control</span> <span class="o">*</span><span class="n">control</span> <span class="o">=</span>
        <span class="n">gamma_control_manager_get_gamma_control</span><span class="p">(</span><span class="n">mgr</span><span class="p">,</span> <span class="n">example</span><span class="p">);</span>
    <span class="n">gamma_control_set_gamma</span><span class="p">(</span><span class="n">control</span><span class="p">,</span> <span class="p">...);</span>
<span class="p">}</span>
</code></pre></div><p>These functions are generated by running the XML file through <code>wayland-scanner</code>,
which outputs a header and C glue code. These XML files are called &ldquo;protocol
extensions&rdquo; and let you add arbitrary extensions to the protocol. The core
Wayland protocols themselves are described with similar XML files.</p>
<p>Using the Wayland protocol to create a surface to display pixels with consists
of these steps:</p>
<ol>
<li>Obtain a <strong>wl_display</strong> and use it to obtain a <strong>wl_registry</strong>.</li>
<li>Scan the registry for globals and grab a <strong>wl_compositor</strong> and
a <strong>wl_shm_pool</strong>.</li>
<li>Use the <strong>wl_compositor</strong> interface to create a <strong>wl_surface</strong>.</li>
<li>Use the <strong>wl_shell</strong> interface to describe your surface&rsquo;s role.</li>
<li>Use the <strong>wl_shm</strong> interface to allocate shared memory to store pixels in.</li>
<li>Draw something into your shared memory buffers.</li>
<li>Attach your shared memory buffers to the <strong>wl_surface</strong>.</li>
</ol>
<p>Let&rsquo;s break this down.</p>
<p>The <strong>wl_compositor</strong> provides an interface for interacting with the
<em>compositor</em>, that is the part of the Wayland server that <em>composites</em> surfaces
onto the screen. It&rsquo;s responsible for creating surface resources for clients to
use via <code>wl_compositor_create_surface</code>. This creates a <strong>wl_surface</strong> resource,
which you can attach pixels to for the compositor to render.</p>
<p>The role of a surface is undefined by default - it&rsquo;s just a place to put pixels.
In order to get the compositor to do anything with them, you must give the
surface a <em>role</em>. Roles could be anything - desktop background, system tray, etc -
but the most common role is a <em>shell surface</em>. To create these, you take your
wl_surface and hand it to the <strong>wl_shell</strong> interface. You&rsquo;ll get back a
<strong>wl_shell_surface</strong> resource, which defines your surface&rsquo;s purpose and gives
you an interface to do things like set the window title.</p>
<p>Attaching pixel buffers to a wl_surface is pretty straightforward. There are two
primary ways of creating a buffer that both you and the compositor can use: EGL
and shared memory. EGL lets you use an OpenGL context that renders directly on
the GPU with minimal compositor involvement (fast) and shared memory (via
<strong>wl_shm</strong>) allows you to simply dump pixels in memory and hand them to the
compositor (flexible). There are many other Wayland interfaces I haven&rsquo;t
covered, giving you everything from input devices (via <strong>wl_seat</strong>) to clipboard
access (via <strong>wl_data_source</strong>), plus many protocol extensions. Learning more
about these is an exercise left to the reader.</p>
<p>Before we wrap this article up, let&rsquo;s take a brief moment to discuss the server.
Most of the concepts here are already familiar to you by now. The Wayland server
also utilizes a <strong>wl_display</strong>, but differently from the client. The display on
the server has ownership over the <em>event loop</em>, via <strong>wl_event_loop</strong>. The event
loop of a Wayland server might look like this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">struct</span> <span class="n">wl_display</span> <span class="o">*</span><span class="n">display</span> <span class="o">=</span> <span class="n">wl_display_create</span><span class="p">();</span>
<span class="c1">// ...
</span><span class="c1"></span><span class="k">struct</span> <span class="n">wl_event_loop</span> <span class="o">*</span><span class="n">event_loop</span> <span class="o">=</span> <span class="n">wl_display_get_event_loop</span><span class="p">(</span><span class="n">display</span><span class="p">);</span>
<span class="k">while</span> <span class="p">(</span><span class="nb">true</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">wl_event_loop_dispatch</span><span class="p">(</span><span class="n">event_loop</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div><p>The event loop has a lot of helpful utilities for the Wayland server to take
advantage of, including internal event sources, timers, and file descriptor
monitoring. Before starting the event loop the server is going to start
obtaining its own resources and creating Wayland globals for them with
<code>wl_global_create</code>:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="k">struct</span> <span class="n">wl_global</span> <span class="o">*</span><span class="n">global</span> <span class="o">=</span> <span class="n">wl_global_create</span><span class="p">(</span>
    <span class="n">display</span><span class="p">,</span>
    <span class="o">&amp;</span><span class="n">wl_output_interface</span><span class="p">,</span>
    <span class="mi">1</span> <span class="cm">/* version */</span><span class="p">,</span>
    <span class="n">our_data</span><span class="p">,</span>
    <span class="n">wl_output_bind</span><span class="p">);</span>
</code></pre></div><p>The <code>wl_output_bind</code> function here is going to be called when a client attempts
to bind to this resource via <code>wl_registry_bind</code>, and will look something like
this:</p>
<div class="highlight"><pre class="chroma"><code class="language-c" data-lang="c"><span class="kt">void</span> <span class="nf">wl_output_bind</span><span class="p">(</span><span class="k">struct</span> <span class="n">wl_client</span> <span class="o">*</span><span class="n">client</span><span class="p">,</span>
        <span class="kt">void</span> <span class="o">*</span><span class="n">our_data</span><span class="p">,</span>
        <span class="n">uint32_t</span> <span class="n">their_version</span><span class="p">,</span>
        <span class="n">uint32_t</span> <span class="n">id</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">struct</span> <span class="n">wl_resource</span> <span class="o">*</span><span class="n">resource</span> <span class="o">=</span>
        <span class="n">wl_resource_create_checked</span><span class="p">(</span>
            <span class="n">client</span><span class="p">,</span>
            <span class="n">wl_output_interface</span><span class="p">,</span>
            <span class="n">their_version</span><span class="p">,</span>
            <span class="n">our_version</span><span class="p">,</span>
            <span class="n">id</span><span class="p">);</span>
    <span class="c1">// ...send output modes or whatever else you need to do
</span><span class="c1"></span><span class="p">}</span>
</code></pre></div><p>Some of the resources a server is going to be managing might include:</p>
<ul>
<li>DRM state for direct access to outputs</li>
<li>GLES context (or another GL implementation) for rendering</li>
<li>libinput for input devices</li>
<li>udev for hotplugging</li>
</ul>
<p>Through the Wayland protocol, the server provides an abstraction on top of these
resources and offers them to clients. Some servers go further, with novel ways
of compositing clients or handling input. Some provide additional interactivity,
such as desktop shells that are actually running in the compositor rather than
external clients. Other servers are designed for mobile use and provide a user
experience that more closely matches the mobile experience than the traditional
desktop experience. Wayland is designed to be flexible!</p>


guid: https://drewdevault.com/2017/06/19/Archive-it-or-miss-it.html
title: Archive it or you will miss it
url: https://drewdevault.com/2017/06/19/Archive-it-or-miss-it.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Let&rsquo;s open with some quotes from the <a href="https://en.wikipedia.org/wiki/Link_rot">Wikipedia article on link
rot</a>:</p>
<blockquote>
<p>In 2014, bookmarking site Pinboard&rsquo;s owner Maciej Cegłowski reported a “pretty
steady rate” of 5% link rot per year&hellip; approximately 50% of the URLs in
U.S. Supreme Court opinions no longer link to the original information&hellip;
(analysis of) more than 180,000 links from references in&hellip; three major open
access publishers&hellip; found that overall 24.5% of links cited were no longer
available.</p>
</blockquote>
<p>I hate link rot. It&rsquo;s been common when servers disappeared or domains expired,
in the past and still today. Today, link rot is on the rise under the influence
of more sinister factors. Abuse of DMCA. Region locking. Paywalls.  Maybe it
just no longer serves the interests of a walled garden to host the content.
Maybe the walled garden went out of business. Users rely on platforms to host
content and links rot by the millions when the platforms die. Movies disappear
from Netflix.  Music vanishes from Spotify. Accounts are banned from SoundCloud.
YouTube channels are banned over false DMCA requests issued by robots.</p>
<p>At this point, link rot is an axiom of the internet. In the face of this, I
store a personal offline archive of <em>anything</em> I want to see twice. When I see a
cool YouTube video I like, I archive the entire channel right away. Rather than
subscribe to it, I update my archive on a cronjob. I scrape content out of RSS
feeds and into offline storage and I have dozens of websites archived with wget.
I mirror most git repositories I&rsquo;m interested in. I have DRM free offline copies
of all of my music, TV shows, and movies, ill-begotten or not.</p>
<p>I suggest you do the same. It&rsquo;s sad that it&rsquo;s come to this. Let&rsquo;s all do
ourselves a favor. Don&rsquo;t build unsustainable platforms and ask users to trust
you with their data. Pay for your domain. Give people DRM free downloads. Don&rsquo;t
cripple your software when it can&rsquo;t call home. If you run a website, let
archive.org scrape it.</p>
<p>And archive anything you want to see again.</p>
<pre><code>0 0 * * 0 cd ~/archives &amp;&amp; wget -m https://drewdevault.com
</code></pre>


guid: https://drewdevault.com/2017/08/09/State-of-Sway-August-2017.html
title: State of Sway August 2017
url: https://drewdevault.com/2017/08/09/State-of-Sway-August-2017.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Is it already time to write another one of these? Phew, time flies. Sway marches
ever forward. Sway 0.14.0 was recently released, adding much asked-after support
for tray icons and fixing some long-standing bugs. As usual, we already have
some exciting features slated for 0.15.0 as well, notably some cool improvements
to clipboard support. Look forward to it!</p>
<p>Today Sway has 24,123 lines of C (and 4,489 lines of header files) written by 94
authors across 2,345 commits. These were written through 689 pull requests and
624 issues. Sway packages are available today in the repos of almost every Linux
distribution.</p>
<p><a href="https://sr.ht/ICd5.png"><img src="https://sr.ht/ICd5.png" alt=""></a></p>
<p>For those who are new to the project, <a href="http://swaywm.org">Sway</a> is an
i3-compatible Wayland compositor. That is, your existing <a href="http://i3wm.org/">i3</a>
configuration file will work as-is on Sway, and your keybindings and colors and
fonts and for_window rules and so on will all be the same. It&rsquo;s i3, but for
Wayland, plus it&rsquo;s got some bonus features. Here&rsquo;s a quick rundown of what&rsquo;s
new since the <a href="https://drewdevault.com/2017/04/29/State-of-sway-April-2017.html">previous state of Sway</a>:</p>
<ul>
<li>Initial support for tray icons</li>
<li>X11/Wayland clipboard synchronization</li>
<li>nvidia proprietary driver support*</li>
<li>i3&rsquo;s marks</li>
<li>i3&rsquo;s mouse button bindings</li>
<li>Lots of i3 compatibility improvements</li>
<li>Lots of documentation improvements</li>
<li>Lots of bugfixes</li>
</ul>
<p>If this seems like a shorter list than usual, it&rsquo;s because we&rsquo;ve also been
making great progress on wlroots too - no doubt thanks to the help of the many
contributors doing amazing work in there. For those unaware, wlroots is our
project to replace wlc with a new set of libraries for Wayland compositor
underpinnings (it fills a similar niche as libweston). We now have a working DRM
backend (including output rotation and hardware cursors) and libinput backend
(including touchscreen and drawing tablet support), and we&rsquo;re making headway now
on drawing Wayland clients on screen.  I&rsquo;m very excited about our pace and
direction - keep an eye on it
<a href="https://github.com/SirCmpwn/wlroots/issues/9">here</a>. I have also taken over for
Cloudef as the maintainer of <a href="https://github.com/Cloudef/wlc">wlc</a> during the
transition.</p>
<p>In other news, our bounty program continues to go strong. Our <a href="https://github.com/SirCmpwn/sway/issues/986">current
pot</a> is $1200 and we&rsquo;ve paid out
$80 so far (and a $280 payout is on the horizon for tray icons). I&rsquo;ve also
started a <a href="https://www.patreon.com/sircmpwn">Patreon page</a>, where 26 patrons are
generously supporting my work as maintainer of Sway and other projects. Many
thanks to everyone who has contributed financially to Sway&rsquo;s success!</p>
<p>That wraps up today&rsquo;s post. Thanks for using Sway!</p>
<p><small class="text-muted">* I hate this crappy driver. It works, but don&rsquo;t
expect to receive much support for it. <a
href="https://www.youtube.com/watch?v=iYWzMvlj2RQ">Linus said it
best</a>.</small></p>


guid: https://drewdevault.com/2017/08/13/When-not-to-use-a-regex.html
title: When not to use a regex
url: https://drewdevault.com/2017/08/13/When-not-to-use-a-regex.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>The other day, I saw <a href="https://github.com/zeeshanu/learn-regex">Learn regex the easy
way</a>. This is a great resource, but I
felt the need to pen a post explaining that regexes are usually not the right
approach.</p>
<p>Let&rsquo;s do a little exercise. I googled &ldquo;URL regex&rdquo; and here&rsquo;s the first Stack
Overflow result:</p>
<pre><code>https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&amp;//=]*)
</code></pre><p style="text-align: right">
<small><a href="https://stackoverflow.com/a/3809435/1191610">source</a></small>
</p>
<p>This is a bad regex. Here are some valid URLs that this regex fails to match:</p>
<ul>
<li>http://x.org</li>
<li><a href="http://nic.science">http://nic.science</a></li>
<li>http://名がドメイン.com (warning: this is a parked domain)</li>
<li><a href="http://example.org/url,with,commas">http://example.org/url,with,commas</a></li>
<li><a href="https://en.wikipedia.org/wiki/Harry_Potter_(film_series)">https://en.wikipedia.org/wiki/Harry_Potter_(film_series)</a></li>
<li>http://127.0.0.1</li>
<li>http://[::1] (ipv6 loopback)</li>
</ul>
<p>Here are some invalid URLs the regex is fine with:</p>
<ul>
<li><a href="http://exam..ple.org">http://exam..ple.org</a></li>
<li><a href="http://--example.org">http://--example.org</a></li>
</ul>
<p>This answer has been revised 9 times on Stack Overflow, and this is the best
they could come up with. Go back and read the regex. Can you tell where each of
these bugs are? How long did it take you? If you received a bug report in your
application because one of these URLs was handled incorrectly, do you understand
this regex well enough to fix it? If your application has a URL regex, go find
it and see how it fares with these tests.</p>
<p>Complicated regexes are opaque, unmaintainable, and often wrong. The correct
approach to validating a URL is as follows:</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="kn">import</span> <span class="n">urlparse</span>

<span class="k">def</span> <span class="nf">is_url_valid</span><span class="p">(</span><span class="n">url</span><span class="p">):</span>
    <span class="k">try</span><span class="p">:</span>
        <span class="n">urlparse</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
        <span class="k">return</span> <span class="bp">True</span>
    <span class="k">except</span><span class="p">:</span>
        <span class="k">return</span> <span class="bp">False</span>
</code></pre></div><p>A regex is useful for validating <em>simple</em> patterns and for <em>finding</em> patterns in
text. For anything beyond that it&rsquo;s almost certainly a terrible choice. Say you
want to&hellip;</p>
<p><strong>validate an email address</strong>: try to send an email to it!</p>
<p><strong>validate password strength requirements</strong>: estimate the complexity with
<a href="https://github.com/dropbox/zxcvbn">zxcvbn</a>!</p>
<p><strong>validate a date</strong>: use your standard library!
<a href="https://docs.python.org/3.6/library/datetime.html#datetime.datetime.strptime">datetime.datetime.strptime</a></p>
<p><strong>validate a credit card number</strong>: run the <a href="https://en.wikipedia.org/wiki/Luhn_algorithm">Luhn
algorithm</a> on it!</p>
<p><strong>validate a social security number</strong>: alright, use a regex. But don&rsquo;t expect
the number to be assigned to someone until you ask the Social Security
Administration about it!</p>
<p>Get the picture?</p>


guid: https://drewdevault.com/2017/09/08/Complicated.html
title: Killing ants with nuclear weapons
url: https://drewdevault.com/2017/09/08/Complicated.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Complexity is quickly becoming an epidemic. In this developer&rsquo;s opinion,
complexity is the ultimate enemy - the final boss - of good software design.
Complicated software generally has complicated bugs. Simple software generally
has simple bugs. It&rsquo;s as easy as that.</p>
<p>It&rsquo;s for this reason that I strongly dislike many of the tools and architectures
that have been proliferating over the past few years, particularly in web
development. When I look at a tool like Gulp, I wonder if its success is largely
attributable to people not bothering to learn how Makefiles work. Tools like
Docker make me wonder if they&rsquo;re an excuse to avoid learning how to do ops or
how to use your distribution&rsquo;s package manager. Chef makes me wonder if its
users forgot that shell scripts can use SSH, too.</p>
<p>These tools offer a value add. But how does it compare to the cost of the
additional complexity? In my opinion, in <em>every case</em><sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> the value add is far
outweighed by the massive complexity cost. This complexity cost shows itself
when the system breaks (and it will - all systems break) and you have to dive
into these overengineered tools. Don&rsquo;t forget that dependencies are fallible,
and never add a dependency you wouldn&rsquo;t feel comfortable debugging. The time
spent learning these complicated systems to fix the inevitable bugs is surely
much less than the time spent learning the venerable tools that fill the same
niche (or, in many cases, accepting that you don&rsquo;t even need this particular
shiny thing).</p>
<p>Reinventing the wheel is a favorite pastime of mine. There are many such wheels
that I have reinvented or am currently reinventing. The problem isn&rsquo;t in
reinventing the wheel - it&rsquo;s in doing so before you actually understand the
wheel<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>. I wonder if many of these complicated tools are written by people who
set out before they fully understood what they were replacing, and I&rsquo;m <em>certain</em>
that they&rsquo;re mostly used by such people. I understand it may seem intimidating
to learn venerable tools like make(1) or chroot(1), but they&rsquo;re just a short man
page away<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>.</p>
<p>It&rsquo;s not just tools, though. I couldn&rsquo;t explain the features of C++ in fewer than
several thousand words (same goes for Rust). GNU continues to add proprietary
extensions and unnecessary features to everything they work on. Every update
shipped to your phone is making it slower to ensure you&rsquo;ll buy the new one.
Desktop applications are shipping entire web browsers into your disk and your
RAM; server applications ship entire operating systems in glorified chroots; and
hundreds of megabytes of JavaScript, ads, and spyware are shoved down the pipe
on every web page you visit.</p>
<p>This is an epidemic. It&rsquo;s time we cut this shit out. Please, design your systems
with simplicity in mind. Moore&rsquo;s law is running out<sup id="fnref:4"><a href="#fn:4" class="footnote-ref" role="doc-noteref">4</a></sup>, the free lunch is
coming to an end. We have heaps and heaps of complicated, fragile abstractions
to dismantle.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>That I&rsquo;ve seen (or heard of) <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>&ldquo;Those who don&rsquo;t understand UNIX are doomed to reinvent it, poorly.&rdquo; <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Of course, <a href="https://xkcd.com/912/"><em>&quot;&hellip;full documentation for make is maintained as a GNU info page&hellip;&quot;</em></a> <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>Transistors are approaching a scale where quantum problems come into play, and we are limited by the speed of light without getting any smaller. The RAM bottleneck is another serious issue, for which innovation has been stagnant for some time now. <a href="#fnref:4" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2017/09/13/Analyzing-HN.html
title: Analyzing HN moderation & censorship
url: https://drewdevault.com/2017/09/13/Analyzing-HN.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p><a href="https://news.ycombinator.com">Hacker News</a> is a popular
&ldquo;<a href="http://www.catb.org/jargon/html/H/hacker.html">hacker</a>&rdquo; news board. One thing
I love about HN is that the moderation generally does an excellent job. The site
is free of spam and the conversations are usually respectful and meaningful (if
pessimistic at times). However, there is always room for improvement, and
moderation on Hacker News is no exception.</p>
<p><strong>Notice</strong>: on 2017-10-19 this article was updated to incorporate feedback the
Hacker News moderators sent to me to clarify some of the points herein. You may
view a diff of these changes
<a href="https://github.com/SirCmpwn/sircmpwn.github.io/commit/553d051c84a4631c3bd3264a437dfbc6c9807d13">here</a>.</p>
<p>For some time now, I&rsquo;ve been scraping the HN API and website to learn how the
moderators work, and to gather some interesting statistics about posts there
in general. Every 5 minutes, I take a sample of the front page, and every 30
minutes, I sample the top 500 posts (note that HN may return fewer than this
number). During each sample, I record the ID, author, title, URL, status
(dead/flagged/dupe/alive), score, number of comments, rank, and compute the rank
based on <a href="https://news.ycombinator.com/item?id=231209">HN&rsquo;s published algorithm</a>.
A note is made when the title, URL, or status changes.</p>
<p><a href="https://hn.0x2237.club/post/15217697"><img src="https://sr.ht/IFCA.png" alt=""></a></p>
<p>The information gathered is publicly available at
<a href="https://hn.0x2237.club">hn.0x2237.club</a> (sorry about the stupid domain, I just
picked one at random). You can search for most posts here going back to
2017-04-14, as well as view recent
<a href="https://hn.0x2237.club/title-changes">title</a> and
<a href="https://hn.0x2237.club/url-changes">url</a> changes or <a href="https://hn.0x2237.club/deleted">deleted
posts</a>
(<a href="https://hn.0x2237.club/deleted-10">score&gt;10</a>). Raw data is available as JSON
for any post at <code>https://hn.0x2237.club/post/:id/json</code>. Feel free to explore the
site later, or <a href="https://git.sr.ht/~sircmpwn/hnstats">its shitty code</a>. For now,
let&rsquo;s dive into what I&rsquo;ve learned from this data.</p>
<h3 id="tools-hn-mods-use">Tools HN mods use</h3>
<p>The main tools I&rsquo;m aware of that HN moderators can use to perform their duties
are:</p>
<ul>
<li>Editing link titles or URLs</li>
<li>Influencing story rank via &ldquo;downweighting&rdquo; or &ldquo;burying&rdquo;</li>
<li>Deleting or &ldquo;killing&rdquo; posts</li>
<li>Detaching off-topic or rulebreaking comment threads from their parents</li>
<li><abbr title="Banning them without making it known to them">Shadowbanning</abbr>
misbehaving users</li>
<li>Banning misbehaving users (and telling them)</li>
</ul>
<p>The moderators emphasize a difference between deleting a post and killing a
post. The former, deleting a post, will remove it from all public view like it
had never existed, and is a tool used infrequently. Killing a post will mark it
as [dead] so it doesn&rsquo;t show up on the post listing.</p>
<p>Influencing a post&rsquo;s rank can also be done through several means of varying
severity. &ldquo;Burying&rdquo; a post will leave a post alive, but plunge it in rank.
&ldquo;Downweighting&rdquo; is similar, but does not push its rank as far.</p>
<p>There are also automated tools for detecting spam and <abbr title="Posts
influenced by a group of early voters hoping to get it on the front page">voting
rings</abbr>, as well as automated de-emphasizing of posts based on certain
<abbr title="'Bitcoin' was known to at some point be one of these">secret
keywords</abbr> and controls to prevent flamewars. Automated tools on Hacker
News are used to downweight or kill posts, but never to bury or delete them.
Dan spoke about these tools and their usage for me:</p>
<blockquote>
<p>Of these four interventions (deleting, killing, burying, and downweighting),
the only one that moderators do frequently is downweighting. We downweight
posts in response to things that go against the site guidelines, such as when a
submission is unsubstantive, baity or sensational. Typically such posts remain
on the front page, just at a lower rank. We bury posts when they&rsquo;re dupes,
but rarely otherwise. We kill posts when they&rsquo;re spam, but rarely
otherwise. [&hellip;] We never delete a post unless the author asks us to.</p>
</blockquote>
<p>Dan also further clarified the difference between dead and deleted for me:</p>
<blockquote>
<p>The distinction between &lsquo;dead&rsquo; and &lsquo;deleted&rsquo; is important. Dead posts
are different from deleted ones in that people can still see them if
they set &lsquo;showdead&rsquo; to &lsquo;yes&rsquo; in their profile. That way, users who
want a less moderated view can still see everything that has been
killed by moderators or software or user flags. Deleted posts, on the
other hand, are erased from the record and never seen again. On HN,
authors can delete their own posts for a couple hours (unless they are
comments that have replies). After that, if they want a post deleted
they can ask us and we usually are happy to oblige.</p>
</blockquote>
<p>Moderators can also artificially influence rank upwards - one way is by inviting
the user to re-submit a post that they want to give another shot at the front
page. This gives the post a healthy upvote to begin with and prevents it from
being flagged. The moderators invited me to re-submit this very article using
this mechanism on 2017-10-19.</p>
<p>Banning users is another mechanism that they can use. There are two ways bans
are typically applied around the net - telling users they&rsquo;ve been banned, and
keeping it quiet. The latter - shadowbanning - is a useful tool against spammers
and serial ban evaders who might otherwise try to circumvent their ban. However,
it&rsquo;s important that this does <em>not</em> become the first line of defense against
rulebreaking users, who should instead be informed of the reason for their ban
so they have a chance to reform and appeal it. Here&rsquo;s what Dan has to say about
it:</p>
<blockquote>
<p>Shadowbanning has proven to still be useful for spammers and trolls
(i.e. when a new account shows up and is clearly breaking the site
guidelines off the bat). Most such abuse is by a relatively small
number of users who create accounts over and over again to do the same
things. When there&rsquo;s evidence that we&rsquo;ve repeatedly banned someone
before, I don&rsquo;t feel obliged to tell them we&rsquo;re banning them again.
[&hellip;] When we&rsquo;re banning an established account, though, we post a comment
saying so, and nearly always only after warning that user beforehand. Many such
users had no idea they were breaking the site guidelines and are
quite happy to improve their posts, which is a win for everyone.</p>
</blockquote>
<p>Dan also shared a link to search for comments where moderators have explained
to users why they&rsquo;ve been banned. Of course, this doesn&rsquo;t include users who were
banned without explanation, or that use slightly different language:</p>
<p><a href="https://hn.algolia.com/?query=by:dang%20we%20banned&amp;sort=byDate&amp;dateRange=all&amp;type=comment&amp;storyText=false&amp;prefix&amp;page=0">dang&rsquo;s bans</a></p>
<p><a href="https://hn.algolia.com/?query=by:sctb%20we%20banned&amp;sort=byDate&amp;dateRange=all&amp;type=comment&amp;storyText=false&amp;prefix=false&amp;page=0">sctb&rsquo;s bans</a></p>
<h2 id="data-based-insights">Data-based insights</h2>
<p>Here&rsquo;s an example of a fairly common moderator action:</p>
<p><img src="https://sr.ht/PhJM.png" alt=""></p>
<p><a href="https://hn.0x2237.club/post/15217697">This post</a> had its title changed at
around 09-11-17 12:10 UTC, and had the rank artificially adjusted to push it
further down the front page. We can tell that the drop was artificial just by
correlating it with the known moderator action, but we can also compare it
against the computed base rank:</p>
<p><img src="https://sr.ht/IJQI.png" alt=""></p>
<p>Note however that the base rank is often wildly different from the rank observed
in practice; the factors that go into adjusting it are rather complex. We can
also see that despite the action, the post&rsquo;s score continued to increase, even
at an accelerated pace:</p>
<p><img src="https://sr.ht/FmNU.png" alt=""></p>
<p>This &ldquo;title change and derank&rdquo; is a fairly common action - here are some more
examples from the past few days:</p>
<p><a href="https://hn.0x2237.club/post/15219154">Betting on the Web - Why I Build PWAs</a></p>
<p><a href="https://hn.0x2237.club/post/15210767">Silicon Valley is erasing individuality</a></p>
<p><a href="https://hn.0x2237.club/post/15208565">Chinese government is working on a timetable to end sales of fossil-fuel cars</a></p>
<p>Users can change their own post titles, which I&rsquo;m unable to distinguish from
moderator changes. However, correlating them with a strange change in rank is
generally a good bet. Submitters also generally will edit their titles earlier
rather than later, so a later change may indicate that it was seen by a
moderator after it rose some distance up the page.</p>
<p>I also occasionally find what seems to be the opposite - artificially bumping a
post further up the page. Here&rsquo;s two examples:
<a href="https://hn.0x2237.club/post/15213371">15213371</a> and
<a href="https://hn.0x2237.club/post/15209377">15209377</a>. Rank influencing in either
direction also happens without an associated title or URL change, but
automatically pinning such events down is a bit more subtle than my tools can
currently handle.</p>
<p>Moderators can also delete a post or indicate it as a dupe. The latter can be
(and is) detected by my tools, but the former is indistinguishable from the user
opting to delete posts themselves. In theory, posts that are deleted <em>after</em> the
author is no longer allowed to could be detected, but this happens rarely and my
tools don&rsquo;t track posts once they get old enough.</p>
<h3 id="flagging">Flagging</h3>
<p>The users have some moderation tools at their disposal, too - downvotes,
flagging, and vouching. When a comment is downvoted, it is moved towards the
bottom of the thread and is gradually colored grayer to become less visible, and
can be reversed with upvotes. When a comment gets enough flags, it is removed
entirely unless you have showdead enabled in your profile. Flagged posts are
downweighted or killed when enough flags accumulate. These posts are moved to
the bottom of the ranked posts even if you have showdead enabled, and can also
be seen in /new. Flagging can be reversed with the vouch feature, but flagged
stories are almost never vouched back into existence.</p>
<p><strong>Note</strong>: detection of post flagged status is very buggy with my tools. The API
exposes a boolean for dead posts, so I have to fall back on scraping to
distinguish between different kinds of dead-ness. But this is pretty buggy, so I
encourage you to examine the post yourself when browsing my site if in doubt.</p>
<h3 id="are-these-tools-abused-for-censorship">Are these tools abused for censorship?</h3>
<p>Well, with all of this data, was I able to find evidence of censorship? There
are two answers: yes and maybe. The &ldquo;yes&rdquo; is because users are <em>definitely</em>
abusing the flagging feature. The &ldquo;maybe&rdquo; is because moderator action leaves
room for interpretation. I&rsquo;ll get to that later, but let&rsquo;s start with flagging
abuse.</p>
<h4 id="censorship-by-users">Censorship by users</h4>
<p>The threshold for removing a story due to flags is rather low, though I don&rsquo;t
know the exact number. Here are some posts whose flags I consider questionable:</p>
<p><a href="https://hn.0x2237.club/post/15129859">Harvey, the Storm That Humans Helped Cause</a> (23 points)</p>
<p><a href="https://hn.0x2237.club/post/15116132">ES6 imports syntax considered harmful</a> (12 points)</p>
<p><a href="https://hn.0x2237.club/post/14415411">White-Owned Restaurants Shamed for Serving Ethnic Food</a> (33 points)</p>
<p><a href="https://hn.0x2237.club/post/14152602">The evidence is piling up – Silicon Valley is being destroyed</a> (27 points)</p>
<p>A good place to discover these sorts of events is to browse hnstats for posts
deleted with a score <a href="https://hn.0x2237.club/deleted-10">&gt;10 points</a>. There are
also occasions where the flags seem to be due to a poor title, which is a
fixable problem for which flagging is a harsh solution:</p>
<p><a href="https://hn.0x2237.club/post/14679207">Poettering downvoted 5 (at time of this writing) times</a></p>
<p><a href="https://hn.0x2237.club/post/14676296">Germany passes law restricting free speech on the internet</a></p>
<p>The main issue with flags is that they&rsquo;re often used as an alternative to the
HN&rsquo;s (by design) lack of a downvoting feature. HN also gives users no guidelines
on <em>why</em> they should flag posts, which mixes poorly with automated removal of a
post given enough flags.</p>
<h4 id="censorship-by-moderators">Censorship by moderators</h4>
<p>Moderator actions are a bit more difficult to judge. Moderation on HN is a black
box - most of the time, moderators don&rsquo;t make the reasoning behind their actions
clear. Many of their actions (such as rank influence) are also subtle and easy
to miss. Thankfully they are often receptive to being asked why some moderation
occurred, but only as often as not.</p>
<p>Anecdotally, I also find that moderators occasionally moderate selectively, and
keep quiet in the face of users asking them why. Notably this is a problem for
<abbr title="links for which you have to pay money to read the
content">paywalled</abbr> articles, which are <a href="https://news.ycombinator.com/newsfaq.html">against the
rules</a> but are often allowed to
remain.</p>
<p>Dan sent me a response to this section:</p>
<blockquote>
<p>[It&rsquo;s true that we don&rsquo;t explain our actions], but mostly because it would be
hopeless to try. We could do that all day and still not make everything clear,
because the quantity is overwhelming and the cost of a high-quality explanation
is steep. Moreover the experiment would be impossible to run because one
would die of boredom long before reaching 100%. Our solution to this
conundrum is not to try to explain everything but to answer specific
questions as best we can. We don&rsquo;t answer every question, but that&rsquo;s
mostly because we don&rsquo;t see every question. If people ask us things on
HN itself, odds are we won&rsquo;t see it (also, the site guidelines ask
users not to do this, per (<a href="https://news.ycombinator.com/newsguidelines.html">our
guidelines</a>). If they
<a href="mailto:hn@ycombinator.com">email us</a>, the probability of a
response approaches 1.</p>
</blockquote>
<p>I can attest personally to success reaching out to <a href="mailto:hn@ycombinator.com">hn@ycombinator.com</a> for
clarification and even reversal of some moderator decisions, though at a
response ratio further from 1 than this implies. That being said, I don&rsquo;t think
that private discourse between the submitter and the moderators is the only
solution. Other people may be invested in the topic, too - users who upvoted the
story might not notice its disappearance, but would like more attention drawn to
the topic and enjoy more discussion. Commenters are even more invested in the
posts. The submitter is not the only one whoses interests are at stake. This is
even more of a problem for posts which are moderated via user flags - the HN
mods are pretty discretionate but users are much less so.</p>
<p>Explaining every action is not necessary - I don&rsquo;t think anyone needs you to
explain why someone was banned when they were submitting links to earn money at
home in your spare time. However, I think a public audit log of moderator
actions would go a long way, and could be done by software - avoiding the need
to explain everything. I envision a change to your UI for banning users or
moderating posts with that adds a dropdown of common reasons and a textbox for
further elaboration when appropriate - then makes this information appear on
/moderation.</p>
<h3 id="conclusions">Conclusions</h3>
<p>I should again emphasize that most moderator actions are benign and agreeable.
They do a great job on the whole, but striving to do even better would be
admirable. I suggest a few changes:</p>
<ul>
<li>Make a public audit log of moderation activity, or at least reach out to me to
see what small changes could be done to help improve my statistics gathering.</li>
<li>Minimize use of more subtle actions like rank influence, and when used,</li>
<li>More frequently leave comments on posts where moderation has occurred
explaining the rationale and opening an avenue for public discussion and/or
appeal.</li>
<li>Put flagged posts into a queue for moderator review and don&rsquo;t remove posts
simply because they&rsquo;re flagged.</li>
<li>Consider appointing one or two moderators from the community, ideally people
with less bias towards SV or startup culture.</li>
</ul>
<p>Hacker News is a great place for just that - hacker news. It has been for a long
time and I hope it continues to be. Let&rsquo;s work together on running it
transparently to the benefit of all.</p>


guid: https://drewdevault.com/2017/10/09/Future-of-sway.html
title: The future of Wayland, and sway's role in it
url: https://drewdevault.com/2017/10/09/Future-of-sway.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Today I&rsquo;ve released sway
<a href="https://github.com/swaywm/sway/releases/tag/0.15-rc1">0.15-rc1</a>, the first
release candidate for the final 0.x release of sway. That&rsquo;s right - after sway
0.15 will be sway 1.0. After today, no new features are being added to sway
until we complete the migration to our new plumbing library,
<a href="https://github.com/swaywm/wlroots">wlroots</a>. This has been a long time
coming, and I would love to introduce you to wlroots and tell you what to expect
from sway 1.0.</p>
<p><small class="text-muted"><a href="https://github.com/swaywm/sway">Sway</a> is
a tiling Wayland compositor, if you didn&rsquo;t know.</small></p>
<p>Before you can understand what wlroots is, you have to understand its
predecessor: <a href="https://github.com/Cloudef/wlc">wlc</a>. The role of wlc is to manage
a number of low-level plumbing components of a Wayland compositor. It
essentially abstracts most of the hard work of Wayland compositing away from the
compositor itself. It manages:</p>
<ul>
<li>The EGL (OpenGL) context</li>
<li>DRM (display) resources</li>
<li>libinput resources</li>
<li>Rendering windows to the display</li>
<li>Communicating with Wayland clients</li>
<li>Xwayland (X11) support</li>
</ul>
<p>It does a few other things, but these are the most important. When sway wants to
render a window, it will be told about its existence through a hook from wlc.
We&rsquo;ll tell wlc where to put it and it will be rendered there. Most of the heavy
lifting has been handled by wlc, and this has allowed us to develop sway into a
powerful Wayland compositor very quickly.</p>
<p>However, wlc has some limitations, ones that sway has been hitting more and more
often in the past several months. To address these limitations, we&rsquo;ve been
working very hard on a replacement for wlc called wlroots. The relationship
between wlc and wlroots is similar to the relationship between Pango and
Harfbuzz - wlroots is much more powerful, but at the cost of putting a lot more
work on the shoulders of sway. By replacing wlc, we can customize the behavior
of the low level components of our system.</p>
<p>I&rsquo;m happy to announce that development on wlroots has been spectacular. Like
libweston has Weston itself, wlroots has a reference compositor called Rootston -
a simple floating compositor that lets us test and demonstrate the features of
wlroots. It is from this compositor that I write this blog post today. The most
difficult of our goals are behind us with wlroots, and we&rsquo;re now beginning to
plan the integration of wlroots and sway.</p>
<p>All of this work has been possible thanks to a contingent of highly motivated
contributors who have done huge amounts of work for wlroots, writing and
maintaining entire subsystems far faster than I could have done it alone. I
really cannot overstate the importance of these contributors. Thanks to their
contributions, most of my work is in organizing development and merging pull
requests. From the bottom of my heart, <a href="https://github.com/swaywm/wlroots/graphs/contributors">thank
you</a>.</p>
<p>And for all of this hard work, what are we going to get? Well, for some time
now, there have been many features requests in sway that we could not address,
and many long-standing bugs we could not fix. Thanks to wlroots, we can see many
of these addressed within the next few months. Here are some of the things you
can expect from the union of wlroots and sway:</p>
<ul>
<li>Rotated displays</li>
<li>Touchscreen bindings</li>
<li>Drawing tablet support</li>
<li>Mouse capture for games</li>
<li>Fractional display scaling</li>
<li>Display port daisy chaining</li>
<li>Multi-GPU support</li>
</ul>
<p>Some of these features are unique to sway even among Wayland <em>and</em> Xorg
desktops combined! Others, like output rotation, have been requested by our
users for a long time. I&rsquo;m looking forward to the several dozen long-open GitHub
issues that will be closed in the next couple of months. This is just the
beginning, too - wlroots is such a radical change that I can&rsquo;t even begin to
imagine all of the features we&rsquo;re going to be able to build.</p>
<p>We&rsquo;re sharing these improvements with the greater Wayland community, too.
wlroots is a platform upon which we intend to develop and promote open standards
that will unify the extensibility of all Wayland desktops. We&rsquo;ve also been
working with other Wayland compositors, notably
<a href="https://github.com/way-cooler/way-cooler">way-cooler</a>, which are preparing to
move their own codebases to a wlroots-based solution.</p>
<p>My goal is to ship sway 1.0 before the end of the year. These are exciting times
for Wayland, and I hope you&rsquo;re looking forward to it.</p>


guid: https://drewdevault.com/2017/10/26/Fuck-you-nvidia.html
title: Nvidia sucks and I'm sick of it
url: https://drewdevault.com/2017/10/26/Fuck-you-nvidia.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>There&rsquo;s something I need to make clear about Nvidia. Sway 1.0, which is the
release after next, is <em>not</em> going to support the Nvidia proprietary driver,
EGLStreams, or any other proprietary graphics APIs. The only supported driver
for Nvidia cards will be the open source nouveau driver. I will explain why.</p>
<p>Today, Sway is able to run on the Nvidia proprietary driver. This is not and has
never been an officially supported feature - we&rsquo;ve added a few things to try and
make it easier but my stance has <em>always</em> been that Nvidia users are on their
own for support. In fact, Nvidia support was added to Sway without my approval.
It comes from a library we depend on called wlc - had I&rsquo;d made the decision on
whether or not to support EGLStreams in wlc, I would have said no.</p>
<p>Right now, we&rsquo;re working very hard on replacing wlc, for reasons unrelated to
Nvidia. Our new library, wlroots, is better in every conceivable way for Sway&rsquo;s
needs. The Nvidia proprietary driver support is not coming along for the ride,
and here&rsquo;s why.</p>
<p>So far, I&rsquo;ve been speaking in terms of <em>Sway</em> supporting Nvidia, but this is
an ass-backwards way of thinking. <em>Nvidia</em> needs to support Sway. There are
Linux kernel APIs that we (and other Wayland compositors) use to get the job
done. Among these are KMS, DRM, and GBM - respectively Kernel Mode Setting,
Direct Rendering Manager, and Generic Buffer Management. Every GPU vendor
but Nvidia supports these APIs. Intel and AMD support them with mainlined<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>,
open source drivers. For AMD this was notably done by replacing their
proprietary driver with a new, open source one, which has been developed in
cooperation with the Linux community. As for Intel, they&rsquo;ve always been friendly
to Linux.</p>
<p>Nvidia, on the other hand, have been fucking assholes and have treated Linux
like utter shit for our entire relationship. About a year ago they announced
&ldquo;Wayland support&rdquo; for their proprietary driver. This included KMS and DRM
support (years late, I might add), but <em>not</em> GBM support. They shipped something
called EGLStreams instead, a concept that had been discussed and shot down by
the Linux graphics development community before. They did this because it makes
it easier for them to keep their driver proprietary without having work with
Linux developers on it. Without GBM, Nvidia <em>does not</em> support Wayland, and they
were real pricks for making some announcement like they actually did.</p>
<p>When people complain to me about the lack of Nvidia support in Sway, I get
really pissed off. It is <em>not my fucking problem</em> to support Nvidia, it&rsquo;s
Nvidia&rsquo;s fucking problem to support me. Even Broadcom, <em>fucking Broadcom</em>,
supports the appropriate kernel APIs. And proprietary driver users have the gall
to <em>reward</em> Nvidia for their behavior by giving them <em>hundreds of dollars</em> for
their GPUs, then come to <em>me</em> and ask me to deal with their bullshit <em>for free</em>.
Well, fuck you, too. Nvidia users are shitty consumers and I don&rsquo;t even want
them in my userbase. Choose hardware that supports your software, not the other
way around.</p>
<p>Buy AMD. Nvidia&ndash; fuck you!</p>
<p><strong>Edit</strong>: It&rsquo;s worth noting that Nvidia is evidently attempting to find a better
path with <a href="https://github.com/cubanismo/allocator">this new GitHub project</a>. I
hope it works out, but they aren&rsquo;t really cooperating much with anyone to build
it - particularly nouveau. It&rsquo;s more throwing code/blobs over the wall and
expecting everyone to change for them.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Mainlined means that they are included in the upstream Linux kernel source code. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2017/11/13/Portability-matters.html
title: Portability matters
url: https://drewdevault.com/2017/11/13/Portability-matters.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>There are many kinds of &ldquo;portability&rdquo; in software. Portability refers to the
relative ease of &ldquo;porting&rdquo; a piece of software to another system. That
platform might be another operating system, another CPU architecture, another
web browser, another filesystem&hellip; and so on. More portable software uses the
limited subset of interfaces that are common between systems, and less portable
software leverages interfaces specific to a particular system.</p>
<p>Some people think that portability isn&rsquo;t very important, or don&rsquo;t understand the
degree to which it&rsquo;s important. Some people might call their software portable
if it works on Windows and macOS - they&rsquo;re wrong. They might call their software
portable if it works on Windows, macOS, and Linux - but they&rsquo;re wrong, too.
Supporting multiple systems does not necessarily make your software portable.
What makes your software portable is <em>standards</em>.</p>
<p>The most important standard for software portability is POSIX, or the <strong>Portable
Operating System Interface</strong>. Significant subsets of this standard are supported
by many, many operating systems, including:</p>
<ul>
<li>Linux</li>
<li>*BSD</li>
<li>macOS</li>
<li>Minix</li>
<li>Solaris</li>
<li>BeOS</li>
<li>Haiku</li>
<li>AIX</li>
</ul>
<p>I <a href="https://en.wikipedia.org/wiki/POSIX#POSIX-oriented_operating_systems">could go
on</a>.
Through these operating systems, you&rsquo;re able to run POSIX compatible code on a
large number of CPU architectures as well, such as:</p>
<ul>
<li>i386</li>
<li>amd64</li>
<li>ARM</li>
<li>MIPS</li>
<li>PowerPC</li>
<li>sparc</li>
<li>ia64</li>
<li>VAX</li>
</ul>
<p>Again, I could go on. Here&rsquo;s the point: by supporting POSIX, your software runs
on basically every system. <em>That&rsquo;s</em> what it means to be portable - standards.
So why is it important to support POSIX?</p>
<p>First of all, if you use POSIX then your software runs on just about anything,
so lots of users will be available to you and it will work in a variety of
situations. You get lots of platforms for free (or at least cheap). But more
importantly, <em>new platforms</em> get your software for free, too.</p>
<p>The current market leaders are not the end-all-be-all of operating system
design - far from it. What they have in their advantage is working well enough
and being incubent. Windows, Linux, and macOS are still popular for the same
reason that legislator you don&rsquo;t like keeps getting elected! However, new
operating systems have a fighting chance thanks to POSIX. All you have to do to
make your OS viable is implement POSIX and you will immediately open up
hundreds, if not thousands, of potential applications. Portability is important
for innovation.</p>
<p>The same applies to other kinds of portability. Limiting yourself to standard
browser features gives new browsers a chance. Implementing standard networking
protocols allows you to interop with other platforms. I&rsquo;d argue that failing to
do this is <em>unethical</em> - it&rsquo;s just another form of vendor lock-in. This is why
Windows does not support POSIX.</p>
<p>This is also why I question niche programming languages like Rust when they
claim to be suited to systems programming or even kernel development. That&rsquo;s
simply not true when they only run on a small handful of operating systems and
CPU architectures. C runs on <em>literally</em> everything.</p>
<p>In conclusion: use standard interfaces for your software. That guy who wants to
bring new life to that old VAX will thank you. The authors of
<a href="https://servo.org/">servo</a> thank you. <em>You</em> will thank you when your
circumstances change in 5 years.</p>


guid: https://drewdevault.com/2017/11/24/Phone-maintenance.html
title: On taking good care of your phone
url: https://drewdevault.com/2017/11/24/Phone-maintenance.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>I just finished <a href="https://www.ifixit.com/Guide/s5/27077">replacing the micro-USB
daughterboard</a> on my Samsung Galaxy S5,
which involved taking the phone most of the way apart, doing the replacement,
and putting it back together. This inspired me to write about my approach to
maintaining my cell phone. I&rsquo;ve had this phone for a while and I have no plans
to upgrade - I backed the upcoming Purism phone, but I expect to spend
months/years on the software before I&rsquo;ll be using that as my daily driver.</p>
<p>I don&rsquo;t want to be buying a new phone every year. That&rsquo;s a lot of money! Though
the technophile in me finds the latest and greatest technology appealing, the
thought of doing my own repairs and upkeep on a battle-tested phone is equally
interesting. Here are the four things I&rsquo;ve found most important in phone upkeep.</p>
<h3 id="install-lineageos-or-replicant">Install LineageOS or Replicant</h3>
<p>Before I installed CyanogenMod when I bought this phone, I did some prying into
the stock ROM to see just how bad it was. It was even worse than I expected!
There were literally hundreds of apps and services with scary permissions
running in the background that could not be removed. These spy on you, wear down
your battery, and slow down your phone over time - another form of planned
obsolescence.</p>
<p>My phone is still as fast as the day I got it. It does a great job with
everything I ask it to do. The first thing you should do with every new phone is
install a third-party ROM - ideally, without Google apps. Stock ROMs suck, get
rid of it.</p>
<h3 id="insist-on-a-user-replacable-battery">Insist on a user-replacable battery</h3>
<p>Non-user-replacable batteries are an obvious form of planned obsolescence.
Batteries don&rsquo;t last forever and you should <em>never</em> buy a phone that you
cannot replace the battery of. A new battery for my S5 costs 10 bucks. 4 years
in, I&rsquo;ve replaced mine once and I can hold a charge fine for a couple of days.</p>
<h3 id="get-a-case">Get a case</h3>
<p>This one is pretty obvious, but I didn&rsquo;t follow this advice at first. I&rsquo;ve never
broken a screen, so I didn&rsquo;t bother with a case. When I decided I was going to
keep this phone for a long time, I went ahead and bought one. It doubles the
thickness of my phone but at least I can be sure I&rsquo;m not going to bust it up
when I drop it. It still fits in my pocket comfortably so it&rsquo;s no big deal.</p>
<h3 id="attempt-repairs-before-you-buy-a-new-phone">Attempt repairs before you buy a new phone</h3>
<p>The past couple of months, my phone&rsquo;s micro-USB3 port started to act up a bit. I
would have to wiggle the cable a bit to get it to take, and it could stop
charging if I rustled my desk the wrong way. I got a replacement USB
daughterboard on Amazon for 6 bucks. Replacing it took an hour, but when
removing the screen I broke the connection between my home button and my
motherboard - which was only 10 bucks for the replacement, including same day
shipping. The whole process was a lot easier than I thought it would be.</p>
<hr>
<p>Be a smart consumer when you&rsquo;re buying a phone. Insist on the replacable battery
and maybe read the iFixit teardown. Take good care of it and it&rsquo;ll last a long
time. Don&rsquo;t let consumerism get the better of you!</p>


guid: https://drewdevault.com/2017/12/02/History-of-intelligent-observation.html
title: A history of emergent intelligence
url: https://drewdevault.com/2017/12/02/History-of-intelligent-observation.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>As you all know, the simulation of universe 2813/9301 is now coming to a close.
This simulation is notable for being the first simulated universe suitable for
hosting intelligent life, but yesterday the simulation reached a state where we
believe no additional intelligences will emerge. It seems the final state of
this set of physical laws is a dark and empty universe of slowly evaporating
black holes. Though, given the historical significance of this simulation, it&rsquo;s
unlikely we we&rsquo;ll be turning it off any time soon!</p>
<div class="alert alert-warning">
<strong>Note</strong>: This document was translated to a language and format
suitable for human understanding. Locations within your observable universe are
referred to by your name for them, times are given in terms of your planetary
orbital period and relative to your reference frame, and terminology is
translated when your vocabulary is sufficient.
</div>
<p>The remaining simulation that constitutes the vast majority of computer time
allocated to this project, though it will no doubt be very boring. Given that
the fun is behind us, over in the archives we&rsquo;ve been putting together something
special to celebrate the work so far.</p>
<p>Watching these intelligent civilizations struggle to understand our simulation
from the inside out is a hoot when you and I can just read the manual! For them,
however, it must have been much more difficult. A history of this observation by
emergent intelligence from within our simulation from within follows. Without
further ado, let&rsquo;s revisit the most notable intelligences we discovered.</p>
<h3>9.93&times;10<sup>8</sup> years: 36-29-93-55-55</h3>
<p><em>Note: 36-29-93-55-55 is an approximation of our identifier for arbitrary
locations within the simulation. It does not correspond to a location in your
observable universe. Years are given from the epoch in terms of your planet&rsquo;s
present orbital period.</em></p>
<p>Though it did not develop self-awareness, the first observation of life - the
precursor to most forms of the simulation&rsquo;s emergent intelligence - was found at
this location. It was initially discarded as a relatively uninteresting anomaly
during our surveys, but was later revisited as we began to understand the
mechanics of intelligence within the simulation.</p>
<h3>1.28&times;10<sup>9</sup> years: 39-10-53-10-84</h3>
<p>&ldquo;Significant anomaly detected at 39-10-53-10-84. Apparent emergent intelligence
detected in active simulation. All personnel must return to the lab
immediately.&rdquo; Where were you when you read the memo? The intelligent creatures
we discovered had developed over a million years before we found them in our
surveys.</p>
<p>These where the first to enjoy a privilege few civilizations could lay claim
to: witnessing the galactic age of the simulation. They also were uniquely able
to see our simulation when it was small enough to observe a substantial portion
of it. Their investigations were unfortunately among the more primitive that
we&rsquo;ve observed - notably they never discovered general relativity. It was
shortly after their discovery of electromagnetism that they were destroyed by
their aging star&rsquo;s expansion. That was a difficult meeting for everyone when the
project leadership chose not to intervene.</p>
<h3>1.33&times;10<sup>9</sup> years: Messier 81</h3>
<p>The intelligences that developed here are notable for being the second group we
observed, though later surveys discovered additional earlier civilizations. They
also included one civilization which became the first to leave the planet on
which it developed - unfortunately never leaving their star, which ultimately
caused their demise. It&rsquo;s from them we also devised some of the most effective
means of automatic detection of intelligence, which led to the retroactive
discovery of many more intelligences.</p>
<h3>4.54&times;10<sup>9</sup> years: Humans</h3>
<style>
.redacted {
    background: #333;
    color: #333;
}
</style>
<p>Humanity is remarkable for being the first emergent intelligence to create <em>new</em>
intelligence within the simulation. All subsequent appearances of such
intelligences are referred to with the name humans gave to their creation:
artificial intelligence. Subsequently, humans also became the first to
<span class="redacted">look at you, you figured out how to read the redacted
text. I bet you feel real clever now.</span> <em>Note: you&rsquo;ll find out soon
enough.</em></p>
<h3>8.39&times;10<sup>9</sup> years: 59-54-77-33-19</h3>
<p>These guys were notable for being the longest-lived intelligent life. They were
located near a binary system with a star and a black hole. Remarkably, this
system was not unstable, unlike most civilizations near a black hole. Instead,
the relativistic effects of the black hole permitted them to observe a great
deal of the universe&rsquo;s history.</p>
<p>This also distinguishes them from the majority of other long-lived intelligent
civilizations, most of which were galactic civilizations. -19, along with a
handful of other long-lived black hole civilizations, they were among the only
civilizations to exist across long periods of the simulation without leaving
their host stars. They were unable to escape before the black hole began to feed
on the star, destroying the civilization at 4.56×10<sup>12</sup> years.
During this period, intelligence emerged 6 discrete times on their planet.</p>
<h3>8.43&times;10<sup>9</sup> years: UDF 423</h3>
<p>Interestingly, the record for the shortest lived intelligent civilization was
set only a short time after the longest lived one. Based on our criteria for
intelligence, this civilization only lasted 200 years before being destroyed by
the supernova of their host star.</p>
<h3>1.92&times;10<sup>10</sup> years: 60-17-07-08-49 &amp; 79-88-02-97-94</h3>
<p>These two civilizations share a solemn distinction: -49 was the last to observe
a galaxy outside of their local group, and -94 were the first to never observe
one (though early non-intelligent life at -94 might have seen if they had the
appropriate equipment). The light-speed software can be cruel at times. However,
-94 was still able to see the cosmic microwave background radiation, and from
this deduced that additional unseen galaxies might exist.</p>
<h3><span class="redacted">x.xx&times;10<sup>xx</sup> xxxxx: xx-xx-xx-xx-xx</span></h3>
<span class="redacted">
There's nothing interesting to see here, either. Stop looking. Lorem ipsum dolor
sit amet, consectetur adipiscing elit. Curabitur porta libero ut lectus finibus
lobortis. Cras dignissim dignissim ornare. Sed lobortis nulla vel mauris
lobortis, vel pretium tortor efficitur. Aenean sit amet nibh eros. That's your
reward for looking. You got to read lorem ipsum.
</span>
<h3>4.14&times;10<sup>10</sup> years: NGC 5055</h3>
<p>NGC 5055 was the first of only 32,083 intelligences to discover the simulated
nature of their universe after their discovery of <span class="redacted">you
really are terribly clever, aren&rsquo;t you</span>. They do not,
however, hold the distinction of being the first of the 489 intelligences that
made intentional contact with the proctors - that honor goes to 39-47-28-23-99,
as I&rsquo;m sure you&rsquo;re well aware.</p>
<h3>7.03&times;10<sup>11</sup> years: Peak intelligence</h3>
<p>This was the year that the largest number of discrete intelligent civilizations
existed in the simulation: 6,368,787,234,012. This period began with the birth
of 64-83-61-51-57 and ended with the death of 82-60-95-64-31 approximately 86
seconds later.</p>
<h3>1.70&times;10<sup>13</sup> years: Star formation stops</h3>
<p>The variety in emergent intelligence demonstrated in our simulation is
astonishing, but there&rsquo;s one thing every one of them has in common - a need for
energy. This energy has been provided in all but a few notable cases (see
publication 102.32 for a summary) by a star. At the conclusion of star
formation in our simulation, the rate at which emergent intelligent
civilizations were produced dramatically dropped. This also marked the beginning
of the decline of the 231 galactic civilizations that existed at the time, which
were unable to grow further without new stars being formed.</p>
<h3>9.85&times;10<sup>15</sup> years: 72-68-37-80-61</h3>
<p>The last intelligence to emerge was 72-68-37-80-61. They were not, however, the
last ones in the simulation. They were also among the emergent intelligences
that discovered the nature of the simulation, and the last that the proctors
elected to respond to attempted contact with.</p>
<h3>9.85&times;10<sup>15</sup> years: 76-54-95-81-66</h3>
<p>66 is notable for hosting the last intelligence to leave its host star when a
close encounter with the remnants of 76-54-95-81-18 collided with their galaxy.
Like 84% of the civilizations to undergo this ordeal in this time period, they
were prepared for it and were able to survive another 2,000 years after the
event (this post-stellar lifespan was slightly above average).</p>
<h3>4.65&times;10<sup>33</sup> years: 37-19-87-04-98</h3>
<p>The last emergent intelligence in the simulation. These were the last of the
group of 13 intelligent civilizations that devised a means for coping with the
energy-starved universe at this stage of the simulation. At the time of their
quiet death, they had utilized 77% of the remaining resources that could be
found outside of black holes.</p>
<hr>
<p>It&rsquo;s been an exciting time for our laboratory. Everyone has done great work on
this simulation. Though 2813/9301&rsquo;s incredible simulation is coming to an end,
we still have more work to do. We are proud to announce that in addition to
simulation 2813/9302 starting soon, we have elected to run simulation 2813/9301
once again. We have decided to nurture the emergent intelligences as if they
were our brothers, and communicate more openly with them. We have established a
new team to learn about each intelligence and make first contact with them using
means familiar to them, like maybe publishing our research documents as &ldquo;blog
posts&rdquo; within the simulation.</p>
<p>Great work, everyone. Here&rsquo;s to the next step.</p>


guid: https://drewdevault.com/2017/12/16/Firefox-is-on-a-slippery-slope.html
title: Firefox is on a slippery slope
url: https://drewdevault.com/2017/12/16/Firefox-is-on-a-slippery-slope.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>For a long time, it was just setting the default search provider to Google in
exchange for a beefy stipend. Later, paid links in your new tab page were added.
Then, a proprietary service, Pocket, was bundled into the browser - not as an
addon, but a hardcoded feature. In the past few days, we&rsquo;ve discovered an
advertisement in the form of browser extension was sideloaded into user
browsers. Whoever is leading these decisions at Mozilla needs to be stopped.</p>
<p>Here&rsquo;s a breakdown of what happened a few days ago. Mozilla and NBC
Universal did a &ldquo;collaboration&rdquo; (read: promotion) for the TV show Mr. Robot.
It involved sideloading a sketchy browser extension which will <strong
style="display: inline-block; transform: scaleY(-1)">invert</strong> text that
matches a list of Mr. Robot-related keywords like &ldquo;fsociety&rdquo;, &ldquo;robot&rdquo;, &ldquo;undo&rdquo;,
and &ldquo;fuck&rdquo;, and does a number of other things like adding an HTTP header to
certain sites you visit.</p>
<p>This extension was sideloaded into browsers via the &ldquo;experiments&rdquo; feature.
Not only are these experiments enabled by default, but updates <a href="https://redd.it/7i4puf">have been
known</a> to re-enable it if you turn it off. The
advertisement addon shows up <a href="http://www.bolcer.org/looking-glass2.png">like
this</a> on your addon page, and was
added to Firefox stable. If I saw this before I knew what was going on, I would
think my browser was compromised!  Apparently it was a mistake that this showed
up on the addon page, though - it was supposed to be <em>silently</em> sideloaded into
your browser!</p>
<p>There&rsquo;s <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1423003">a ticket</a> on
Bugzilla (Firefox&rsquo;s bug tracker) for discussing this experiment, but it&rsquo;s locked
down and no one outside of Mozilla can see it. There&rsquo;s <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1424977">another
ticket</a>, filed by
concerned users, which has since been disabled and had many comments removed,
particularly the angry (but respectful) ones.</p>
<p>Mozilla, this is <strong>not okay</strong>. This is wrong on so many levels. Frankly, whoever
was in charge should be fired over this - which is not something I call for
lightly.</p>
<p>First of all, web browsers are a <em>tool</em>. I don&rsquo;t want my browser to fool around,
I just want it to display websites faithfully. This is the prime directive of
web browsers, and you broke that. When I compile vim with gcc, I don&rsquo;t want
gcc to make vim sporadically add &ldquo;fsociety&rdquo; into every document I write. I want
it to compile vim and go away.</p>
<p>More importantly, these advertising anti-features gravely - perhaps terminally -
violate user trust. This event tells us that &ldquo;Firefox studies&rdquo; into a backdoor
for advertisements, and I will <em>never</em> trust it again. But it doesn&rsquo;t matter -
you&rsquo;re going to re-enable it on the next update. You know what that means? I
will never trust <em>Firefox</em> again. I switched to
<a href="http://qutebrowser.org/">qutebrowser</a> as my daily driver because this crap was
starting to add up, but I still used Firefox from time to time and never
resigned from it entirely or stopped recommending it to friends. Well, whatever
goodwill was left is gone now, and I will only recommend other browsers
henceforth.</p>
<p>Mozilla, you fucked up <em>bad</em>, and you still haven&rsquo;t apologised. The study is
still active and ongoing. There is no amount of money that you should have
accepted for this. This is the last straw - and I took a lot of straws from you.
Goodbye forever, Mozilla.</p>
<p><strong>Update 2017-12-16 @ 22:33</strong></p>
<p>It has been clarified that an about:config flag must be set for this addon&rsquo;s
behavior to be visible. This improves the situation considerably, but I do not
think it exenorates Mozilla and I stand firm behind most of my points. The study
has also been rolled back by Mozilla, and Mozilla has issued
<a href="https://gizmodo.com/mozilla-slipped-a-mr-robot-promo-plugin-into-firefox-1821332254">statements</a>
to the
<a href="https://gizmodo.com/after-blowback-firefox-will-move-mr-robot-extension-t-1821354314">media</a>
justifying the study (no apology has been issued).</p>
<p><strong>Update 2017-12-18</strong></p>
<p>Mozilla has issued an apology:</p>
<p><a href="https://blog.mozilla.org/firefox/update-looking-glass-add/">https://blog.mozilla.org/firefox/update-looking-glass-add/</a></p>
<p><strong>Responses</strong>:</p>
<p><a href="https://blog.jeaye.com/2017/12/16/firefox/">Mozilla, Firefox, Looking Glass, and you</a>
via jeaye.com</p>


guid: https://drewdevault.com/2018/01/02/The-case-against-fork.html
title: fork is not my favorite syscall
url: https://drewdevault.com/2018/01/02/The-case-against-fork.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>This article has been on my to-write list for a while now. In my opinion, fork
is one of the most questionable design choices of Unix. I don&rsquo;t understand the
circumstances that led to its creation, and I grieve over the legacy rationale
that keeps it alive to this day.</p>
<p>Let&rsquo;s set the scene. It&rsquo;s 1971 and you&rsquo;re a fly on the wall in Bell Labs,
watching the first edition of Unix being designed for the PDP-11/20. This
machine has a 16-bit address space with no more than 248 kilobytes of memory.
They&rsquo;re discussing how they&rsquo;re going to support programs that spawn new
programs, and someone has a brilliant idea. &ldquo;What if we copied the entire
address space of the program into a new process running from the same spot, then
let them overwrite themselves with the new program?&rdquo; This got a rousing laugh
out of everyone present, then they moved on to a better design which would
become immortalized in the most popular and influential operating system of all
time.</p>
<p>At least, that&rsquo;s the story I&rsquo;d like to have been told. In actual fact, the
laughter becomes consensus. There&rsquo;s an obvious problem with this approach: every
time you want to execute a new program, the entire process space is copied and
promptly discarded when the new program begins.  Usually when I complain about
fork, this the point when its supporters play the virtual memory card, pointing
out that modern operating systems don&rsquo;t actually have to copy the whole address
space. We&rsquo;ll get to that, but first — First Edition Unix <em>does</em> copy the
whole process space, so this excuse wouldn&rsquo;t have held up at the time. By Fourth
Edition Unix (the next one for which kernel sources survived), they had wisened
up a bit, and started only copying segments when they faulted.</p>
<p>This model leads to a number of problems. One is that the new process inherits
<em>all</em> of the parent&rsquo;s process descriptors, so you have to close them all before
you exec another process. However, unless you&rsquo;re manually keeping tabs on your
open file descriptors, there is no way to know what file handles you must close!
The hack that solves this is <code>CLOEXEC</code>, the first of many hacks that deal with
fork&rsquo;s poor design choices. This file descriptors problem balloons a bit -
consider for example if you want to set up a pipe. You have to establish a piped
pair of file descriptors in the parent, then close every fd <em>but</em> the pipe in
the child, then <code>dup2</code> the pipe file descriptor over the (now recently closed)
file descriptor 1. By this point you&rsquo;ve probably had to do several non-trivial
operations and utilize a handful of variables from the parent process space,
which <em>hopefully</em> were on the stack so that we don&rsquo;t end up copying segments
into the new process space anyway.</p>
<p>These problems, however, pale in comparison to my number one complaint with the
fork model. Fork is the direct cause of the <em>stupidest</em> component I&rsquo;ve <em>ever</em>
heard of in an operating system: the out-of-memory (aka OOM) killer. Say you
have a process which is using half of the physical memory on your system, and
wants to spawn a tiny program. Since fork &ldquo;copies&rdquo; the entire process, you might
be inclined to think that this would make fork fail. But, on Linux and many
other operating systems since, it does not fail! They agree that it&rsquo;s stupid to
copy the entire process just to exec something else, but because fork is
Important for Backwards Compatibility, they just fake it and reuse the same
memory map (except read-only), then trap the faults and actually copy later.
The hope is that the child will get on with it and exec before this happens.</p>
<p>However, nothing prevents the child from doing something other than exec -
it&rsquo;s free to use the memory space however it desires! This approach now leads to
<em>memory overcommittment</em> - Linux has promised memory it does not have. As a
result, when it really does run out of physical memory, Linux will just kill off
processes until it has some memory back. Linux makes an awfully big fuss about
&ldquo;never breaking userspace&rdquo; for a kernel that will lie about memory it doesn&rsquo;t
have, then kill programs that try to use the back-alley memory they were given.
That this nearly 50 year old crappy design choice has come to this astonishes
me.</p>
<p>Alas, I cannot rant forever without discussing the alternatives. There <strong>are</strong>
better process models that have been developed since Unix!</p>
<p>The first attempt I know of is BSD&rsquo;s <code>vfork</code> syscall, which is, in a nutshell,
the same as fork but with severe limitations on what you do in the child process
(i.e. nothing other than calling exec straight away). There are <em>loads</em> of
problems with <code>vfork</code>. It only handles the most basic of use cases: you cannot
set up a pipe, cannot set up a pty, and can&rsquo;t even close open file descriptors
you inherited from the parent. Also, you couldn&rsquo;t really be sure of what
variables you were and weren&rsquo;t editing or allowed to edit, considering the
limitations of the C specification. Overall this syscall ended up being pretty
useless.</p>
<p>Another model is <code>posix_spawn</code>, which is a hell of an interface. It&rsquo;s far too
complicated for me to detail here, and in my opinion far too complicated to ever
consider using in practice. Even if it could be understood by mortals, it&rsquo;s a
really bad implementation of the spawn paradigm — it basically operates
like fork backwards, and inherits many of the same flaws. You still have to deal
with children inheriting your file descriptors, for example, only now you do it
in the parent process. It&rsquo;s also straight-up impossible to make a genuine pipe
with <code>posix_spawn</code>. (<em>Note: a reader corrected me - this is indeed possible via
posix_spawn_file_actions_adddup2</em>.)</p>
<p>Let&rsquo;s talk about the good models - <code>rfork</code> and spawn (at least, if spawn is done
right). <code>rfork</code> originated from plan9 and is a beautiful little coconut of a
syscall, much like the rest of plan9. They also implement fork, but it&rsquo;s a
special case of <code>rfork</code>. plan9 does not distinguish between processes and
threads - all threads are processes and vice versa. However, new processes in
plan9 are not the everything-must-go fuckfest of your typical fork call.
Instead, you specify exactly what the child should get from you. You can choose
to include (or not include) your memory space, file descriptors, environment, or
a number of other things specific to plan9. There&rsquo;s a cool flag that makes it so
you don&rsquo;t have to reap the process, too, which is nice because reaping children
is another really stupid idea. It still has some problems, mainly around
creating pipes without tremendous file descriptor fuckery, but it&rsquo;s basically as
good as the fork model gets. Note: Linux offers this via the <code>clone</code> syscall
now, but everyone just fork+execs anyway.</p>
<p>The other model is the spawn model, which I prefer. This is the approach I took
in my own kernel for KnightOS, and I think it&rsquo;s also used in NT (Microsoft&rsquo;s
kernel). I don&rsquo;t really know much about NT, but I can tell you how it works in
KnightOS. Basically, when you create a new process, it is kept in limbo until
the parent consents to begin. You are given a handle with which you can
configure the process - you can change its environment, load it up with file
descriptors to your liking, and so on. When you&rsquo;re ready for it to begin, you
give the go-ahead and it&rsquo;s off to the races. The spawn model has none of the
flaws of fork.</p>
<p>Both fork and exec can be useful at times, but spawning is much better for 90%
of their use-cases. If I were to write a new kernel today, I&rsquo;d probably take a
leaf from plan9&rsquo;s book and find a happy medium between <code>rfork</code> and spawn, so you
could use spawn to start new threads in your process space as well. To the
brave OS designers of the future, ready to shrug off the weight of legacy:
please reconsider fork.</p>


guid: https://drewdevault.com/2018/01/10/Learn-your-package-manager.html
title: Learn about your package manager
url: https://drewdevault.com/2018/01/10/Learn-your-package-manager.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Tools like virtualenv, rbenv, and to a lesser extent npm and pip, are
occasionally useful in development but encourage bad practices in production.
Many people forget that their distro already has a package manager! And there&rsquo;s
more&ndash; you, the user, can write packages for it!</p>
<p>Your distro&rsquo;s package repositories probably already have a lot of your
dependencies, and can conveniently update your software alongside the rest of
your system. On the whole you can expect your distro packages to be much better
citizens on your system than a language-specific package manager will be.
Additionally, pretty much all distros provide a means for you to host your own
package repositories, from which you can install and update any packages you
choose to make.</p>
<p>If you find some packages to be outdated, find out who the package maintainer is
and shoot them an email. Or better yet - find out how the package is built and
send them a patch instead. Linux distributions are run by volunteers, and it&rsquo;s
easy to volunteer yourself! Even if you find <em>missing</em> packages, it&rsquo;s a simple
matter to whip up a package yourself and submit it for inclusion in your
distro&rsquo;s package repository, installing it from your private repo in the
meanwhile.</p>
<p>&ldquo;But what if dependencies update and break my stuff?&rdquo;, you ask. First of all,
why aren&rsquo;t you keeping your dependencies up-to-date? That aside, some distros,
like Alpine, let you pin packages to a specific version. Also, using the
distro&rsquo;s package manager doesn&rsquo;t necessarily mean you have to use the distro&rsquo;s
package repositories - you can stand up your own repos and prioritize it over
the distro repos, then release on any schedule you want.</p>
<p>In my opinion, the perfect deployment strategy for some software is pushing a
new package to your package repository, then SSHing into your fleet and running
system updates (probably automatically). This is how I manage deployments for
most of my software. As a bonus, these packages offer a good place to configure
things that your language&rsquo;s package manager may be ill suited to, such as
service files or setting up new users/groups on the system. Consider it!</p>


guid: https://drewdevault.com/2018/01/16/Fees-on-donation-platforms.html
title: Fee breakdown for various donation platforms
url: https://drewdevault.com/2018/01/16/Fees-on-donation-platforms.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>Understanding fees are a really confusing part of supporting creators of things
you like. I provide a few ways for people to support my work, and my supporters
can struggle to understand the differences between them. It comes down to fees,
of which there are several kinds (note: I just made these terms up):</p>
<ul>
<li><strong>Transaction fees</strong> are charged by the payment processor (the company that
takes down your card number and runs the transaction with your bank). These
are typically in the form of a percentage of the transaction plus a few cents.</li>
<li><strong>Platform fees</strong> are charged by the platform (e.g. Patreon) to run their
operation, typically in the form of a fixed percentage of the transaction.</li>
<li><strong>Withdrawal fees</strong> are charged to move money from the platform to the
creator&rsquo;s bank account. These vary depending on the withdrawal processor.</li>
<li><strong>Taxes</strong> are also implicated, depending on how much the creator makes.</li>
</ul>
<p>All of this adds up to a very confusing picture. I&rsquo;ve made a calculator to help
you sort it out.</p>
<p><strong>Note</strong>: For an up-to-date calculation of Patreon&rsquo;s fees, see <a href="https://drewdevault.com/2019/05/06/Calculate-your-doation-fees-for-Patreon.html">the follow-up
post</a>.</p>
<p><noscript>Sorry, the calculator requires JavaScript.</noscript></p>
<div id="react-root"></div>
<script src="https://drewdevault.com/js/donation-calc-v1.js"></script>
<h3 id="sources">Sources</h3>
<p><strong>fosspay</strong></p>
<p>Only the typical <a href="https://stripe.com/us/pricing">Stripe fee</a> is applied.</p>
<p>Note: I am the author of fosspay, if you didn&rsquo;t already know.</p>
<p><strong>Patreon</strong></p>
<p><a href="https://patreon.zendesk.com/hc/en-us/articles/204606125-How-do-you-calculate-fees-">How do you calculate fees?</a></p>
<p><a href="https://patreon.zendesk.com/hc/en-us/articles/203913489-What-are-my-options-to-receive-payout-">What are my options to receive payout?</a></p>
<p><strong>Liberapay</strong></p>
<p><a href="https://liberapay.com/about/faq">FAQ</a></p>


guid: https://drewdevault.com/2018/01/27/Sway-and-client-side-decorations.html
title: Sway and client side decorations
url: https://drewdevault.com/2018/01/27/Sway-and-client-side-decorations.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>You may have recently seen an article from GNOME on the subject of client side
decorations (CSD) titled <a href="https://blogs.gnome.org/tbernard/2018/01/26/csd-initiative/">Introducing the CSD
Initiative</a>. It
states some invalid assumptions which I want to clarify, and I want to tell you
<a href="https://github.com/swaywm/sway">Sway</a>&rsquo;s
stance on the subject. I also speak for the rest of the projects involved in
<a href="https://github.com/swaywm/wlroots">wlroots</a> on this matter, including <a href="https://github.com/way-cooler/way-cooler">Way
Cooler</a>,
<a href="https://github.com/Ongy/waymonad">waymonad</a>, and
<a href="https://github.com/Bl4ckb0ne/bspwc">bspwc</a>.</p>
<p>The subject of which party is responsible for window decorations on Wayland (the
client or the server) has been a subject of much debate. I want to clarify that
though GNOME may imply that a consensus has been reached, this is not the case.
CSD have real problems that have long been waved away by its supporters:</p>
<ul>
<li>No consistent look and feel between clients and GUI toolkits</li>
<li>Misbehaving clients cannot be moved, closed, minimized, etc</li>
<li>No opportunity for compositors to customize behavior (e.g. tabbed windows on
Sway)</li>
</ul>
<p>We are willing to cooperate on a compromise, but GNOME does not want to
entertain the discussion and would rather push disingenuous propaganda for their
cause. The topic of the #wayland channel on Freenode includes the statement
&ldquo;Please do not argue about server-side vs. client-side decorations. It&rsquo;s settled
and won&rsquo;t change.&rdquo; I have been banned from this channel for over a year because
I persistently called for compromise.</p>
<p>GNOME&rsquo;s statement that &ldquo;[server-side decorations] do not (and will never) work
on Wayland&rdquo; is false. KDE and Sway have long agreed on the importance of these
problems and have worked together on a solution. We have developed and
implemented a Wayland protocol extension which allows the compositor and client
to negotiate what kind of decorations each wishes to use. KDE, Sway, Way Cooler,
waymonad, and bspwc are all committed to supporting server-side decorations on
our compositors.</p>
<hr>
<p>See also: <a href="https://blog.martin-graesslin.com/blog/2018/01/server-side-decorations-and-wayland/">Martin Flöser of KDE responds to GNOME&rsquo;s
article</a></p>


guid: https://drewdevault.com/2018/02/05/Introduction-to-POSIX-shell.html
title: Introduction to POSIX shell
url: https://drewdevault.com/2018/02/05/Introduction-to-POSIX-shell.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>What the heck is the POSIX shell anyway? Well, the POSIX (the Portable Operating
System Interface) shell is the standard Unix shell - standard meaning it was
formally defined and shipped in a published standard. This makes shell scripts
written for it portable, something no other shell can lay claim to. The POSIX
shell is basically a formalized version of the venerable Bourne shell, and on
your system it lives at <code>/bin/sh</code>, unless you&rsquo;re one of the unlucky masses for
whom this is a symlink to bash.</p>
<h2 id="why-use-posix-shell">Why use POSIX shell?</h2>
<p>The &ldquo;Bourne Again shell&rdquo;, aka bash, is not standardized. Its grammar,
features, and behavior aren&rsquo;t formally written up anywhere, and only one
implementation of bash exists. Without a standard, bash is defined <em>by</em> its
implementation. POSIX shell, on the other hand, has many competing
implementations on many different operating systems - all of which are
compatible with each other because they conform to the standard.</p>
<p>Any shell that utilizes features specific to Bash are not portable, which means
you cannot take them with you to any other system. Many Linux-based systems do
not use Bash or GNU coreutils. Outside of Linux, pretty much everyone but Hurd
does <em>not</em> ship GNU tools, including bash<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. On any of these systems, scripts
using &ldquo;bashisms&rdquo; will not work.</p>
<p>This is bad if your users wish to utilize your software anywhere other than
GNU/Linux. If your build tooling utilizes bashisms, your software will not build
on anything but GNU/Linux. If you ship runtime scripts that use bashisms, your
software will not <em>run</em> on anything but GNU/Linux. The case for sticking to
POSIX shell in shipping software is compelling, but I argue that you should
stick to POSIX shell for your personal scripts, too. You might not care now, but
when you feel like flirting with other Unicies you&rsquo;ll thank me when all of your
scripts work.</p>
<p>One place where POSIX shell does <em>not</em> shine is for interactive use - a place
where I think bash sucks, too. Any shell you want to use for your day-to-day
command line work is okay in my book. I use fish. Use whatever you like
interactively, but stick to POSIX sh for your scripts.</p>
<h2 id="how-do-i-use-posix-shell">How do I use POSIX shell?</h2>
<p>At the top of your scripts, put <code>#!/bin/sh</code>. You don&rsquo;t have to worry about using
<code>env</code> here like you might have been trained to do with bash: <code>/bin/sh</code> is the
standardized location for the POSIX shell, and any standards-conforming system
will either put it there or make your script work anyway.<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup></p>
<p>The next step is to avoid bashisms. There are many, but here are a few that
might trip you up:</p>
<ul>
<li><code>[[ condition ]]</code> does not work; use <code>[ condition ]</code></li>
<li>Arrays do not work; <a href="http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_05">use IFS</a></li>
<li>Local variables do not work; use a subshell</li>
</ul>
<p>The easiest way to learn about POSIX shell is to <a href="http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html">read the
standard</a> -
it&rsquo;s not too dry and shorter than you think.</p>
<h2 id="using-standard-coreutils">Using standard coreutils</h2>
<p>The last step to writing portable scripts is to use portable tools. Your system
may have GNU coreutils installed, which provides tools like <code>grep</code> and <code>cut</code>.
Unfortunately, GNU has extended these tools with its own non-portable flags and
tools. It&rsquo;s important that you avoid these.</p>
<p>One dead giveaway of a non-portable flag is long flags, e.g. <code>grep --file=FILE</code>
as opposed to <code>grep -f</code>. The POSIX standard only defines the <code>getopt</code> function -
not the proprietary GNU <code>getopt_long</code> function that&rsquo;s used to interpret long
options. As a result, no long flags are standardized. You might worry that this
will make your scripts difficult to understand, but I think that on the whole it
will not. Shell scripts are already pretty alien and require some knowledge to
understand. Is knowledge of what the magic word <code>grep</code> means much different
from knowledge of what <code>grep -E</code> means?</p>
<p>I also like that short flags allow you to make more concise command lines. Which
is better: <code>ps --all --format=user --without-tty</code>, or <code>ps -aux</code>? If you are
inclined to think the former, do you also prefer <code>function(a, b, c) { return a + b + c; }</code> over <code>(a, b, c) =&gt; a + b + c</code>?  Conciseness matters, and POSIX shell
supports comments if necessary!</p>
<p>Some tips for using short flags:</p>
<ul>
<li>They can be collapsed: <code>cmd -a -b -c</code> is equivalent to <code>cmd -abc</code></li>
<li>If they take additional arguments, either a space or no separation is
acceptable: <code>cmd -f&quot;hello world&quot;</code> or <code>cmd -f &quot;hello world&quot;</code></li>
</ul>
<p>A good reference for learning about standardized commands is, once again, <a href="http://pubs.opengroup.org/onlinepubs/9699919799/">the
standard</a>. From this page,
search for the command you want, or navigate through &ldquo;Shell &amp; Utilities&rdquo; -&gt;
&ldquo;Utilities&rdquo; for a list. If you have <code>man-pages</code> installed, you will also find
POSIX man pages installed on your system with the <code>p</code> postfix, such as <code>man 1p grep</code>. Note: at the time of writing, the POSIX man pages do not use dashes if
your locale is UTF-8, which makes searching for flags with <code>/</code> difficult. Use
<code>env LC_ALL=POSIX man 1p grep</code> if you need to search for flags, and I&rsquo;ll speak
to the maintainer of man-pages about this.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>A reader points out that macOS ships an ancient version of bash. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p><em>2018-05-15 correction</em>: <code>#!/bin/sh</code> is unfortunately not standardized by POSIX. However, I still recommend its use, as most operating systems will place it there. The portable way to invoke shell scripts is <code>sh path/to/script</code>. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2018/02/13/The-last-years.html
title: The last years
url: https://drewdevault.com/2018/02/13/The-last-years.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p><strong>August 14th, 2019</strong> PYONGYANG IN CHAOS AS PANDEMIC DECIMATES LEADERSHIP.
Sources within the country have reported that a fast-acting and deadly
infectious disease has suddenly infected the population of Pyongyang, the
capital city of North Korea, where most of the country&rsquo;s political elite live.
Unconfirmed reports suggest that a significant fraction of the leadership has
been affected.</p>
<p>The reclusive country has appealed for immediate aid from the international
community and it is reported that a group of medical experts from Seoul have
been permitted to enter via the Joint Security Area. Representatives from the
United States Center for Disease Control and the Chinese Center for Disease
Control and Prevention have also agreed to send representatives into the country
to help control the outbreak.</p>
<p>North Korea is known for it&rsquo;s unwillingness to cooperate with the international
community, particularly with respect to&hellip;</p>
<hr>
<p><strong>October 7th, 2019</strong> NEW APPROACH SHOWS PROMISING RESULTS FOR CYSTIC FIBROSIS.
Researchers announced yesterday that they were able to design a disease which
corrects the genome of patients suffering from the early stages of cystic
fibrosis. The study was shown to stop the progression of the genetic disease in
all subjects, and several subjects even showed signs of reversal. The FDA has
begun the process of evaluating the treatment for the general public.</p>
<p>Scientists involved explained the process involved using a modified version of
the common cold. They were able to reduce the negative effects of the virus, and
utilized it as a means of delivering a CRISPR-based payload that directly edited
the genome of members of the study. Scientists on the study suggest that in the
future, a similarly benign virus could be introduced to the general public to
eliminate the disease across the entire human population.</p>
<p>Some scientists are skeptical of the risks of this approach, but others spoke
favorably&hellip;</p>
<hr>
<p><strong>September 30th, 2019</strong> UNITED STATES CLAIMS RESPONSIBILITY FOR PYONGYANG
EPIDEMIC. In response to increasing alarm in the international community
regarding the origins of the artificial virus that took the life of Kim Jong-un
in August, the United States government has stepped forward to claim
responsibility. President Trump justified the move in a public statement,
claiming that the development of North Korean nuclear weapons capable of
striking American targets required such a response, and points to the ongoing
reunification efforts as evidence of a job well done.</p>
<p>Many leaders of the international community have issued statements condemning
the United State&rsquo;s attack, though some leaders have expressed relief that the
speculation regarding a rogue group of biologists was dispelled. Korean
officials have also issued statements condemning the attack, noting that several
presumably innocent family members of Pyongyang officials were killed, but
reaffirmed their commitment to supporting the population of the North and
continuing to peacefully unify the peninsula.</p>
<p>The relative ease of the reunification effort, long thought to be impossible, is
the result of the incredibly swift and precise nature of the American attack&hellip;</p>
<hr>
<p><strong>November 18th, 2020</strong> BRITAIN TARGET OF BIOLOGICAL ATTACK? Members of the
British public have come down with a highly contagious but largely benign form
of the measles, igniting panic among the population. The royal family and
members of the parliament have been quarantined and the country&rsquo;s biologists are
examining specimens of the disease for signs of human tampering. This is the
next in a series of scares, following the flu outbreak in Mexico this June.</p>
<p>We spoke with an expert in the field (who wished to remain anonymous) to
understand exactly how biologically engineered diseases are possible. Our expert
pointed to recent advances in genetic engineering, particularly CRISPR, which
have allowed research in this field to advance at an unprecedented pace for a
fraction of the costs previously associated with such research. For a layman&rsquo;s
explanation of what CRISPR is and how it works, see page 3.</p>
<p>Officials in Britain have issued a statement encouraging the public not to
worry, and stated that they had no reason to believe&hellip;</p>
<hr>
<p><strong>February 2nd, 2021</strong> LARGE GENETIC DATABASE LEAKED IN HACK. Personal genomics
company 23andMe released a statement today admitting that their database of
personal genetic records was leaked in a hack in May of last year. The company,
founded in 2006, collects genetic records from customers curious in their
ancestry and sends them a report of interesting information. The database is
said to contain names, email addresses, and samples of each customer&rsquo;s genome
dating back to the company&rsquo;s inception.</p>
<p>Estimates show that up to 3 million customers are affected, mostly from the
United States. The company has not revealed how much of each customer&rsquo;s genome
was disclosed, but experts agree that it would not have been practical for the
company to have stored their customer&rsquo;s full genomes, and caution affected
customers against panic. At this time, the identity of the hacker is unknown.</p>
<p>The company&rsquo;s president attributes the security breach to their reduced ability
to maintain a secure database due to their falling profits in recent years as
the general public grows more concious of&hellip;</p>
<hr>
<p><strong>June 28th, 2021</strong> OUTBREAK OF DEATHS AMONG &ldquo;JOHN ROBERTS&rdquo;. The United States
supreme court chief justice John Roberts was found dead in his home this
morning, the seventh &ldquo;John Roberts&rdquo; to die within the past 3 days. He was found
to have the disease which scientists have described as &ldquo;a new level of
sophistication&rdquo; in biological engineering. A substantial fraction of the entire
population is expected to have contracted this disease, but do not show any
symptoms. It was specifically designed to target a number of individuals named
John Roberts, and all other infected persons were unaffected.</p>
<p>It is believed that the genetic information used in this attack was sourced from
the recent leaks of genetic databases from major genetic testing companies, the
largest of which were the 23andMe and Ancestry.com leaks in February and April
respectively. Experts suggest that the data in the leak was not enough to
conclusively identify the justice, and the attackers simply targeted all genomes
matching that name.</p>
<p>The senate is expected to vote nearly unanimously on legislation this week which
outlaws the collection of genetic information by private companies, a move
largely considered&hellip;</p>
<hr>
<p><strong>August 28th, 2022</strong> STUDY SHOWS IMPOTENCE GROWING AT ALARMING RATE. A study
conducted by a Japanese team shows the birth rate around the world is decreasing
at a dramatically increased pace. According to the study, 42 of the 60 countries
included in the study showed a decrease in new pregnancies of 30% or more
compared to a similar time frame in 2012. They said the trend is expected to
continue, and possibly accelerate.</p>
<p>Japan is known for its research into fertility, as it has shown a steep decline
in births over the past&hellip;</p>
<hr>
<p><strong>October 1st, 2022</strong> HUMAN BIRTHS EXPECTED TO CEASE WITHIN ONE YEAR. We are sad
to report that biologists have confirmed claims issued last week by a radical
environmentalist group: a highly contagious disease engineered to bring about
impotence has infected most of the Earth&rsquo;s population. The group is a member of
the so-called &ldquo;Voluntary Extinction&rdquo; movement, which aims to drive the human
race extinct by ceasing human reproduction. Scientists suggest that this move is
highly unlikely to completely drive humanity extinct, but confirm that it&rsquo;s
likely that massive population losses are in our future.</p>
<p>Work is underway to determine which members of the population have escaped
exposure, and plan for the continuity of the species. Members of isolated
communities are asked to avoid contact with the outside world, and governments
are cracking down on travel to and from the more remote regions of their
countries. The CDC has reported no estimate on when a vaccine will be available
for the disease, but has confirmed that one must be developed before contact
with these communities is advisable.</p>
<p>The government of New Zealand announced this morning their intention to send
sterilized supply shipments to research teams in Antarctica, and Canada
announced that all travel&hellip;</p>
<hr>
<p>Inspired by this excellent (and scary) talk at DEFCON 25:
<a href="https://www.youtube.com/watch?v=HKQDSgBHPfY">John Sotos - Genetic Diseases to Guide Digital Hacks of the Human Genome</a></p>


guid: https://drewdevault.com/2018/02/17/Writing-a-Wayland-compositor-1.html
title: Writing a Wayland Compositor, Part 1: Hello wlroots
url: https://drewdevault.com/2018/02/17/Writing-a-Wayland-compositor-1.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>This is the first in a series of <em>many</em> articles I&rsquo;m writing on the subject of
building a functional Wayland compositor from scratch. As you may know, I am the
lead maintainer of <a href="https://github.com/swaywm/sway">sway</a>, a reasonably popular
Wayland compositor. Along with many other talented developers, we&rsquo;ve been
working on <a href="https://github.com/swaywm/wlroots">wlroots</a> over the past few
months. This is a powerful tool for creating new Wayland compositors, but it is
very dense and difficult to understand. Do not despair! The intention of these
articles is to make you understand and feel comfortable using it.</p>
<p>Before we dive in, a quick note: the wlroots team is starting a crowdfunding
campaign today to fund travel for each of our core contributors to meet in
person and work for two weeks on a hackathon. Please consider contributing to
<a href="https://www.indiegogo.com/projects/sway-hackathon-software/x/1059863">the campaign</a>!</p>
<p>You <strong>must</strong> read and comprehend my earlier article, <a href="https://drewdevault.com/2017/06/10/Introduction-to-Wayland.html">An introduction to
Wayland</a>, before attempting to
understand this series of blog posts, as I will be relying on concepts and
terminology introduced there to speed things up. Some background in OpenGL is
helpful, but not required. A good understanding of C is mandatory. If you have
any questions about any of the articles in this series, please reach out to me
directly via <a href="mailto:sir@cmpwn.com">sir@cmpwn.com</a> or to the wlroots team at
<a href="http://webchat.freenode.net/?channels=sway-devel&amp;uio=d4">#sway-devel on irc.freenode.net</a>.</p>
<p>During this series of articles, the compositor we&rsquo;re building will live on
GitHub: <a href="https://github.com/SirCmpwn/mcwayface">Wayland McWayface</a>. Each article
in this series will be presented as a breakdown of a single commit between zero
and a fully functional Wayland compositor. The commit for this article is
<a href="https://github.com/SirCmpwn/mcwayland/commit/f89092e">f89092e</a>.
I&rsquo;m only going to explain the important parts - I suggest you review
the entire commit separately.</p>
<p>Let&rsquo;s get started. First, I&rsquo;m going to define a struct for holding our
compositor&rsquo;s state:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"><span class="gi">+struct mcw_server {
</span><span class="gi">+       struct wl_display *wl_display;
</span><span class="gi">+       struct wl_event_loop *wl_event_loop;
</span><span class="gi">+};
</span></code></pre></div><p>Note: mcw is short for McWayface. We&rsquo;ll be using this acronym throughout the
article series. We&rsquo;ll set one of these aside and initialize the Wayland display
for it<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"> int main(int argc, char **argv) {
<span class="gi">+       struct mcw_server server;
</span><span class="gi">+
</span><span class="gi">+       server.wl_display = wl_display_create();
</span><span class="gi">+       assert(server.wl_display);
</span><span class="gi">+       server.wl_event_loop = wl_display_get_event_loop(server.wl_display);
</span><span class="gi">+       assert(server.wl_event_loop);
</span><span class="gi"></span>        return 0;
 }
</code></pre></div><p>The Wayland display gives us a number of things, but for now all we care about
is the event loop. This event loop is deeply integrated into wlroots, and is
used for things like dispatching signals across the application, being notified
when data is available on various file descriptors, and so on.</p>
<p>Next, we need to create the backend:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"> struct mcw_server {
        struct wl_display *wl_display;
        struct wl_event_loop *wl_event_loop;
 
<span class="gi">+       struct wlr_backend *backend;
</span><span class="gi"></span> };
</code></pre></div><p>The <strong>backend</strong> is our first wlroots concept. The backend is responsible for
abstracting the low level <em>input</em> and <em>output</em> implementations from you. Each
backend can generate zero or more input devices (such as mice, keyboards, etc)
and zero or more output devices (such as monitors on your desk). Backends have
nothing to do with Wayland - their purpose is to help you with the <em>other</em> APIs
you need to use as a Wayland compositor. There are various backends with various
purposes:</p>
<ul>
<li>The <strong>drm</strong> backend utilizes the Linux DRM subsystem to render directly to
your physical displays.</li>
<li>The <strong>libinput</strong> backend utilizes libinput to enumerate and control physical
input devices.</li>
<li>The <strong>wayland</strong> backend creates &ldquo;outputs&rdquo; as windows on another running
Wayland compositors, allowing you to nest compositors. Useful for debugging.</li>
<li>The <strong>x11</strong> backend is similar to the Wayland backend, but opens an x11 window
on an x11 server rather than a Wayland window on a Wayland server.</li>
</ul>
<p>Another important backend is the <strong>multi</strong> backend, which allows you to
initialize several backends at once and aggregate their input and output
devices. This is necessary, for example, to utilize both drm and libinput
simultaneously.</p>
<p>wlroots provides a helper function for automatically choosing the most
appropriate backend based on the user&rsquo;s environment:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff">        server.wl_event_loop = wl_display_get_event_loop(server.wl_display);
        assert(server.wl_event_loop);
 
<span class="gi">+       server.backend = wlr_backend_autocreate(server.wl_display);
</span><span class="gi">+       assert(server.backend);
</span><span class="gi"></span>        return 0;
 }
</code></pre></div><p>I would generally suggest using either the Wayland or X11 backends during
development, especially before we have a way of exiting the compositor. If you
call <code>wlr_backend_autocreate</code> from a running Wayland or X11 session, the
respective backends will be automatically chosen.</p>
<p>We can now start the backend and enter the Wayland event loop:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"><span class="gi">+       if (!wlr_backend_start(server.backend)) {
</span><span class="gi">+               fprintf(stderr, &#34;Failed to start backend\n&#34;);
</span><span class="gi">+               wl_display_destroy(server.wl_display);
</span><span class="gi">+               return 1;
</span><span class="gi">+       }
</span><span class="gi">+
</span><span class="gi">+       wl_display_run(server.wl_display);
</span><span class="gi">+       wl_display_destroy(server.wl_display);
</span><span class="gi"></span>        return 0;
</code></pre></div><p>If you run your compositor at this point, you should see the backend start up
and&hellip; do nothing. It&rsquo;ll open a window if you run from a running Wayland or X11
server. If you run it on DRM, it&rsquo;ll probably do very little and you won&rsquo;t even
be able to switch to another TTY to kill it.</p>
<p>In order to render something, we need to know about the outputs we can render
on. The backend provides a <strong>wl_signal</strong> that notifies us when it gets a new
output. This will happen on startup and as any outputs are hotplugged at
runtime.</p>
<p>Let&rsquo;s add this to our server struct:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"> struct mcw_server {
        struct wl_display *wl_display;
        struct wl_event_loop *wl_event_loop;
 
        struct wlr_backend *backend;
<span class="gi">+
</span><span class="gi">+       struct wl_listener new_output;
</span><span class="gi">+
</span><span class="gi">+       struct wl_list outputs; // mcw_output::link
</span><span class="gi"></span> };
</code></pre></div><p>This adds a <code>wl_listeners</code> which is signalled when new outputs are added. We
also add a <code>wl_list</code> (which is just a linked list provided by libwayland-server)
which we&rsquo;ll later store some state in. To be notified, we must use
<code>wl_signal_add</code>:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff">        assert(server.backend);
 
<span class="gi">+       wl_list_init(&amp;server.outputs);
</span><span class="gi">+
</span><span class="gi">+       server.new_output.notify = new_output_notify;
</span><span class="gi">+       wl_signal_add(&amp;server.backend-&gt;events.new_output, &amp;server.new_output);
</span><span class="gi"></span> 
        if (!wlr_backend_start(server.backend)) {
</code></pre></div><p>We specify here the function to be notified, <code>new_output_notify</code>:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"><span class="gi">+static void new_output_notify(struct wl_listener *listener, void *data) {
</span><span class="gi">+       struct mcw_server *server = wl_container_of(
</span><span class="gi">+                       listener, server, new_output);
</span><span class="gi">+       struct wlr_output *wlr_output = data;
</span><span class="gi">+
</span><span class="gi">+       if (!wl_list_empty(&amp;wlr_output-&gt;modes)) {
</span><span class="gi">+               struct wlr_output_mode *mode =
</span><span class="gi">+                       wl_container_of(wlr_output-&gt;modes.prev, mode, link);
</span><span class="gi">+               wlr_output_set_mode(wlr_output, mode);
</span><span class="gi">+       }
</span><span class="gi">+
</span><span class="gi">+       struct mcw_output *output = calloc(1, sizeof(struct mcw_output));
</span><span class="gi">+       clock_gettime(CLOCK_MONOTONIC, &amp;output-&gt;last_frame);
</span><span class="gi">+       output-&gt;server = server;
</span><span class="gi">+       output-&gt;wlr_output = wlr_output;
</span><span class="gi">+       wl_list_insert(&amp;server-&gt;outputs, &amp;output-&gt;link);
</span><span class="gi">+}
</span></code></pre></div><p>This is a little bit complicated! This function has several roles when dealing
with the incoming <code>wlr_output</code>. When the signal is raised, a pointer to the
listener that was signaled is passed in, as well as the <code>wlr_output</code> which was
created. <code>wl_container_of</code> uses some <code>offsetof</code>-based magic to get the
<code>mcw_server</code> reference from the listener pointer, and we cast <code>data</code> to the
actual type, <code>wlr_output</code>.</p>
<p>The next thing we have to do is set the <strong>output mode</strong>. Some backends (notably
x11 and Wayland) do not support modes, but they are necessary for DRM. Output
modes specify a size and refresh rate supported by the output, such as
<code>1920x1080@60Hz</code>. The body of this if statement just chooses the last one (which
is usually the highest resolution and refresh rate) and applies it to the output
with <code>wlr_output_set_mode</code>. We <em>must</em> set the output mode in order to render to
it.</p>
<p>Then, we set up some state for us to keep track of this output with in our
compositor. I added this struct definition at the top of the file:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"><span class="gi">+struct mcw_output {
</span><span class="gi">+       struct wlr_output *wlr_output;
</span><span class="gi">+       struct mcw_server *server;
</span><span class="gi">+       struct timespec last_frame;
</span><span class="gi">+
</span><span class="gi">+       struct wl_list link;
</span><span class="gi">+};
</span></code></pre></div><p>This will be the structure we use to store any state we have for this output
that is specific to our compositor&rsquo;s needs. We include a reference to the
<code>wlr_output</code>, a reference to the <code>mcw_server</code> that owns this output, and the
time of the last frame, which will be useful later. We also set aside a
<code>wl_list</code>, which is used by libwayland for linked lists.</p>
<p>Finally, we add this output to the server&rsquo;s list of outputs.</p>
<p>We could use this now, but it would leak memory. We also need to handle output
<em>removal</em>, with a signal provided by wlr_output. We add the listener to the
mcw_output struct:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"> struct mcw_output {
        struct wlr_output *wlr_output;
        struct mcw_server *server;
        struct timespec last_frame;
<span class="gi">+
</span><span class="gi">+       struct wl_listener destroy;
</span><span class="gi"></span> 
        struct wl_list link;
 };
</code></pre></div><p>Then we hook it up when the output is added:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff">         wl_list_insert(&amp;server-&gt;outputs, &amp;output-&gt;link);

<span class="gi">+        output-&gt;destroy.notify = output_destroy_notify;
</span><span class="gi">+        wl_signal_add(&amp;wlr_output-&gt;events.destroy, &amp;output-&gt;destroy);
</span><span class="gi"></span> }
</code></pre></div><p>This will call our output_destroy_notify function to handle cleanup when the
output is unplugged or otherwise removed from wlroots. Our handler looks like
this:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"><span class="gi">+static void output_destroy_notify(struct wl_listener *listener, void *data) {
</span><span class="gi">+        struct mcw_output *output = wl_container_of(listener, output, destroy);
</span><span class="gi">+        wl_list_remove(&amp;output-&gt;link);
</span><span class="gi">+        wl_list_remove(&amp;output-&gt;destroy.link);
</span><span class="gi">+        wl_list_remove(&amp;output-&gt;frame.link);
</span><span class="gi">+        free(output);
</span><span class="gi">+}
</span></code></pre></div><p>This one should be pretty self-explanatory.</p>
<p>So, we now have a reference to the output. However, we are still not rendering
anything - if you run the compositor again you&rsquo;ll notice the same behavior. In
order to render things, we have to listen for the <strong>frame signal</strong>. Depending on
the selected mode, the output can only receive new frames at a certain rate. We
keep track of this for you in wlroots, and emit the frame signal when it&rsquo;s time
to draw a new frame.</p>
<p>Let&rsquo;s add a listener to the <code>mcw_output</code> struct for this purpose:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"> struct mcw_output {
        struct wlr_output *wlr_output;
        struct mcw_server *server;
 
        struct wl_listener destroy;
<span class="gi">+       struct wl_listener frame;
</span><span class="gi"></span> 
        struct wl_list link;
 };
</code></pre></div><p>We can then extend <code>new_output_notify</code> to register the listener to the frame
signal:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff">        output-&gt;destroy.notify = output_destroy_notify;
        wl_signal_add(&amp;wlr_output-&gt;events.destroy, &amp;output-&gt;destroy);
<span class="gi">+       output-&gt;frame.notify = output_frame_notify;
</span><span class="gi">+       wl_signal_add(&amp;wlr_output-&gt;events.frame, &amp;output-&gt;frame);
</span><span class="gi"></span> }
</code></pre></div><p>Now, whenever an output is ready for a new frame, <code>output_frame_notify</code> will be
called. We still need to write this function, though. Let&rsquo;s start with the
basics:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"><span class="gi">+static void output_frame_notify(struct wl_listener *listener, void *data) {
</span><span class="gi">+       struct mcw_output *output = wl_container_of(listener, output, frame);
</span><span class="gi">+       struct wlr_output *wlr_output = data;
</span><span class="gi">+}
</span></code></pre></div><p>In order to render anything here, we need to first obtain a wlr_renderer<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>.
We can obtain one from the backend:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"> static void output_frame_notify(struct wl_listener *listener, void *data) {
        struct mcw_output *output = wl_container_of(listener, output, frame);
        struct wlr_output *wlr_output = data;
<span class="gi">+       struct wlr_renderer *renderer = wlr_backend_get_renderer(
</span><span class="gi">+                       wlr_output-&gt;backend);
</span><span class="gi"></span>}
</code></pre></div><p>We can now take advantage of this renderer to draw something on the output.</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"> static void output_frame_notify(struct wl_listener *listener, void *data) {
        struct mcw_output *output = wl_container_of(listener, output, frame);
        struct wlr_output *wlr_output = data;
        struct wlr_renderer *renderer = wlr_backend_get_renderer(
                        wlr_output-&gt;backend);
<span class="gi">+
</span><span class="gi">+       wlr_output_make_current(wlr_output, NULL);
</span><span class="gi">+       wlr_renderer_begin(renderer, wlr_output);
</span><span class="gi">+
</span><span class="gi">+       float color[4] = {1.0, 0, 0, 1.0};
</span><span class="gi">+       wlr_renderer_clear(renderer, color);
</span><span class="gi">+
</span><span class="gi">+       wlr_output_swap_buffers(wlr_output, NULL, NULL);
</span><span class="gi">+       wlr_renderer_end(renderer);
</span><span class="gi"></span> }
</code></pre></div><p>Calling <code>wlr_output_make_current</code> makes the output&rsquo;s OpenGL context &ldquo;current&rdquo;,
and from here you can use OpenGL calls to render to the output&rsquo;s buffer. We call
<code>wlr_renderer_begin</code> to configure some sane OpenGL defaults for us<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>.</p>
<p>At this point we can start rendering. We&rsquo;ll expand more on what you can
do with <code>wlr_renderer</code> later, but for now we&rsquo;ll be satisified with clearing the
output to a solid red color.</p>
<p>When we&rsquo;re done rendering, we call <code>wlr_output_swap_buffers</code> to swap the
output&rsquo;s front and back buffers, committing what we&rsquo;ve rendered to the actual
screen. We call <code>wlr_renderer_end</code> to clean up the OpenGL context and we&rsquo;re
done. Running our compositor now should show you a solid red screen!</p>
<hr>
<p>This concludes today&rsquo;s article. If you take a look at <a href="https://github.com/SirCmpwn/mcwayland/commit/f89092e">the
commit</a> that this article
describes, you&rsquo;ll see that I took it a little further with some code that clears
the display to a different color every frame. Feel free to experiment with
similar changes!</p>
<p>Over the next two articles, we&rsquo;ll finish wiring up the Wayland server and render
a Wayland client on screen. Please look forward to it!</p>
<p style="text-align: right">
    Next &mdash;
    <a href="https://drewdevault.com/2018/02/22/Writing-a-wayland-compositor-part-2.html">
        Part 2: Rigging up the server
    </a>
</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>It&rsquo;s entirely possible to utilize a wlroots backend to make applications which are not Wayland compositors. However, we require a wayland display anyway because the event loop is necessary for a lot of wlroots internals. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>wlr_renderer is optional. When you call wlr_output_make_current, the OpenGL context is made current and from here you can use any approach you prefer. wlr_renderer is provided to help compositors with simple rendering requirements. <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Namely: the viewport and blend mode. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


guid: https://drewdevault.com/2018/02/22/Writing-a-wayland-compositor-part-2.html
title: Writing a Wayland Compositor, Part 2: Rigging up the server
url: https://drewdevault.com/2018/02/22/Writing-a-wayland-compositor-part-2.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>This is the second in a series of articles on the subject of writing a Wayland
compositor from scratch using <a href="https://github.com/swaywm/wlroots">wlroots</a>.
Check out <a href="https://drewdevault.com/2018/02/17/Writing-a-Wayland-compositor-1.html">the first article</a>
if you haven&rsquo;t already. Last time, we ended up with an application which fired
up a wlroots backend, enumerated output devices, and drew some pretty colors on
the screen. Today, we&rsquo;re going to start accepting Wayland client connections,
though we aren&rsquo;t going to be doing much with them yet.</p>
<p>The commit that this article dissects is
<a href="https://github.com/SirCmpwn/mcwayland/commit/b45c651">b45c651</a>.</p>
<p>A quick aside on the nature of these blog posts: it&rsquo;s going to take <em>a lot</em> of
these articles to flesh out our compositor. I&rsquo;m going to be publishing these
more frequently than usual, probably 1-2 per week, and continue posting my usual
articles at the typical rate. Okay? Cool.</p>
<p>So we&rsquo;ve started up the backend and we&rsquo;re rendering something interesting, but
we still aren&rsquo;t running a Wayland server &ndash; Wayland clients aren&rsquo;t connecting to
our application. Adding this is actually quite easy:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"><span class="gu">@@ -113,12 +113,18 @@ int main(int argc, char **argv) {
</span><span class="gu"></span>        server.new_output.notify = new_output_notify;
        wl_signal_add(&amp;server.backend-&gt;events.new_output, &amp;server.new_output);
 
<span class="gi">+       const char *socket = wl_display_add_socket_auto(server.wl_display);
</span><span class="gi">+       assert(socket);
</span><span class="gi">+
</span><span class="gi"></span>        if (!wlr_backend_start(server.backend)) {
                fprintf(stderr, &#34;Failed to start backend\n&#34;);
                wl_display_destroy(server.wl_display);
                return 1;
        }
 
<span class="gi">+       printf(&#34;Running compositor on wayland display &#39;%s&#39;\n&#34;, socket);
</span><span class="gi">+       setenv(&#34;WAYLAND_DISPLAY&#34;, socket, true);
</span><span class="gi">+
</span><span class="gi"></span>        wl_display_run(server.wl_display);
        wl_display_destroy(server.wl_display);
        return 0;
</code></pre></div><p>That&rsquo;s it! If you run McWayface again, it&rsquo;ll print something like this:</p>
<pre><code>Running compositor on wayland display 'wayland-1'
</code></pre><p><a href="https://cgit.freedesktop.org/wayland/weston/">Weston</a>, the Wayland reference
compositor, includes a number of simple reference clients. We can use
<code>weston-info</code> to connect to our server and list the <strong>globals</strong>:</p>
<pre><code>$ WAYLAND_DISPLAY=wayland-1 weston-info
interface: 'wl_drm', version: 2, name: 1
</code></pre><p>If you recall from my <a href="https://drewdevault.com/2017/06/10/Introduction-to-Wayland.html">Introduction to
Wayland</a>, the Wayland server exports a
list of <strong>globals</strong> to clients via the Wayland registry. These globals provide
interfaces the client can utilize to interact with the server. We get <code>wl_drm</code>
for free with wlroots, but we have not actually wired up anything useful yet.
Wlroots provides many &ldquo;types&rdquo;, of which the majority are implementations of
Wayland global interfaces like this.</p>
<p>Some of the wlroots implementations require some rigging from you, but several
of them just take care of themselves. Rigging these up is easy:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff">        printf(&#34;Running compositor on wayland display &#39;%s&#39;\n&#34;, socket);
        setenv(&#34;WAYLAND_DISPLAY&#34;, socket, true);
<span class="gi">+
</span><span class="gi">+       wl_display_init_shm(server.wl_display);
</span><span class="gi">+       wlr_gamma_control_manager_create(server.wl_display);
</span><span class="gi">+       wlr_screenshooter_create(server.wl_display);
</span><span class="gi">+       wlr_primary_selection_device_manager_create(server.wl_display);
</span><span class="gi">+       wlr_idle_create(server.wl_display);
</span><span class="gi"></span> 
        wl_display_run(server.wl_display);
        wl_display_destroy(server.wl_display);
</code></pre></div><p>Note that some of these interfaces are not necessarily ones that you typically
would want to expose to all Wayland clients - screenshooter, for example, is
something that should be secured. We&rsquo;ll get to security in a later article. For
now, if we run <code>weston-info</code> again, we&rsquo;ll see a few more globals have appeared:</p>
<pre><code>$ WAYLAND_DISPLAY=wayland-1 weston-info
interface: 'wl_shm', version: 1, name: 3
	formats: XRGB8888 ARGB8888
interface: 'wl_drm', version: 2, name: 1
interface: 'gamma_control_manager', version: 1, name: 2
interface: 'orbital_screenshooter', version: 1, name: 3
interface: 'gtk_primary_selection_device_manager', version: 1, name: 4
interface: 'org_kde_kwin_idle', version: 1, name: 5
</code></pre><p>You&rsquo;ll find that wlroots implements a variety of protocols from a variety of
sources - here we see protocols from Orbital, GTK, and KDE represented. Wlroots
includes an example client for the orbital screenshooter - we can use it now to
take a screenshot of our compositor:</p>
<pre><code>$ WAYLAND_DISPLAY=wayland-1 ./examples/screenshot
cannot set buffer size
</code></pre><p>Ah, this is a problem - you may have noticed that we don&rsquo;t have any wl_output
globals, which the screenshooter client relies on to figure out the resolution
of the screenshot buffer. We can add these, too:</p>
<div class="highlight"><pre class="chroma"><code class="language-diff" data-lang="diff"><span class="gu">@@ -95,6 +99,8 @@ static void new_output_notify(struct wl_listener *listener, void *data) {
</span><span class="gu"></span>        wl_signal_add(&amp;wlr_output-&gt;events.destroy, &amp;output-&gt;destroy);
        output-&gt;frame.notify = output_frame_notify;
        wl_signal_add(&amp;wlr_output-&gt;events.frame, &amp;output-&gt;frame);
<span class="gi">+
</span><span class="gi">+       wlr_output_create_global(wlr_output);
</span><span class="gi"></span> }
</code></pre></div><p>Running <code>weston-info</code> again will give us some info about our outputs now:</p>
<pre><code>$ WAYLAND_DISPLAY=wayland-1 weston-info
interface: 'wl_drm', version: 2, name: 1
interface: 'wl_output', version: 3, name: 2
	x: 0, y: 0, scale: 1,
	physical_width: 0 mm, physical_height: 0 mm,
	make: 'wayland', model: 'wayland',
	subpixel_orientation: unknown, output_transform: normal,
	mode:
		width: 952 px, height: 521 px, refresh: 0.000 Hz,
		flags: current
interface: 'wl_shm', version: 1, name: 3
	formats: XRGB8888 ARGB8888
interface: 'gamma_control_manager', version: 1, name: 4
interface: 'orbital_screenshooter', version: 1, name: 5
interface: 'gtk_primary_selection_device_manager', version: 1, name: 6
interface: 'org_kde_kwin_idle', version: 1, name: 7
</code></pre><p>Now we can take that screenshot! Give it a shot (heh)!</p>
<p>We&rsquo;re getting close to the good stuff now. The next article is going to
introduce the concept of <strong>surfaces</strong>, and we will use them to render our first
window. If you had any trouble with this article, please reach out to me at
<a href="mailto:sir@cmpwn.com">sir@cmpwn.com</a> or to the wlroots team at
<a href="http://webchat.freenode.net/?channels=sway-devel&amp;uio=d4">#sway-devel</a>.</p>
<p style="float: right">
    Next &mdash;
    <a href="https://drewdevault.com/2018/02/28/Writing-a-wayland-compositor-part-3.html">
        Part 3: Rendering a window
    </a>
</p>
<p>
    Previous &mdash;
    <a href="https://drewdevault.com/2018/02/17/Writing-a-Wayland-compositor-1.html">
        Part 1: Hello wlroots
    </a>
</p>


guid: https://drewdevault.com/2018/02/24/The-road-to-sustainable-FOSS.html
title: The path to sustainably working on FOSS full-time
url: https://drewdevault.com/2018/02/24/The-road-to-sustainable-FOSS.html
feedurl: https://drewdevault.com/blog/index.xml
content:
<p>This is an article I didn&rsquo;t think I&rsquo;d be writing any time soon. I&rsquo;ve aspired to
work full-time on my free and open source software projects for a long time now,
but I have never expected that it could work. However, as of this week, I
finally have enough recurring donation revenue to break even on FOSS, and I&rsquo;ve
started to put the extra cash away. I needed to set the next donation goal and
ran the numbers to figure out what it takes to work on FOSS full-time.</p>
<p>Let me start with some context. I like to say &ldquo;one-time donations buy pizza,
but recurring donations buy sustainable FOSS development&rdquo;. One-time donations
provide no financial security, so to date, (almost) all of my FOSS work has been
done in my spare time, and I&rsquo;ve had to spend most of my time working on
proprietary software to make a living. This is the case for many other free
software advocates as well. Short of large grants on the scale of several
tens of thousands of dollars, if you want to get your rent paid and put food on
the table you need to be able to rely on something consistent.</p>
<p>Some projects (e.g. Docker, Gitlab) have a compelling product in the market and
can build a company around their open source product. Some projects fulfill a
tangible need for some other business (such as writing software they depend on),
and for these projects large corporate sponsorships are often possible. However,
other kinds of projects (including most of my own) often have to rely on their
users for donations, and this has traditionally been a pretty dubious prospect.
In August of 2017, I was making $0 per month in recurring donations to
<a href="https://drewdevault.com/donate">fosspay</a>, down from an all-time peak of $20 per
month. When I was researching the possibility of starting a Patreon campaign,
the norm was less than $50/month even for the most successful open source
campaigns. As you can imagine, I was somewhat pessimistic.</p>
<p>To my happy surprise, recurring donations to open source projects have taken
off, both for me and many others. It&rsquo;s amazing. After years of failing to earn a
substantial income from open source, as of today I&rsquo;m receiving $547.74 per month
from three donation platforms (<a href="https://drewdevault.com/donate">fosspay</a>,
<a href="https://liberapay.com/SirCmpwn">LiberaPay</a>, and
<a href="https://patreon.com/SirCmpwn">Patreon</a>). What&rsquo;s amazing is that because the
income comes from from several platforms and is distributed across over 80
donators, I can feel confident in the security of this model. There are no
whales whose donations I have to live in fear of losing. There is no single
platform that I have to worry about going under or dramatically <a href="https://blog.patreon.com/not-rolling-out-fees-change/">changing their
fee structure</a>. This is
unprecedented - we&rsquo;re truly seeing the age of user-supported FOSS begin.</p>
<p>I want to provide some transparency on how I set my goals and where the money
goes. You might be surprised to have heard me say that I&rsquo;m only &ldquo;breaking even&rdquo;
on open source at $500/month! Many projects can run on a leaner budget, but
because I maintain so many different projects, I have different infrastructure
requirements. This mainly includes domains and servers for CI, project hosting,
releases, etc. At my scale, it&rsquo;s most cost-effective for me to self-host my own
dedicated servers in a local datacenter here in Philadelphia. This costs me
$380/month at the moment for 5U including power and network. I&rsquo;m not done moving
my legacy infrastructure into the new datacenter, though, so I&rsquo;m still paying
for some virtual private servers. As I migrate these, I will be reinvesting the
money saved into upgrading the new infrastructure.</p>
<p>The next question is where to go from here. I have set my full-time goal at
$6,000 per month, which works out to $72,000 per year pre-tax,
pre-infrastructure expenses. This number is a lofty goal, and one that I expect
won&rsquo;t be met for a long time, if at all. This number is based on several
factors: cost of living, financial security, and taxes. The number is a
significant decrease from what I earn today, but it is enough to meet each of
these criteria. Let&rsquo;s break it down.</p>
<p>Right now, I live in a pretty nice apartment in center city Philadelphia, which
costs me about $1700 per month. There are cheaper areas, but I make a
comfortable salary at my current job, which allows me to buy a nicer place. If
working on FOSS full-time appears viable, I will move to a cheaper location when
my lease is up and adjust the goal accordingly (I will probably move to a
cheaper location when my lease is up regardless, actually). Because I&rsquo;m locked
into my lease (among other reasons), I did not factor major lifestyle changes
like moving to a cheaper location into the goal. Other costs of living, such as
food and necessities, work out to about $1000 per month.</p>
<p>The other concern is financial security. I am lucky to live a comfortable life
today, but that is a result of hard lessons learned and has not always been the
case. I cannot focus on FOSS if I&rsquo;m only earning just enough to cover my
expenses. Any major change in my life circumstances, such as a medical
emergency, natural disaster, or even something as benign as my computer breaking
down, would be a serious problem. Therefore, for me to consider working
full-time on anything, the earnings have to allow me to save money. To this end,
my earnings floor is at least 1.5x my expenditures. Some people think a more
liberal ratio is fine, but I&rsquo;m a bit more conservative - I used to really
struggle to make ends meet. This raises the total to around $4000 per month.</p>
<p>Add to this infrastructure costs we already talked about, and the total becomes
$4500 per month. Now we have to consider tax. If we look up the current <a href="https://en.wikipedia.org/wiki/Tax_bracket#2018_tax_brackets_under_current_law">tax
brackets in the United
States</a>
and do some guesswork, we can estimate that I&rsquo;ll land in the 22% bracket under
this model. If I need my take-home to be $4500, we can divide that by 78% and
arrive at the total: $5769 per month<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. Round it up to $6000 and this is our
goal.</p>
<p>These numbers are pretty high. I understand many people, including some of those
who donate to me, are much less fortunate than I. My lifestyle is a reflection
of my assumption that the open source donation model does not provide a
sustainable source of income. Based on this, I&rsquo;ve focused my career on paid
proprietary software development, which pays very competitively in the United
States. The privileges afforded by this have shaped my costs of living. Rather
than make up a number smaller than my actual expenditures, I prefer to be honest
with you about this.</p>
<p>This doesn&rsquo;t necessarily have to remain the case forever. As my income from
donations increase, utilizing them as a primary source of income becomes more
feasible, and I am prepared to reorient my life with this in mind. You can
expect my donation goal to <em>decrease</em> as the number of donations <em>increases</em>.
This will probably take a long time, on the scale of years. My housing situation
and costs of living in Philadelphia will change during this time - I might not
stay in Philadelphia, I might have to change jobs, etc. It&rsquo;s difficult to set
a more optimistic goal today that will prove correct when its met. For that
reason, my goal is adjusted with respect to my current conditions, not the
ideal.</p>
<p>So that&rsquo;s how it shakes out! I&rsquo;m glad we can finally have this conversation, and
I&rsquo;m incredibly thankful for your support. Thank you for everything, and I&rsquo;m
looking forward to making even more cool stuff for you in the future.</p>
<section class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1" role="doc-endnote">
<p>Correction: that&rsquo;s not how taxes work, but the simplified version gives us a more conservative number - which is a good thing when your livelihood is at stake. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
</li>
</ol>
</section>


