353 lines
15 KiB
HTML
353 lines
15 KiB
HTML
<!doctype html>
|
|
<html>
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
|
|
|
<meta name="theme-color" content="#00cc00">
|
|
<link rel="icon" type="image/png" sizes="192x192" href="https://tilde.team/apple-touch-icon-precomposed.png">
|
|
<link rel="icon" type="image/png" sizes="96x96" href="https://tilde.team/favicon-96x96.png">
|
|
|
|
<link rel="stylesheet" href="https://tilde.team/css/dracula.css">
|
|
<link rel="stylesheet" href="extra.css">
|
|
|
|
<link rel="alternate" type="application/rss+xml" title="subscribe to this page..." href="feed.rss" />
|
|
<title>blog // ~ben</title>
|
|
</head><body>
|
|
<div class="container">
|
|
|
|
<div id="divbodyholder">
|
|
<div class="headerholder"><div class="header">
|
|
<div id="title">
|
|
<h1 class="nomargin"><a class="ablack" href="https://tilde.team/~ben/blog/index.html">blog // ~ben</a></h1>
|
|
<div id="description">a blog about tildes and other things</div>
|
|
</div></div></div>
|
|
<div id="divbody"><div class="content">
|
|
<h3><a class="ablack" href="proactive-redundancy.html">
|
|
proactive redundancy
|
|
</a></h3>
|
|
<!-- bashblog_timestamp: #201811151839.26# -->
|
|
<div class="subtitle">November 15, 2018 —
|
|
~ben
|
|
</div>
|
|
<!-- text begin -->
|
|
|
|
<p>after the <a href="november-13-post-mortem.html">fiasco</a> earlier this week, i've been taking steps to minimize
|
|
the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage. </p>
|
|
|
|
<p>the first thing that i set up was a handful of additional ircd nodes: see <a href="https://tilde.chat/wiki/?page=servers">the tilde.chat wiki</a> for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team. </p>
|
|
|
|
<p>i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. <code>host tilde.chat</code> will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of <code>{your,team,bsd,slash}.tilde.chat</code>. </p>
|
|
|
|
<p>this creates the additional problem that visiting the <a href="https://tilde.chat">tilde.chat site</a> will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to <a href="https://tildegit.org/tildeverse/tilde.chat/issues/8">debug</a>. the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see <code>host chat.freenode.net</code> for the list of servers).</p>
|
|
|
|
<p>i'm not sure how to make any of the other services more resilient. it's something that i have been and will continue to research moving forward.</p>
|
|
|
|
<p>the other main step that i have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in <a href="https://tools.ietf.org/html/rfc1918">rfc 1918</a>.</p>
|
|
|
|
<p>i'd like to consider at least this risk to be mitigated.</p>
|
|
|
|
<p>thanks for reading,</p>
|
|
|
|
<p>~ben</p>
|
|
|
|
<p><strong>update</strong>: the round robin host is now <em>irc</em>.tilde.chat, which resolves the site issues that we were having, due to the duplicated deployments.</p>
|
|
|
|
<p>tags: <a href='tag_sysadmin.html'>sysadmin</a>, <a href='tag_tilde.html'>tilde</a></p>
|
|
|
|
<!-- text end -->
|
|
<h3><a class="ablack" href="november-13-post-mortem.html">
|
|
november 13 post mortem
|
|
</a></h3>
|
|
<!-- bashblog_timestamp: #201811132020.33# -->
|
|
<div class="subtitle">November 13, 2018 —
|
|
~ben
|
|
</div>
|
|
<!-- text begin -->
|
|
|
|
<p>we had something of an outage on november 13, 2018 on tilde.team. </p>
|
|
|
|
<p>i awoke, not suspecting anything to be amiss. as soon as i logged in to check my email and irc mentions, it became clear.</p>
|
|
|
|
<p>tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.</p>
|
|
|
|
<blockquote>
|
|
<p>We have indications that there was an attack from your server.
|
|
Please take all necessary measures to avoid this in the future and to solve the issue.</p>
|
|
</blockquote>
|
|
|
|
<p>at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.</p>
|
|
|
|
<p>when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the <a href="https://mail.tilde.team">webmail</a> to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!</p>
|
|
|
|
<p>here, i launch in to full debugging mode: what command was it? who ran it? </p>
|
|
|
|
<p>search <code>~/.bash_history</code> per user was not very successful. nothing i could find was related to net or map. i had checked <code>sudo grep nmap /home/*/.bash_history</code> and many other commands. </p>
|
|
|
|
<p>at this point, i had connected with other ~teammates across other irc nets (<a href="https://hashbang.sh/">#!</a>, <a href="https://tilde.town">~town</a>, etc). among suggestions to check <code>/var/log/syslog</code>, <code>/var/log/kern.log</code>, and <code>dmesg</code>, i finally decided to check <code>ps</code>. <code>ps -ef | grep nmap</code> yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for <a href="/~fosslinux/">~fosslinux</a>.</p>
|
|
|
|
<p>i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police <code>nmap</code> when it isn't scanning on every port?</p>
|
|
|
|
<p>after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that <a href="/~fosslinux/">~fosslinux</a> had only run <code>nmap</code> for addresses in the <code>10.0.0.0/8</code> space. the <code>10/8</code> address space is intended to not be addressable outside the local space. how could <a href="https://hetzner.com">hetzner</a> have found out about a localhost network probe!?</p>
|
|
|
|
<p>finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.</p>
|
|
|
|
<p>it's definitely time to research redundancy options!</p>
|
|
|
|
<p>tags: <a href='tag_post-mortem.html'>post-mortem</a>, <a href='tag_linux.html'>linux</a>, <a href='tag_sysadmin.html'>sysadmin</a></p>
|
|
|
|
<!-- text end -->
|
|
<h3><a class="ablack" href="italy.html">
|
|
italy
|
|
</a></h3>
|
|
<!-- bashblog_timestamp: #201809201732.33# -->
|
|
<div class="subtitle">September 20, 2018 —
|
|
~ben
|
|
</div>
|
|
<!-- text begin -->
|
|
|
|
<p>i just got back from a 10-day backpacking trip to italy and i'd like to share some of the photos i took!</p>
|
|
|
|
<p>the travel plan was rome -> venice -> florence -> naples -> pompei/vesuvius -> capri -> amalfi</p>
|
|
|
|
<p>this is the roman forum (with colosseum in the background) as seen from the palatine.</p>
|
|
|
|
<p><img src="https://bhh.sh/pub/photos/italy/roman-forum.jpg" alt="" title="" /></p>
|
|
|
|
<p class="readmore"><a href="./italy.html">read more...</a></p>
|
|
<h3><a class="ablack" href="utterances.html">
|
|
utterances
|
|
</a></h3>
|
|
<!-- bashblog_timestamp: #201809052134.13# -->
|
|
<div class="subtitle">September 05, 2018 —
|
|
~ben
|
|
</div>
|
|
<!-- text begin -->
|
|
|
|
<p>i somehow stumbled upon <a href="https://utteranc.es">utterances</a> today at lunch. (i think someone had it forked on their github page).</p>
|
|
|
|
<p>no matter how i found it, i still decided to add it to my blog here with <a href="https://tildegit.org/team/bashblog">bashblog</a>. utterances is a commenting system that leverages github issues. so, for example a comment on <a href="https://tilde.team/~ben/blog/upsides-of-new-dns-nameservers.html">a post</a> shows up on github <a href="https://github.com/benharri/tilde/issues/1#issuecomment-418732788">like this</a>.</p>
|
|
|
|
<p>now we just need to figure out if it can be pointed at a gitea instance like <a href="https://tildegit.org">tildegit</a>. might be time for a PR!</p>
|
|
|
|
<p>tags: <a href='tag_blog.html'>blog</a></p>
|
|
|
|
|
|
|
|
<!-- text end -->
|
|
<h3><a class="ablack" href="no-more-google.html">
|
|
no more google
|
|
</a></h3>
|
|
<!-- bashblog_timestamp: #201808142336.05# -->
|
|
<div class="subtitle">August 14, 2018 —
|
|
~ben
|
|
</div>
|
|
<!-- text begin -->
|
|
|
|
<p>not sure if this is appropriately tagged, but i didn't feel like making a new
|
|
one.</p>
|
|
|
|
<p>i figured i should probably get some notes down about moving off google.</p>
|
|
|
|
<p>to start, i'll get a list of the things i was able to easily replace:</p>
|
|
|
|
<ul>
|
|
<li>gmail => <a href="https://tilde.team/wiki/?page=email">@tilde.team mail</a></li>
|
|
<li>google drive => <a href="https://syncthing.net">syncthing</a> (with a persistent node running on my personal vps)</li>
|
|
</ul>
|
|
|
|
<p>i'm still using:</p>
|
|
|
|
<ul>
|
|
<li>gplay music/youtube</li>
|
|
<li>google maps (open streetmap isn't good enough to replace it)</li>
|
|
<li>google photos - but this is going to be replaced long-term with syncthing</li>
|
|
</ul>
|
|
|
|
<p>tags: <a href='tag_linux.html'>linux</a>, <a href='tag_net-neutrality.html'>net-neutrality</a></p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<!-- text end -->
|
|
<h3><a class="ablack" href="upsides-of-new-dns-nameservers.html">
|
|
upsides of new dns nameservers
|
|
</a></h3>
|
|
<!-- bashblog_timestamp: #201808141505.38# -->
|
|
<div class="subtitle">August 14, 2018 —
|
|
~ben
|
|
</div>
|
|
<!-- text begin -->
|
|
|
|
<ul>
|
|
<li>no more google</li>
|
|
<li>no more google</li>
|
|
<li>automated certbot validation for letsencrypt wildcard certs!! no more manual TXT records every three months!</li>
|
|
</ul>
|
|
|
|
<p>tags: <a href='tag_dns.html'>dns</a>, <a href='tag_linux.html'>linux</a>, <a href='tag_tilde.html'>tilde</a></p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<!-- text end -->
|
|
<h3><a class="ablack" href="dns-shenanigans-post-mortem.html">
|
|
dns shenanigans post-mortem
|
|
</a></h3>
|
|
<!-- bashblog_timestamp: #201808141503.49# -->
|
|
<div class="subtitle">August 14, 2018 —
|
|
~ben
|
|
</div>
|
|
<!-- text begin -->
|
|
|
|
<p>let's start by saying i probably should have done a bit more research before
|
|
diving head-first into this endeavor.</p>
|
|
|
|
<p>i've been thinking about transferring my domains off google domains for some
|
|
time now, as part of my personal goal to self host and limit my dependence on
|
|
google and other large third-party monstrosities. along that line, i asked for
|
|
registrar recommendations. <a href="https://tomasino.tilde.team">~tomasino</a> responded
|
|
with <a href="https://namesilo.com">namesilo</a>. i found that they had $3.99 registrations
|
|
for .team and .zone domains, which is 1/10th the cost of the $40 registration
|
|
on google domains.</p>
|
|
|
|
<p>i started out by getting the list of domains from the google console. 2 or 3
|
|
of them had been registered within the last 60 days, so i wasn't able to
|
|
transfer those just yet. i grabbed all the domain unlock codes and dropped
|
|
them into namesilo. i failed to realize that the dns panel on google domains
|
|
would disappear as soon as it went through, but more importantly that the
|
|
nameservers would be left pointing to the old defunct google domains ones.</p>
|
|
|
|
<p>i updated the nameservers as soon as i realized this error from the namesilo
|
|
panel. some of the domains propagated quickly. others, not so much. tilde.team
|
|
was still in a state of flux between the old and new nameservers.</p>
|
|
|
|
<p>in a rush to get the dns problem fixed, and under recommendation from several
|
|
people on irc, i decided to switch the nameservers for tilde.team and tilde.zone
|
|
to cloudflare, leaving another layer of flux for the dns to be stuck in...</p>
|
|
|
|
<p>of the five domains that i moved to cloudflare, 3 returned with a dnssec error,
|
|
claiming that i needed to remove the DS record from that zone. d'oh!</p>
|
|
|
|
<p>i removed the dnssec from those affected domains, so we should be good to go
|
|
as soon as it all propagates through the fickle beast that is dns.</p>
|
|
|
|
<p>tags: <a href='tag_linux.html'>linux</a>, <a href='tag_sysadmin.html'>sysadmin</a>, <a href='tag_tilde.html'>tilde</a>, <a href='tag_dns.html'>dns</a></p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<!-- text end -->
|
|
<h3><a class="ablack" href="lxd-networking-and-additional-ips.html">
|
|
lxd networking and additional IPs
|
|
</a></h3>
|
|
<!-- bashblog_timestamp: #201807261534.50# -->
|
|
<div class="subtitle">July 26, 2018 —
|
|
~ben
|
|
</div>
|
|
<!-- text begin -->
|
|
|
|
<p>now that tilde.team is on a fancy-shmancy new dedi server, i've tried to get a secondary IP address
|
|
assigned to a lxd container (which i plan to use for my personal stuff). lxd shows that the secondary
|
|
IP is being picked up by that container, but i'm still seeing the host machine's IP as the external
|
|
address.</p>
|
|
|
|
<p>i'm not sure how i'll need to configure the network settings on the host machine (now that we're running
|
|
ubuntu 18.04 and it uses netplan for configs and not /etc/network/interfaces). another confusing thing is
|
|
that the main config in /etc/netplan says that the network config is handled by systemd-networkd...</p>
|
|
|
|
<p>at least i have through the end of the year when my current vps runs out to get this up and running.</p>
|
|
|
|
<p>ping me on <a href="https://tilde.chat">irc</a> or <a href="mailto:ben@tilde.team">email</a> if you have experience with this.</p>
|
|
|
|
<p>tags: <a href='tag_linux.html'>linux</a>, <a href='tag_sysadmin.html'>sysadmin</a>, <a href='tag_ubuntu.html'>ubuntu</a></p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<!-- text end -->
|
|
<h3><a class="ablack" href="dotfiles.html">
|
|
dotfiles
|
|
</a></h3>
|
|
<!-- bashblog_timestamp: #201807221926.26# -->
|
|
<div class="subtitle">July 22, 2018 —
|
|
~ben
|
|
</div>
|
|
<!-- text begin -->
|
|
|
|
<p>finally got around to updating my <a href="https://git.tilde.team/ben/dotfiles">dotfiles</a> to use gnu stow.
|
|
i adapted <a href="https://github.com/jamestomasino/dotfiles/blob/master/Makefile">~tomasino's makefile</a>
|
|
for use with the configs that i'm keeping with it.</p>
|
|
|
|
<p>now i just need to figure out why my ssh config doesn't copy/symlink my config to ~/.ssh when it
|
|
already exists.</p>
|
|
|
|
<p>tags: <a href='tag_linux.html'>linux</a>, <a href='tag_dotfiles.html'>dotfiles</a>, <a href='tag_git.html'>git</a></p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<!-- text end -->
|
|
<h3><a class="ablack" href="bashblog-and-your-gopherhole.html">
|
|
bashblog and your gopherhole
|
|
</a></h3>
|
|
<!-- bashblog_timestamp: #201807221144.03# -->
|
|
<div class="subtitle">July 22, 2018 —
|
|
~ben
|
|
</div>
|
|
<!-- text begin -->
|
|
|
|
<p>i've created <a href="https://git.tildeverse.org/meta/bashblog">a repo</a> for the tilde.team customizations to <a href="https://github.com/cfenollosa/bashblog">bashblog</a>.</p>
|
|
|
|
<p>it will now make sure that your ~/public_gopher exists and symlink your blog into it with a nice gophermap to list all the markdown styled posts.</p>
|
|
|
|
<p>try it out and let me know if there are any problems!</p>
|
|
|
|
<p>tags: <a href='tag_tilde.html'>tilde</a>, <a href='tag_blog.html'>blog</a></p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<!-- text end -->
|
|
<div id="all_posts"><a href="all_posts.html">archive</a> — <a href="all_tags.html">all tags</a> — <a href="feed.rss">rss</a></div>
|
|
</div>
|
|
<div id="footer">CC by-nc-nd <a href="https://tilde.team/~ben/">~ben</a> — <a href="mailto:ben@tilde.team">ben@tilde.team</a><br/>
|
|
generated with <a href="https://tildegit.org/team/bashblog">bashblog</a>, a single bash script to easily create blogs like this one</div>
|
|
</div></div>
|
|
<script src="https://utteranc.es/client.js"
|
|
repo="benharri/tilde"
|
|
issue-term="title"
|
|
crossorigin="anonymous"
|
|
theme="github-dark"
|
|
async>
|
|
</script>
|
|
|
|
</div>
|
|
<br>
|
|
</body></html>
|