396 lines
12 KiB
HTML
396 lines
12 KiB
HTML
<!doctype html>
|
||
<html>
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
||
|
||
<meta name="theme-color" content="#00cc00">
|
||
<link rel="icon" type="image/png" sizes="192x192" href="https://tilde.team/apple-touch-icon-precomposed.png">
|
||
<link rel="icon" type="image/png" sizes="96x96" href="https://tilde.team/favicon-96x96.png">
|
||
|
||
<link rel="stylesheet" href="https://tilde.team/css/hacker.css">
|
||
<link rel="stylesheet" href="extra.css">
|
||
|
||
<link rel="alternate" type="application/rss+xml" title="subscribe to this page..." href="feed.rss" />
|
||
<title>blog // ~ben</title>
|
||
</head><body>
|
||
<div class="container">
|
||
|
||
<div id="divbodyholder">
|
||
<div class="headerholder"><div class="header">
|
||
<div id="title">
|
||
<h1 class="nomargin"><a class="ablack" href="https://tilde.team/~ben/blog/index.html">blog // ~ben</a></h1>
|
||
<div id="description">a blog about tildes and other things</div>
|
||
</div></div></div>
|
||
<div id="divbody"><div class="content">
|
||
<h3><a class="ablack" href="proactive-redundancy.html">
|
||
proactive redundancy
|
||
</a></h3>
|
||
<!-- bashblog_timestamp: #201811151839.26# -->
|
||
<div class="subtitle">November 15, 2018 —
|
||
~ben
|
||
</div>
|
||
<!-- text begin -->
|
||
|
||
<p>after the <a href="november-13-post-mortem.html">fiasco</a> earlier this week, i've been taking steps to minimize
|
||
the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage. </p>
|
||
|
||
<p>the first thing that i set up was a handful of additional ircd nodes: see <a href="https://tilde.chat/wiki/?page=servers">the tilde.chat wiki</a> for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team. </p>
|
||
|
||
<p>i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. <code>host tilde.chat</code> will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of <code>{your,team,bsd,slash}.tilde.chat</code>. </p>
|
||
|
||
<p>this creates the additional problem that visiting the <a href="https://tilde.chat">tilde.chat site</a> will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to <a href="https://tildegit.org/tildeverse/tilde.chat/issues/8">debug</a>. the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see <code>host chat.freenode.net</code> for the list of servers).</p>
|
||
|
||
<p>i'm not sure how to make any of the other services more resilient. it's something that i have been and will continue to research moving forward.</p>
|
||
|
||
<p>the other main step that i have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in <a href="https://tools.ietf.org/html/rfc1918">rfc 1918</a>.</p>
|
||
|
||
<p>i'd like to consider at least this risk to be mitigated.</p>
|
||
|
||
<p>thanks for reading,</p>
|
||
|
||
<p>~ben</p>
|
||
|
||
<p>tags: <a href='tag_sysadmin.html'>sysadmin</a>, <a href='tag_tilde.html'>tilde</a></p>
|
||
<!-- text end -->
|
||
<h3><a class="ablack" href="november-13-post-mortem.html">
|
||
november 13 post mortem
|
||
</a></h3>
|
||
<!-- bashblog_timestamp: #201811132020.33# -->
|
||
<div class="subtitle">November 13, 2018 —
|
||
~ben
|
||
</div>
|
||
<!-- text begin -->
|
||
|
||
<p>we had something of an outage on november 13, 2018 on tilde.team. </p>
|
||
|
||
<p>i awoke, not suspecting anything to be amiss. as soon as i logged in to check my email and irc mentions, it became clear.</p>
|
||
|
||
<p>tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.</p>
|
||
|
||
<blockquote>
|
||
<p>We have indications that there was an attack from your server.
|
||
Please take all necessary measures to avoid this in the future and to solve the issue.</p>
|
||
</blockquote>
|
||
|
||
<p>at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.</p>
|
||
|
||
<p>when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the <a href="https://mail.tilde.team">webmail</a> to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!</p>
|
||
|
||
<p>here, i launch in to full debugging mode: what command was it? who ran it? </p>
|
||
|
||
<p>search <code>~/.bash_history</code> per user was not very successful. nothing i could find was related to net or map. i had checked <code>sudo grep nmap /home/*/.bash_history</code> and many other commands. </p>
|
||
|
||
<p>at this point, i had connected with other ~teammates across other irc nets (<a href="https://hashbang.sh/">#!</a>, <a href="https://tilde.town">~town</a>, etc). among suggestions to check <code>/var/log/syslog</code>, <code>/var/log/kern.log</code>, and <code>dmesg</code>, i finally decided to check <code>ps</code>. <code>ps -ef | grep nmap</code> yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for <a href="/~fosslinux/">~fosslinux</a>.</p>
|
||
|
||
<p>i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police <code>nmap</code> when it isn't scanning on every port?</p>
|
||
|
||
<p>after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that <a href="/~fosslinux/">~fosslinux</a> had only run <code>nmap</code> for addresses in the <code>10.0.0.0/8</code> space. the <code>10/8</code> address space is intended to not be addressable outside the local space. how could <a href="https://hetzner.com">hetzner</a> have found out about a localhost network probe!?</p>
|
||
|
||
<p>finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.</p>
|
||
|
||
<p>it's definitely time to research redundancy options!</p>
|
||
|
||
<p>tags: <a href='tag_post-mortem.html'>post-mortem</a>, <a href='tag_linux.html'>linux</a>, <a href='tag_sysadmin.html'>sysadmin</a></p>
|
||
<!-- text end -->
|
||
<h3><a class="ablack" href="quote-of-the-day.html">
|
||
quote of the day
|
||
</a></h3>
|
||
<!-- bashblog_timestamp: #201802130955.06# -->
|
||
<div class="subtitle">February 13, 2018 —
|
||
~ben
|
||
</div>
|
||
<!-- text begin -->
|
||
|
||
<p>Be Alert! - the world needs more Lerts.</p>
|
||
|
||
<p>Tags: <a href='tag_quotes.html'>quotes</a></p>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<!-- text end -->
|
||
<h3><a class="ablack" href="thought-of-the-day14302.html">
|
||
thought of the day
|
||
</a></h3>
|
||
<!-- bashblog_timestamp: #201711271656.36# -->
|
||
<div class="subtitle">November 27, 2017 —
|
||
ben
|
||
</div>
|
||
<!-- text begin -->
|
||
|
||
<p>why do they tell us to use the stairs in case of fire? shouldn't we be using a fire extinguisher?</p>
|
||
|
||
<p>Tags: <a href='tag_thought-of-the-day.html'>thought-of-the-day</a></p>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<!-- text end -->
|
||
<h3><a class="ablack" href="thought-of-the-day2227.html">
|
||
thought of the day
|
||
</a></h3>
|
||
<!-- bashblog_timestamp: #201712031347.36# -->
|
||
<div class="subtitle">December 03, 2017 —
|
||
ben
|
||
</div>
|
||
<!-- text begin -->
|
||
|
||
<p>everything in the universe either is or isn't a potato.</p>
|
||
|
||
<p>Tags: <a href='tag_thought-of-the-day.html'>thought-of-the-day</a>, <a href='tag_words.html'>words</a></p>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<!-- text end -->
|
||
<h3><a class="ablack" href="thought-of-the-day22873.html">
|
||
Thought of the Day
|
||
</a></h3>
|
||
<!-- bashblog_timestamp: #201711271654.07# -->
|
||
<div class="subtitle">November 27, 2017 —
|
||
ben
|
||
</div>
|
||
<!-- text begin -->
|
||
|
||
<p>“Arguing with religious people – It’s like playing chess with a pigeon; no matter how good I am at chess, the pigeon is just going to knock over the pieces, crap on the board and strut around victorious” – Anonymous</p>
|
||
|
||
<p>Tags: <a href='tag_nonsense.html'>nonsense</a>, <a href='tag_quotes.html'>quotes</a></p>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<!-- text end -->
|
||
<h3><a class="ablack" href="thought-of-the-day27904.html">
|
||
thought of the day
|
||
</a></h3>
|
||
<!-- bashblog_timestamp: #201711271658.50# -->
|
||
<div class="subtitle">November 27, 2017 —
|
||
ben
|
||
</div>
|
||
<!-- text begin -->
|
||
|
||
<p>wherever you go, there you are</p>
|
||
|
||
<p>Tags: <a href='tag_thought-of-the-day.html'>thought-of-the-day</a></p>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<!-- text end -->
|
||
<h3><a class="ablack" href="thought-of-the-day.html">
|
||
Thought of the day
|
||
</a></h3>
|
||
<!-- bashblog_timestamp: #201711271649.29# -->
|
||
<div class="subtitle">November 27, 2017 —
|
||
ben
|
||
</div>
|
||
<!-- text begin -->
|
||
|
||
<p>things are not what they appear to be. nor are they otherwise.</p>
|
||
|
||
<p>Tags: <a href='tag_thought-of-the-day.html'>thought-of-the-day</a></p>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<!-- text end -->
|
||
<h3><a class="ablack" href="tildeteam-news.html">
|
||
tilde.team news
|
||
</a></h3>
|
||
<!-- bashblog_timestamp: #201806131507.45# -->
|
||
<div class="subtitle">June 13, 2018 —
|
||
~ben
|
||
</div>
|
||
<!-- text begin -->
|
||
|
||
<p>hey hi hello!</p>
|
||
|
||
<p>it seems that i haven't written anything on my blog in quite a while...</p>
|
||
|
||
<p>time to fix that! i've been quite busy in the last month or so with a lot of new ideas an energy for tilde.team.</p>
|
||
|
||
<p>after rediscovering my account on tilde.town, i hopped in the irc there and my enthusiasm translated into a couple new members over here on the ~team.</p>
|
||
|
||
<p>our irc has been somewhat more active recently which is awesome:)</p>
|
||
|
||
<p>some of the new updates in the last month:</p>
|
||
|
||
<ul>
|
||
<li><a href="https://git.tilde.team">tildegit (our own gitea instance)</a></li>
|
||
<li><a href="https://mail.tilde.team">tildemail</a> with postfix and dovecot for smtp/imap as well as local command line mail in mutt and alpine</li>
|
||
<li><a href="https://git.tildeverse.org/team/tilde-launcher"><code>tilde</code></a> user script wrapper with submission and approval flows</li>
|
||
<li><a href="https://tilde.team/wiki/?page=ssh">password auth disabled</a></li>
|
||
</ul>
|
||
|
||
<p>i'd like to make use of our new mailserver, so shoot me some <a href="mailto:ben@tilde.team">mail</a>.
|
||
i never get enough personal mail. it's all still privacy policy update notices. :(</p>
|
||
|
||
<p>see you soon!</p>
|
||
|
||
<p>tags: <a href='tag_tilde.html'>tilde</a>, <a href='tag_sysadmin.html'>sysadmin</a></p>
|
||
|
||
|
||
<!-- text end -->
|
||
<h3><a class="ablack" href="tildeverseorg.html">
|
||
tildeverse.org
|
||
</a></h3>
|
||
<!-- bashblog_timestamp: #201807152309.22# -->
|
||
<div class="subtitle">July 15, 2018 —
|
||
~ben
|
||
</div>
|
||
<!-- text begin -->
|
||
|
||
<p>since the last time i wrote a post here, i've registered the tildeverse.org domain and started moving some services over that were already intended for tildeverse use.</p>
|
||
|
||
<p>among those are <a href="https://git.tildeverse.org">gitea</a> and <a href="https://links.tildeverse.org">the new link aggregator</a> (which runs the same source as <a href="https://github.com/lobsters/lobsters">lobste.rs</a>).</p>
|
||
|
||
<p>i've also started a phlog in my <a href="https://gopher.tilde.team/tilde.team/~ben">gopherhole</a> with <a href="/~tomasino">~tomasino's</a> new <a href="https://github.com/jamestomasino/burrow">burrow</a> gopherhole tool!</p>
|
||
|
||
<p>i'll try to post a bit more often too with updates from the tildeverse!</p>
|
||
|
||
<p>Tags: <a href='tag_tilde.html'>tilde</a>, <a href='tag_internet.html'>internet</a>, <a href='tag_links.html'>links</a>, <a href='tag_git.html'>git</a></p>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<!-- text end -->
|
||
<div id="all_posts"><a href="all_posts.html">archive</a> — <a href="all_tags.html">all tags</a> — <a href="feed.rss">rss</a></div>
|
||
</div>
|
||
<div id="footer">CC by-nc-nd <a href="https://tilde.team/~ben/">~ben</a> — <a href="mailto:ben@tilde.team">ben@tilde.team</a><br/>
|
||
generated with <a href="https://tildegit.org/team/bashblog">bashblog</a>, a single bash script to easily create blogs like this one</div>
|
||
</div></div>
|
||
<script src="https://utteranc.es/client.js"
|
||
repo="benharri/tilde"
|
||
issue-term="title"
|
||
crossorigin="anonymous"
|
||
theme="github-dark"
|
||
async>
|
||
</script>
|
||
|
||
</div>
|
||
<br>
|
||
</body></html>
|