81 lines
4.2 KiB
HTML
81 lines
4.2 KiB
HTML
<!doctype html>
|
|
<html>
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
|
|
|
<meta name="theme-color" content="#00cc00">
|
|
<link rel="icon" type="image/png" sizes="192x192" href="https://tilde.team/apple-touch-icon-precomposed.png">
|
|
<link rel="icon" type="image/png" sizes="96x96" href="https://tilde.team/favicon-96x96.png">
|
|
|
|
<link rel="stylesheet" href="https://tilde.team/css/hacker.css">
|
|
<link rel="stylesheet" href="extra.css">
|
|
|
|
<link rel="alternate" type="application/rss+xml" title="subscribe to this page..." href="feed.rss" />
|
|
<title>proactive redundancy</title>
|
|
</head><body>
|
|
<div class="container">
|
|
|
|
<div id="divbodyholder">
|
|
<div class="headerholder"><div class="header">
|
|
<div id="title">
|
|
<h1 class="nomargin"><a class="ablack" href="https://tilde.team/~ben/blog/index.html">blog // ~ben</a></h1>
|
|
<div id="description">a blog about tildes and other things</div>
|
|
</div></div></div>
|
|
<div id="divbody"><div class="content">
|
|
<!-- entry begin -->
|
|
<h3><a class="ablack" href="proactive-redundancy.html">
|
|
proactive redundancy
|
|
</a></h3>
|
|
<!-- bashblog_timestamp: #201811151839.26# -->
|
|
<div class="subtitle">November 15, 2018 —
|
|
~ben
|
|
</div>
|
|
<!-- text begin -->
|
|
|
|
<p>after the <a href="november-13-post-mortem.html">fiasco</a> earlier this week, i've been taking steps to minimize
|
|
the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage. </p>
|
|
|
|
<hr />
|
|
|
|
<p>the first thing that i set up was a handful of additional ircd nodes: see <a href="https://tilde.chat/wiki/?page=servers">the tilde.chat wiki</a> for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team. </p>
|
|
|
|
<p>i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. <code>host tilde.chat</code> will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of <code>{your,team,bsd,slash}.tilde.chat</code>. </p>
|
|
|
|
<p>this creates the additional problem that visiting the <a href="https://tilde.chat">tilde.chat site</a> will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to <a href="https://tildegit.org/tildeverse/tilde.chat/issues/8">debug</a>. the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see <code>host chat.freenode.net</code> for the list of servers).</p>
|
|
|
|
<p>i'm not sure how to make any of the other services more resilient. it's something that i have been and will continue to research moving forward.</p>
|
|
|
|
<p>the other main step that i have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in <a href="https://tools.ietf.org/html/rfc1918">rfc 1918</a>.</p>
|
|
|
|
<p>i'd like to consider at least this risk to be mitigated.</p>
|
|
|
|
<p>thanks for reading,</p>
|
|
|
|
<p>~ben</p>
|
|
|
|
<p><strong>update</strong>: the round robin host is now <em>irc</em>.tilde.chat, which resolves the site issues that we were having, due to the duplicated deployments.</p>
|
|
|
|
<p>tags: <a href='tag_sysadmin.html'>sysadmin</a>, <a href='tag_tilde.html'>tilde</a></p>
|
|
|
|
|
|
|
|
<!-- text end -->
|
|
<!-- entry end -->
|
|
</div>
|
|
<div id="footer">CC by-nc-nd <a href="https://tilde.team/~ben/">~ben</a> — <a href="mailto:ben@tilde.team">ben@tilde.team</a><br/>
|
|
generated with <a href="https://tildegit.org/team/bashblog">bashblog</a>, a single bash script to easily create blogs like this one</div>
|
|
</div></div>
|
|
<script src="https://utteranc.es/client.js"
|
|
repo="benharri/tilde"
|
|
issue-term="title"
|
|
crossorigin="anonymous"
|
|
theme="github-dark"
|
|
async>
|
|
</script>
|
|
|
|
</div>
|
|
<br>
|
|
</body></html>
|