90 lines
4.8 KiB
HTML
90 lines
4.8 KiB
HTML
<!doctype html>
|
|
<html>
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
|
|
|
<meta name="theme-color" content="#00cc00">
|
|
<link rel="icon" type="image/png" sizes="192x192" href="https://tilde.team/apple-touch-icon-precomposed.png">
|
|
<link rel="icon" type="image/png" sizes="96x96" href="https://tilde.team/favicon-96x96.png">
|
|
|
|
<link rel="stylesheet" href="https://tilde.team/~ben/gruvbox/gruvbox.css">
|
|
<link rel="stylesheet" href="extra.css">
|
|
|
|
<link rel="alternate" type="application/rss+xml" title="subscribe to this page..." href="feed.rss" />
|
|
<title>november 13 post mortem</title>
|
|
</head><body>
|
|
<main>
|
|
|
|
<div id="divbodyholder">
|
|
<div class="headerholder"><div class="header">
|
|
<div id="title">
|
|
<h1 class="nomargin"><a class="ablack" href="https://tilde.team/~ben/blog/index.html">blog // ~ben</a></h1>
|
|
<div id="description">a blog about tildes and other things</div>
|
|
</div></div></div>
|
|
<div id="divbody"><div class="content">
|
|
<!-- entry begin -->
|
|
<h3><a class="ablack" href="november-13-post-mortem.html">
|
|
november 13 post mortem
|
|
</a></h3>
|
|
<!-- bashblog_timestamp: #201811132020.33# -->
|
|
<div class="subtitle">November 13, 2018 —
|
|
~ben
|
|
</div>
|
|
<!-- text begin -->
|
|
|
|
<p>we had something of an outage on november 13, 2018 on tilde.team. </p>
|
|
|
|
<p>i awoke, not suspecting anything to be amiss. as soon as i logged in to check my email and irc mentions, it became clear.</p>
|
|
|
|
<p>tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.</p>
|
|
|
|
<hr />
|
|
|
|
<blockquote>
|
|
<p>We have indications that there was an attack from your server.
|
|
Please take all necessary measures to avoid this in the future and to solve the issue.</p>
|
|
</blockquote>
|
|
|
|
<p>at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.</p>
|
|
|
|
<p>when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the <a href="https://mail.tilde.team">webmail</a> to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!</p>
|
|
|
|
<p>here, i launch in to full debugging mode: what command was it? who ran it? </p>
|
|
|
|
<p>search <code>~/.bash_history</code> per user was not very successful. nothing i could find was related to net or map. i had checked <code>sudo grep nmap /home/*/.bash_history</code> and many other commands. </p>
|
|
|
|
<p>at this point, i had connected with other ~teammates across other irc nets (<a href="https://hashbang.sh/">#!</a>, <a href="https://tilde.town">~town</a>, etc). among suggestions to check <code>/var/log/syslog</code>, <code>/var/log/kern.log</code>, and <code>dmesg</code>, i finally decided to check <code>ps</code>. <code>ps -ef | grep nmap</code> yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for <a href="/~fosslinux/">~fosslinux</a>.</p>
|
|
|
|
<p>i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police <code>nmap</code> when it isn't scanning on every port?</p>
|
|
|
|
<p>after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that <a href="/~fosslinux/">~fosslinux</a> had only run <code>nmap</code> for addresses in the <code>10.0.0.0/8</code> space. the <code>10/8</code> address space is intended to not be addressable outside the local space. how could <a href="https://hetzner.com">hetzner</a> have found out about a localhost network probe!?</p>
|
|
|
|
<p>finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.</p>
|
|
|
|
<p>it's definitely time to research redundancy options!</p>
|
|
|
|
<p>tags: <a href='tag_post-mortem.html'>post-mortem</a>, <a href='tag_linux.html'>linux</a>, <a href='tag_sysadmin.html'>sysadmin</a></p>
|
|
|
|
|
|
|
|
|
|
<!-- text end -->
|
|
<!-- entry end -->
|
|
</div>
|
|
<div id="footer">CC by-nc-nd <a href="https://tilde.team/~ben/">~ben</a> — <a href="mailto:ben@tilde.team">ben@tilde.team</a><br>
|
|
generated with <a href="https://tildegit.org/team/bashblog">bashblog</a>, a single bash script to easily create blogs like this one</div>
|
|
</div></div>
|
|
<script src="https://utteranc.es/client.js"
|
|
repo="benharri/tilde"
|
|
issue-term="title"
|
|
crossorigin="anonymous"
|
|
theme="github-dark"
|
|
async>
|
|
</script>
|
|
|
|
</main>
|
|
<br>
|
|
</body></html>
|