tildejack
/
tilde
forked from ben/tilde
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

weechat blogpost

This commit is contained in:
Ben Harris 2018-11-19 14:19:02 -05:00
parent 3f555cc186
commit 0e9995e801
17 changed files with 371 additions and 248 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
blog/.backup.tar.gz
View File

Binary file not shown.

1
blog/all_posts.html
View File

@ -27,6 +27,7 @@
<h3>all posts</h3>
<h4 class='allposts_header'>November 2018</h4>
<ul>
<li><a href="./weechat-setup.html">weechat setup</a> &mdash; November 16, 2018</li>
<li><a href="./proactive-redundancy.html">proactive redundancy</a> &mdash; November 15, 2018</li>
<li><a href="./november-13-post-mortem.html">november 13 post mortem</a> &mdash; November 13, 2018</li>
</ul>

4
blog/all_tags.html
View File

@ -46,10 +46,11 @@
<li><a href="tag_humor.html">humor</a> &mdash; 1 post</li>
<li><a href="tag_ideology.html">ideology</a> &mdash; 1 post</li>
<li><a href="tag_internet.html">internet</a> &mdash; 2 posts</li>
<li><a href="tag_irc.html">irc</a> &mdash; 1 post</li>
<li><a href="tag_jokes.html">jokes</a> &mdash; 2 posts</li>
<li><a href="tag_linguistics.html">linguistics</a> &mdash; 1 post</li>
<li><a href="tag_links.html">links</a> &mdash; 1 post</li>
<li><a href="tag_linux.html">linux</a> &mdash; 7 posts</li>
<li><a href="tag_linux.html">linux</a> &mdash; 8 posts</li>
<li><a href="tag_lyrics.html">lyrics</a> &mdash; 1 post</li>
<li><a href="tag_mastodon.html">mastodon</a> &mdash; 1 post</li>
<li><a href="tag_music.html">music</a> &mdash; 1 post</li>
@ -71,6 +72,7 @@
<li><a href="tag_ubuntu.html">ubuntu</a> &mdash; 2 posts</li>
<li><a href="tag_update.html">update</a> &mdash; 1 post</li>
<li><a href="tag_weather.html">weather</a> &mdash; 1 post</li>
<li><a href="tag_weechat.html">weechat</a> &mdash; 1 post</li>
<li><a href="tag_winter.html">winter</a> &mdash; 1 post</li>
<li><a href="tag_words.html">words</a> &mdash; 4 posts</li>
</ul>

68
blog/feed.rss
View File

@ -2,37 +2,32 @@
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
<channel><title>blog // ~ben</title><link>https://tilde.team/~ben/blog/index.html</link>
<description>a blog about tildes and other things</description><language>en</language>
<lastBuildDate>Fri, 16 Nov 2018 12:06:53 -0500</lastBuildDate>
<pubDate>Fri, 16 Nov 2018 12:06:53 -0500</pubDate>
<lastBuildDate>Fri, 16 Nov 2018 23:22:08 -0500</lastBuildDate>
<pubDate>Fri, 16 Nov 2018 23:22:08 -0500</pubDate>
<atom:link href="https://tilde.team/~ben/blog/feed.rss" rel="self" type="application/rss+xml" />
<item><title>
weechat setup
</title><description><![CDATA[
<p>so you decided to switch from irssi to weechat?</p>
<p>what options do you need to set? what plugins? what scripts?</p>
<p>i'll go over some of the most essential of these, and share my full configs.</p>
<hr />
]]></description><link>https://tilde.team/~ben/blog/weechat-setup.html</link>
<guid>https://tilde.team/~ben/blog/./weechat-setup.html</guid>
<dc:creator>~ben</dc:creator>
<pubDate>Fri, 16 Nov 2018 23:11:30 -0500</pubDate></item>
<item><title>
proactive redundancy
</title><description><![CDATA[
<p>after the <a href="november-13-post-mortem.html">fiasco</a> earlier this week, i've been taking steps to minimize
the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage. </p>
<p>the first thing that i set up was a handful of additional ircd nodes: see <a href="https://tilde.chat/wiki/?page=servers">the tilde.chat wiki</a> for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team. </p>
<p>i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. <code>host tilde.chat</code> will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of <code>{your,team,bsd,slash}.tilde.chat</code>. </p>
<p>this creates the additional problem that visiting the <a href="https://tilde.chat">tilde.chat site</a> will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to <a href="https://tildegit.org/tildeverse/tilde.chat/issues/8">debug</a>. the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see <code>host chat.freenode.net</code> for the list of servers).</p>
<p>i'm not sure how to make any of the other services more resilient. it's something that i have been and will continue to research moving forward.</p>
<p>the other main step that i have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in <a href="https://tools.ietf.org/html/rfc1918">rfc 1918</a>.</p>
<p>i'd like to consider at least this risk to be mitigated.</p>
<p>thanks for reading,</p>
<p>~ben</p>
<p><strong>update</strong>: the round robin host is now <em>irc</em>.tilde.chat, which resolves the site issues that we were having, due to the duplicated deployments.</p>
<p>tags: <a href='tag_sysadmin.html'>sysadmin</a>, <a href='tag_tilde.html'>tilde</a></p>
<!-- text end -->
<hr />
]]></description><link>https://tilde.team/~ben/blog/proactive-redundancy.html</link>
<guid>https://tilde.team/~ben/blog/./proactive-redundancy.html</guid>
<dc:creator>~ben</dc:creator>
@ -47,32 +42,7 @@ november 13 post mortem
<p>tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.</p>
<blockquote>
<p>We have indications that there was an attack from your server.
Please take all necessary measures to avoid this in the future and to solve the issue.</p>
</blockquote>
<p>at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.</p>
<p>when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the <a href="https://mail.tilde.team">webmail</a> to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!</p>
<p>here, i launch in to full debugging mode: what command was it? who ran it? </p>
<p>search <code>~/.bash_history</code> per user was not very successful. nothing i could find was related to net or map. i had checked <code>sudo grep nmap /home/*/.bash_history</code> and many other commands. </p>
<p>at this point, i had connected with other ~teammates across other irc nets (<a href="https://hashbang.sh/">#!</a>, <a href="https://tilde.town">~town</a>, etc). among suggestions to check <code>/var/log/syslog</code>, <code>/var/log/kern.log</code>, and <code>dmesg</code>, i finally decided to check <code>ps</code>. <code>ps -ef | grep nmap</code> yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for <a href="/~fosslinux/">~fosslinux</a>.</p>
<p>i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police <code>nmap</code> when it isn't scanning on every port?</p>
<p>after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that <a href="/~fosslinux/">~fosslinux</a> had only run <code>nmap</code> for addresses in the <code>10.0.0.0/8</code> space. the <code>10/8</code> address space is intended to not be addressable outside the local space. how could <a href="https://hetzner.com">hetzner</a> have found out about a localhost network probe!?</p>
<p>finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.</p>
<p>it's definitely time to research redundancy options!</p>
<p>tags: <a href='tag_post-mortem.html'>post-mortem</a>, <a href='tag_linux.html'>linux</a>, <a href='tag_sysadmin.html'>sysadmin</a></p>
<!-- text end -->
<hr />
]]></description><link>https://tilde.team/~ben/blog/november-13-post-mortem.html</link>
<guid>https://tilde.team/~ben/blog/./november-13-post-mortem.html</guid>
<dc:creator>~ben</dc:creator>

94
blog/index.html
View File

@ -24,6 +24,22 @@
<div id="description">a blog about tildes and other things</div>
</div></div></div>
<div id="divbody"><div class="content">
<h3><a class="ablack" href="weechat-setup.html">
weechat setup
</a></h3>
<!-- bashblog_timestamp: #201811162311.30# -->
<div class="subtitle">November 16, 2018 &mdash;
~ben
</div>
<!-- text begin -->
<p>so you decided to switch from irssi to weechat?</p>
<p>what options do you need to set? what plugins? what scripts?</p>
<p>i'll go over some of the most essential of these, and share my full configs.</p>
<p class="readmore"><a href="./weechat-setup.html">read more...</a></p>
<h3><a class="ablack" href="proactive-redundancy.html">
proactive redundancy
</a></h3>
@ -36,27 +52,7 @@ proactive redundancy
<p>after the <a href="november-13-post-mortem.html">fiasco</a> earlier this week, i've been taking steps to minimize
the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage. </p>
<p>the first thing that i set up was a handful of additional ircd nodes: see <a href="https://tilde.chat/wiki/?page=servers">the tilde.chat wiki</a> for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team. </p>
<p>i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. <code>host tilde.chat</code> will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of <code>{your,team,bsd,slash}.tilde.chat</code>. </p>
<p>this creates the additional problem that visiting the <a href="https://tilde.chat">tilde.chat site</a> will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to <a href="https://tildegit.org/tildeverse/tilde.chat/issues/8">debug</a>. the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see <code>host chat.freenode.net</code> for the list of servers).</p>
<p>i'm not sure how to make any of the other services more resilient. it's something that i have been and will continue to research moving forward.</p>
<p>the other main step that i have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in <a href="https://tools.ietf.org/html/rfc1918">rfc 1918</a>.</p>
<p>i'd like to consider at least this risk to be mitigated.</p>
<p>thanks for reading,</p>
<p>~ben</p>
<p><strong>update</strong>: the round robin host is now <em>irc</em>.tilde.chat, which resolves the site issues that we were having, due to the duplicated deployments.</p>
<p>tags: <a href='tag_sysadmin.html'>sysadmin</a>, <a href='tag_tilde.html'>tilde</a></p>
<!-- text end -->
<p class="readmore"><a href="./proactive-redundancy.html">read more...</a></p>
<h3><a class="ablack" href="november-13-post-mortem.html">
november 13 post mortem
</a></h3>
@ -72,32 +68,7 @@ november 13 post mortem
<p>tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.</p>
<blockquote>
<p>We have indications that there was an attack from your server.
Please take all necessary measures to avoid this in the future and to solve the issue.</p>
</blockquote>
<p>at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.</p>
<p>when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the <a href="https://mail.tilde.team">webmail</a> to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!</p>
<p>here, i launch in to full debugging mode: what command was it? who ran it? </p>
<p>search <code>~/.bash_history</code> per user was not very successful. nothing i could find was related to net or map. i had checked <code>sudo grep nmap /home/*/.bash_history</code> and many other commands. </p>
<p>at this point, i had connected with other ~teammates across other irc nets (<a href="https://hashbang.sh/">#!</a>, <a href="https://tilde.town">~town</a>, etc). among suggestions to check <code>/var/log/syslog</code>, <code>/var/log/kern.log</code>, and <code>dmesg</code>, i finally decided to check <code>ps</code>. <code>ps -ef | grep nmap</code> yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for <a href="/~fosslinux/">~fosslinux</a>.</p>
<p>i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police <code>nmap</code> when it isn't scanning on every port?</p>
<p>after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that <a href="/~fosslinux/">~fosslinux</a> had only run <code>nmap</code> for addresses in the <code>10.0.0.0/8</code> space. the <code>10/8</code> address space is intended to not be addressable outside the local space. how could <a href="https://hetzner.com">hetzner</a> have found out about a localhost network probe!?</p>
<p>finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.</p>
<p>it's definitely time to research redundancy options!</p>
<p>tags: <a href='tag_post-mortem.html'>post-mortem</a>, <a href='tag_linux.html'>linux</a>, <a href='tag_sysadmin.html'>sysadmin</a></p>
<!-- text end -->
<p class="readmore"><a href="./november-13-post-mortem.html">read more...</a></p>
<h3><a class="ablack" href="italy.html">
italy
</a></h3>
@ -304,35 +275,6 @@ already exists.</p>
<!-- text end -->
<h3><a class="ablack" href="bashblog-and-your-gopherhole.html">
bashblog and your gopherhole
</a></h3>
<!-- bashblog_timestamp: #201807221144.03# -->
<div class="subtitle">July 22, 2018 &mdash;
~ben
</div>
<!-- text begin -->
<p>i've created <a href="https://git.tildeverse.org/meta/bashblog">a repo</a> for the tilde.team customizations to <a href="https://github.com/cfenollosa/bashblog">bashblog</a>.</p>
<p>it will now make sure that your ~/public_gopher exists and symlink your blog into it with a nice gophermap to list all the markdown styled posts.</p>
<p>try it out and let me know if there are any problems!</p>
<p>tags: <a href='tag_tilde.html'>tilde</a>, <a href='tag_blog.html'>blog</a></p>
<!-- text end -->
<div id="all_posts"><a href="all_posts.html">archive</a> &mdash; <a href="all_tags.html">all tags</a> &mdash; <a href="feed.rss">rss</a></div>
</div>

3
blog/november-13-post-mortem.html
View File

@ -40,6 +40,8 @@ november 13 post mortem
<p>tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.</p>
<hr />
<blockquote>
<p>We have indications that there was an attack from your server.
Please take all necessary measures to avoid this in the future and to solve the issue.</p>
@ -64,7 +66,6 @@ Please take all necessary measures to avoid this in the future and to solve the
<p>it's definitely time to research redundancy options!</p>
<p>tags: <a href='tag_post-mortem.html'>post-mortem</a>, <a href='tag_linux.html'>linux</a>, <a href='tag_sysadmin.html'>sysadmin</a></p>
<!-- text end -->
<!-- entry end -->
</div>

2
blog/november-13-post-mortem.md
View File

@ -6,6 +6,8 @@ i awoke, not suspecting anything to be amiss. as soon as i logged in to check my
tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.
---
> We have indications that there was an attack from your server.
> Please take all necessary measures to avoid this in the future and to solve the issue.

3
blog/proactive-redundancy.html
View File

@ -37,6 +37,8 @@ proactive redundancy
<p>after the <a href="november-13-post-mortem.html">fiasco</a> earlier this week, i've been taking steps to minimize
the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage. </p>
<hr />
<p>the first thing that i set up was a handful of additional ircd nodes: see <a href="https://tilde.chat/wiki/?page=servers">the tilde.chat wiki</a> for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team. </p>
<p>i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. <code>host tilde.chat</code> will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of <code>{your,team,bsd,slash}.tilde.chat</code>. </p>
@ -56,7 +58,6 @@ the impact if tilde.team were to go down. it's still a large spof (single-point-
<p><strong>update</strong>: the round robin host is now <em>irc</em>.tilde.chat, which resolves the site issues that we were having, due to the duplicated deployments.</p>
<p>tags: <a href='tag_sysadmin.html'>sysadmin</a>, <a href='tag_tilde.html'>tilde</a></p>
<!-- text end -->
<!-- entry end -->
</div>

2
blog/proactive-redundancy.md
View File

@ -3,6 +3,8 @@ proactive redundancy
after the [fiasco](november-13-post-mortem.html) earlier this week, i've been taking steps to minimize
the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage.
---
the first thing that i set up was a handful of additional ircd nodes: see [the tilde.chat wiki](https://tilde.chat/wiki/?page=servers) for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team.
i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. `host tilde.chat` will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of `{your,team,bsd,slash}.tilde.chat`.

57
blog/tag_irc.html Normal file
View File

@ -0,0 +1,57 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="theme-color" content="#00cc00">
<link rel="icon" type="image/png" sizes="192x192" href="https://tilde.team/apple-touch-icon-precomposed.png">
<link rel="icon" type="image/png" sizes="96x96" href="https://tilde.team/favicon-96x96.png">
<link rel="stylesheet" href="https://tilde.team/css/dracula.css">
<link rel="stylesheet" href="extra.css">
<link rel="alternate" type="application/rss+xml" title="subscribe to this page..." href="feed.rss" />
<title>blog // ~ben &mdash; posts tagged "irc"</title>
</head><body>
<div class="container">
<div id="divbodyholder">
<div class="headerholder"><div class="header">
<div id="title">
<h1 class="nomargin"><a class="ablack" href="https://tilde.team/~ben/blog/index.html">blog // ~ben</a></h1>
<div id="description">a blog about tildes and other things</div>
</div></div></div>
<div id="divbody"><div class="content">
<h3><a class="ablack" href="weechat-setup.html">
weechat setup
</a></h3>
<!-- bashblog_timestamp: #201811162311.30# -->
<div class="subtitle">November 16, 2018 &mdash;
~ben
</div>
<!-- text begin -->
<p>so you decided to switch from irssi to weechat?</p>
<p>what options do you need to set? what plugins? what scripts?</p>
<p>i'll go over some of the most essential of these, and share my full configs.</p>
<p class="readmore"><a href="weechat-setup.html">read more...</a></p>
</div>
<div id="footer">CC by-nc-nd <a href="https://tilde.team/~ben/">~ben</a> &mdash; <a href="mailto:ben&#64;tilde&#46;team">ben&#64;tilde&#46;team</a><br/>
generated with <a href="https://tildegit.org/team/bashblog">bashblog</a>, a single bash script to easily create blogs like this one</div>
</div></div>
<script src="https://utteranc.es/client.js"
repo="benharri/tilde"
issue-term="title"
crossorigin="anonymous"
theme="github-dark"
async>
</script>
</div>
<br>
</body></html>

43
blog/tag_linux.html
View File

@ -24,6 +24,22 @@
<div id="description">a blog about tildes and other things</div>
</div></div></div>
<div id="divbody"><div class="content">
<h3><a class="ablack" href="weechat-setup.html">
weechat setup
</a></h3>
<!-- bashblog_timestamp: #201811162311.30# -->
<div class="subtitle">November 16, 2018 &mdash;
~ben
</div>
<!-- text begin -->
<p>so you decided to switch from irssi to weechat?</p>
<p>what options do you need to set? what plugins? what scripts?</p>
<p>i'll go over some of the most essential of these, and share my full configs.</p>
<p class="readmore"><a href="weechat-setup.html">read more...</a></p>
<h3><a class="ablack" href="november-13-post-mortem.html">
november 13 post mortem
</a></h3>
@ -39,32 +55,7 @@ november 13 post mortem
<p>tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.</p>
<blockquote>
<p>We have indications that there was an attack from your server.
Please take all necessary measures to avoid this in the future and to solve the issue.</p>
</blockquote>
<p>at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.</p>
<p>when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the <a href="https://mail.tilde.team">webmail</a> to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!</p>
<p>here, i launch in to full debugging mode: what command was it? who ran it? </p>
<p>search <code>~/.bash_history</code> per user was not very successful. nothing i could find was related to net or map. i had checked <code>sudo grep nmap /home/*/.bash_history</code> and many other commands. </p>
<p>at this point, i had connected with other ~teammates across other irc nets (<a href="https://hashbang.sh/">#!</a>, <a href="https://tilde.town">~town</a>, etc). among suggestions to check <code>/var/log/syslog</code>, <code>/var/log/kern.log</code>, and <code>dmesg</code>, i finally decided to check <code>ps</code>. <code>ps -ef | grep nmap</code> yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for <a href="/~fosslinux/">~fosslinux</a>.</p>
<p>i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police <code>nmap</code> when it isn't scanning on every port?</p>
<p>after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that <a href="/~fosslinux/">~fosslinux</a> had only run <code>nmap</code> for addresses in the <code>10.0.0.0/8</code> space. the <code>10/8</code> address space is intended to not be addressable outside the local space. how could <a href="https://hetzner.com">hetzner</a> have found out about a localhost network probe!?</p>
<p>finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.</p>
<p>it's definitely time to research redundancy options!</p>
<p>tags: <a href='tag_post-mortem.html'>post-mortem</a>, <a href='tag_linux.html'>linux</a>, <a href='tag_sysadmin.html'>sysadmin</a></p>
<!-- text end -->
<p class="readmore"><a href="november-13-post-mortem.html">read more...</a></p>
<h3><a class="ablack" href="no-more-google.html">
no more google
</a></h3>

27
blog/tag_post-mortem.html
View File

@ -39,32 +39,7 @@ november 13 post mortem
<p>tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.</p>
<blockquote>
<p>We have indications that there was an attack from your server.
Please take all necessary measures to avoid this in the future and to solve the issue.</p>
</blockquote>
<p>at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.</p>
<p>when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the <a href="https://mail.tilde.team">webmail</a> to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!</p>
<p>here, i launch in to full debugging mode: what command was it? who ran it? </p>
<p>search <code>~/.bash_history</code> per user was not very successful. nothing i could find was related to net or map. i had checked <code>sudo grep nmap /home/*/.bash_history</code> and many other commands. </p>
<p>at this point, i had connected with other ~teammates across other irc nets (<a href="https://hashbang.sh/">#!</a>, <a href="https://tilde.town">~town</a>, etc). among suggestions to check <code>/var/log/syslog</code>, <code>/var/log/kern.log</code>, and <code>dmesg</code>, i finally decided to check <code>ps</code>. <code>ps -ef | grep nmap</code> yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for <a href="/~fosslinux/">~fosslinux</a>.</p>
<p>i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police <code>nmap</code> when it isn't scanning on every port?</p>
<p>after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that <a href="/~fosslinux/">~fosslinux</a> had only run <code>nmap</code> for addresses in the <code>10.0.0.0/8</code> space. the <code>10/8</code> address space is intended to not be addressable outside the local space. how could <a href="https://hetzner.com">hetzner</a> have found out about a localhost network probe!?</p>
<p>finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.</p>
<p>it's definitely time to research redundancy options!</p>
<p>tags: <a href='tag_post-mortem.html'>post-mortem</a>, <a href='tag_linux.html'>linux</a>, <a href='tag_sysadmin.html'>sysadmin</a></p>
<!-- text end -->
<p class="readmore"><a href="november-13-post-mortem.html">read more...</a></p>
</div>
<div id="footer">CC by-nc-nd <a href="https://tilde.team/~ben/">~ben</a> &mdash; <a href="mailto:ben&#64;tilde&#46;team">ben&#64;tilde&#46;team</a><br/>
generated with <a href="https://tildegit.org/team/bashblog">bashblog</a>, a single bash script to easily create blogs like this one</div>

49
blog/tag_sysadmin.html
View File

@ -36,27 +36,7 @@ proactive redundancy
<p>after the <a href="november-13-post-mortem.html">fiasco</a> earlier this week, i've been taking steps to minimize
the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage. </p>
<p>the first thing that i set up was a handful of additional ircd nodes: see <a href="https://tilde.chat/wiki/?page=servers">the tilde.chat wiki</a> for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team. </p>
<p>i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. <code>host tilde.chat</code> will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of <code>{your,team,bsd,slash}.tilde.chat</code>. </p>
<p>this creates the additional problem that visiting the <a href="https://tilde.chat">tilde.chat site</a> will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to <a href="https://tildegit.org/tildeverse/tilde.chat/issues/8">debug</a>. the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see <code>host chat.freenode.net</code> for the list of servers).</p>
<p>i'm not sure how to make any of the other services more resilient. it's something that i have been and will continue to research moving forward.</p>
<p>the other main step that i have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in <a href="https://tools.ietf.org/html/rfc1918">rfc 1918</a>.</p>
<p>i'd like to consider at least this risk to be mitigated.</p>
<p>thanks for reading,</p>
<p>~ben</p>
<p><strong>update</strong>: the round robin host is now <em>irc</em>.tilde.chat, which resolves the site issues that we were having, due to the duplicated deployments.</p>
<p>tags: <a href='tag_sysadmin.html'>sysadmin</a>, <a href='tag_tilde.html'>tilde</a></p>
<!-- text end -->
<p class="readmore"><a href="proactive-redundancy.html">read more...</a></p>
<h3><a class="ablack" href="november-13-post-mortem.html">
november 13 post mortem
</a></h3>
@ -72,32 +52,7 @@ november 13 post mortem
<p>tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.</p>
<blockquote>
<p>We have indications that there was an attack from your server.
Please take all necessary measures to avoid this in the future and to solve the issue.</p>
</blockquote>
<p>at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.</p>
<p>when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the <a href="https://mail.tilde.team">webmail</a> to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!</p>
<p>here, i launch in to full debugging mode: what command was it? who ran it? </p>
<p>search <code>~/.bash_history</code> per user was not very successful. nothing i could find was related to net or map. i had checked <code>sudo grep nmap /home/*/.bash_history</code> and many other commands. </p>
<p>at this point, i had connected with other ~teammates across other irc nets (<a href="https://hashbang.sh/">#!</a>, <a href="https://tilde.town">~town</a>, etc). among suggestions to check <code>/var/log/syslog</code>, <code>/var/log/kern.log</code>, and <code>dmesg</code>, i finally decided to check <code>ps</code>. <code>ps -ef | grep nmap</code> yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for <a href="/~fosslinux/">~fosslinux</a>.</p>
<p>i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police <code>nmap</code> when it isn't scanning on every port?</p>
<p>after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that <a href="/~fosslinux/">~fosslinux</a> had only run <code>nmap</code> for addresses in the <code>10.0.0.0/8</code> space. the <code>10/8</code> address space is intended to not be addressable outside the local space. how could <a href="https://hetzner.com">hetzner</a> have found out about a localhost network probe!?</p>
<p>finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.</p>
<p>it's definitely time to research redundancy options!</p>
<p>tags: <a href='tag_post-mortem.html'>post-mortem</a>, <a href='tag_linux.html'>linux</a>, <a href='tag_sysadmin.html'>sysadmin</a></p>
<!-- text end -->
<p class="readmore"><a href="november-13-post-mortem.html">read more...</a></p>
<h3><a class="ablack" href="dns-shenanigans-post-mortem.html">
dns shenanigans post-mortem
</a></h3>

22
blog/tag_tilde.html
View File

@ -36,27 +36,7 @@ proactive redundancy
<p>after the <a href="november-13-post-mortem.html">fiasco</a> earlier this week, i've been taking steps to minimize
the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage. </p>
<p>the first thing that i set up was a handful of additional ircd nodes: see <a href="https://tilde.chat/wiki/?page=servers">the tilde.chat wiki</a> for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team. </p>
<p>i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. <code>host tilde.chat</code> will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of <code>{your,team,bsd,slash}.tilde.chat</code>. </p>
<p>this creates the additional problem that visiting the <a href="https://tilde.chat">tilde.chat site</a> will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to <a href="https://tildegit.org/tildeverse/tilde.chat/issues/8">debug</a>. the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see <code>host chat.freenode.net</code> for the list of servers).</p>
<p>i'm not sure how to make any of the other services more resilient. it's something that i have been and will continue to research moving forward.</p>
<p>the other main step that i have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in <a href="https://tools.ietf.org/html/rfc1918">rfc 1918</a>.</p>
<p>i'd like to consider at least this risk to be mitigated.</p>
<p>thanks for reading,</p>
<p>~ben</p>
<p><strong>update</strong>: the round robin host is now <em>irc</em>.tilde.chat, which resolves the site issues that we were having, due to the duplicated deployments.</p>
<p>tags: <a href='tag_sysadmin.html'>sysadmin</a>, <a href='tag_tilde.html'>tilde</a></p>
<!-- text end -->
<p class="readmore"><a href="proactive-redundancy.html">read more...</a></p>
<h3><a class="ablack" href="upsides-of-new-dns-nameservers.html">
upsides of new dns nameservers
</a></h3>

57
blog/tag_weechat.html Normal file
View File

@ -0,0 +1,57 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="theme-color" content="#00cc00">
<link rel="icon" type="image/png" sizes="192x192" href="https://tilde.team/apple-touch-icon-precomposed.png">
<link rel="icon" type="image/png" sizes="96x96" href="https://tilde.team/favicon-96x96.png">
<link rel="stylesheet" href="https://tilde.team/css/dracula.css">
<link rel="stylesheet" href="extra.css">
<link rel="alternate" type="application/rss+xml" title="subscribe to this page..." href="feed.rss" />
<title>blog // ~ben &mdash; posts tagged "weechat"</title>
</head><body>
<div class="container">
<div id="divbodyholder">
<div class="headerholder"><div class="header">
<div id="title">
<h1 class="nomargin"><a class="ablack" href="https://tilde.team/~ben/blog/index.html">blog // ~ben</a></h1>
<div id="description">a blog about tildes and other things</div>
</div></div></div>
<div id="divbody"><div class="content">
<h3><a class="ablack" href="weechat-setup.html">
weechat setup
</a></h3>
<!-- bashblog_timestamp: #201811162311.30# -->
<div class="subtitle">November 16, 2018 &mdash;
~ben
</div>
<!-- text begin -->
<p>so you decided to switch from irssi to weechat?</p>
<p>what options do you need to set? what plugins? what scripts?</p>
<p>i'll go over some of the most essential of these, and share my full configs.</p>
<p class="readmore"><a href="weechat-setup.html">read more...</a></p>
</div>
<div id="footer">CC by-nc-nd <a href="https://tilde.team/~ben/">~ben</a> &mdash; <a href="mailto:ben&#64;tilde&#46;team">ben&#64;tilde&#46;team</a><br/>
generated with <a href="https://tildegit.org/team/bashblog">bashblog</a>, a single bash script to easily create blogs like this one</div>
</div></div>
<script src="https://utteranc.es/client.js"
repo="benharri/tilde"
issue-term="title"
crossorigin="anonymous"
theme="github-dark"
async>
</script>
</div>
<br>
</body></html>

118
blog/weechat-setup.html Normal file
View File

@ -0,0 +1,118 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="theme-color" content="#00cc00">
<link rel="icon" type="image/png" sizes="192x192" href="https://tilde.team/apple-touch-icon-precomposed.png">
<link rel="icon" type="image/png" sizes="96x96" href="https://tilde.team/favicon-96x96.png">
<link rel="stylesheet" href="https://tilde.team/css/dracula.css">
<link rel="stylesheet" href="extra.css">
<link rel="alternate" type="application/rss+xml" title="subscribe to this page..." href="feed.rss" />
<title>weechat setup</title>
</head><body>
<div class="container">
<div id="divbodyholder">
<div class="headerholder"><div class="header">
<div id="title">
<h1 class="nomargin"><a class="ablack" href="https://tilde.team/~ben/blog/index.html">blog // ~ben</a></h1>
<div id="description">a blog about tildes and other things</div>
</div></div></div>
<div id="divbody"><div class="content">
<!-- entry begin -->
<h3><a class="ablack" href="weechat-setup.html">
weechat setup
</a></h3>
<!-- bashblog_timestamp: #201811162311.30# -->
<div class="subtitle">November 16, 2018 &mdash;
~ben
</div>
<!-- text begin -->
<p>so you decided to switch from irssi to weechat?</p>
<p>what options do you need to set? what plugins? what scripts?</p>
<p>i'll go over some of the most essential of these, and share my full configs.</p>
<hr />
<h3>options</h3>
<ul>
<li><code>logger.level.irc</code> the default is 9, which includes joins and parts. in most cases you can set this to 3, which only includes messages</li>
<li><code>weechat.look.buffer_notify.default</code> the default here is "all", which will add joins and parts to your hotlist. set it to "message".</li>
<li><code>weechat.look.confirm_quit</code> set this to on. you'll thank me when you type <code>/quit</code> and mean <code>/close</code></li>
<li><code>weechat.look.highlight</code> add a comma-separated list of names/terms you'd like to trigger a highlight here</li>
<li><code>weechat.look.prefix_align_max</code> set this to something between 10 and 20. otherwise, long nicks will crush your available screen real estate.</li>
<li><code>buflist.format.indent</code> try something like <code>${color:237}${if:${buffer.next_buffer.local_variables.type}=~^(channel|private)$?├:└}</code></li>
<li><code>buflist.format.number</code> if you want to skip the . or space after the number, set it to <code>${color:green}${number}</code></li>
<li><code>irc.look.color_nicks_in_names</code> set this to on</li>
<li><code>irc.look.color_nicks_in_nicklist</code> set this to on</li>
<li><code>irc.look.server_buffer</code> set this to independent to prevent automatic merges with the core weechat buffer. especially useful if you plan on using <a href="https://weechat.org/scripts/source/autosort.py.html/">autosort.py</a></li>
<li><code>irc.server_default.autoconnect</code> set this to on so you don't have to set it for every new network you add.</li>
</ul>
<h3>scripts</h3>
<p>these scripts can be managed with the built in <code>/script</code> tool. press i, then enter on the selected script to install it.</p>
<ul>
<li><a href="https://weechat.org/scripts/source/highmon.pl.html/">highmon.pl</a> - set aside a buffer to list the places your nick has been mentioned</li>
<li><a href="https://weechat.org/scripts/source/colorize_nicks.py.html/">colorize_nicks.py</a> - show nicks in chat with colors</li>
<li><a href="https://weechat.org/scripts/source/go.py.html/">go.py</a> - fuzzy quick jump by buffer number of channel name</li>
<li><a href="https://weechat.org/scripts/source/autojoin.py.html/">autojoin.py</a> - use <code>/autojoin --run</code> to save all the channels you're currently in to be autojoined the next time you start weechat</li>
<li><a href="https://weechat.org/scripts/source/autosort.py.html/">autosort.py</a> - use this script in tandem with irc.look.server_buffer=independent to keep you channel and server list in order</li>
<li><a href="https://weechat.org/scripts/source/colorize_lines.pl.html/">colorize_lines.pl</a> - i use this script to highlight the entire line of messages i've been mentioned in. check the options in the source or with <code>/help colorize_lines</code></li>
<li><a href="https://weechat.org/scripts/source/grep.py.html/">grep.py</a> - quickly search history and buffers with <code>/grep</code></li>
</ul>
<h3>plugins</h3>
<p>plugins include the likes of <a href="https://github.com/wee-slack/wee-slack">weechat-slack</a> and the irc plugin.</p>
<p>unless you need slack support, you should be ok without additional plugins</p>
<h3>the rest of my configs</h3>
<p>you can find the rest of my configs <a href="https://tildegit.org/ben/dotfiles/src/branch/master/weechat/.weechat">here</a>.</p>
<p>if you have any questions, feel free to ping me on irc. i'm benharri on freenode and ben on <a href="https://tilde.chat">tilde.chat</a></p>
<h3>screenshot</h3>
<p>here's a screenshot of my current configs</p>
<p><img src="https://ttm.sh/iu.png" alt="" title="" /></p>
<h3>bonus</h3>
<p>if you have an existing setup, you can check the config changes you've made with <code>/set diff</code></p>
<p>additionally, feel free to <a href="https://tildegit.org/ben/dotfiles/src/branch/master/weechat/.weechat/.gitignore">use my .gitignore</a>, add your ~/.weechat to source control, and compare.</p>
<p>hope you've enjoyed customizing your weechat!</p>
<p>tags: <a href='tag_irc.html'>irc</a>, <a href='tag_linux.html'>linux</a>, <a href='tag_weechat.html'>weechat</a></p>
<!-- text end -->
<!-- entry end -->
</div>
<div id="footer">CC by-nc-nd <a href="https://tilde.team/~ben/">~ben</a> &mdash; <a href="mailto:ben&#64;tilde&#46;team">ben&#64;tilde&#46;team</a><br/>
generated with <a href="https://tildegit.org/team/bashblog">bashblog</a>, a single bash script to easily create blogs like this one</div>
</div></div>
<script src="https://utteranc.es/client.js"
repo="benharri/tilde"
issue-term="title"
crossorigin="anonymous"
theme="github-dark"
async>
</script>
</div>
<br>
</body></html>

69
blog/weechat-setup.md Normal file
View File

@ -0,0 +1,69 @@
weechat setup
so you decided to switch from irssi to weechat?
what options do you need to set? what plugins? what scripts?
i'll go over some of the most essential of these, and share my full configs.
---
### options
* `logger.level.irc` the default is 9, which includes joins and parts. in most cases you can set this to 3, which only includes messages
* `weechat.look.buffer_notify.default` the default here is "all", which will add joins and parts to your hotlist. set it to "message".
* `weechat.look.confirm_quit` set this to on. you'll thank me when you type `/quit` and mean `/close`
* `weechat.look.highlight` add a comma-separated list of names/terms you'd like to trigger a highlight here
* `weechat.look.prefix_align_max` set this to something between 10 and 20. otherwise, long nicks will crush your available screen real estate.
* `buflist.format.indent` try something like `${color:237}${if:${buffer.next_buffer.local_variables.type}=~^(channel|private)$?├:└}`
* `buflist.format.number` if you want to skip the . or space after the number, set it to `${color:green}${number}`
* `irc.look.color_nicks_in_names` set this to on
* `irc.look.color_nicks_in_nicklist` set this to on
* `irc.look.server_buffer` set this to independent to prevent automatic merges with the core weechat buffer. especially useful if you plan on using [autosort.py](https://weechat.org/scripts/source/autosort.py.html/)
* `irc.server_default.autoconnect` set this to on so you don't have to set it for every new network you add.
### scripts
these scripts can be managed with the built in `/script` tool. press i, then enter on the selected script to install it.
* [highmon.pl](https://weechat.org/scripts/source/highmon.pl.html/) - set aside a buffer to list the places your nick has been mentioned
* [colorize_nicks.py](https://weechat.org/scripts/source/colorize_nicks.py.html/) - show nicks in chat with colors
* [go.py](https://weechat.org/scripts/source/go.py.html/) - fuzzy quick jump by buffer number of channel name
* [autojoin.py](https://weechat.org/scripts/source/autojoin.py.html/) - use `/autojoin --run` to save all the channels you're currently in to be autojoined the next time you start weechat
* [autosort.py](https://weechat.org/scripts/source/autosort.py.html/) - use this script in tandem with irc.look.server_buffer=independent to keep you channel and server list in order
* [colorize_lines.pl](https://weechat.org/scripts/source/colorize_lines.pl.html/) - i use this script to highlight the entire line of messages i've been mentioned in. check the options in the source or with `/help colorize_lines`
* [grep.py](https://weechat.org/scripts/source/grep.py.html/) - quickly search history and buffers with `/grep`
### plugins
plugins include the likes of [weechat-slack](https://github.com/wee-slack/wee-slack) and the irc plugin.
unless you need slack support, you should be ok without additional plugins
### the rest of my configs
you can find the rest of my configs [here](https://tildegit.org/ben/dotfiles/src/branch/master/weechat/.weechat).
if you have any questions, feel free to ping me on irc. i'm benharri on freenode and ben on [tilde.chat](https://tilde.chat)
### screenshot
here's a screenshot of my current configs
![](https://ttm.sh/iu.png)
### bonus
if you have an existing setup, you can check the config changes you've made with `/set diff`
additionally, feel free to [use my .gitignore](https://tildegit.org/ben/dotfiles/src/branch/master/weechat/.weechat/.gitignore), add your ~/.weechat to source control, and compare.
hope you've enjoyed customizing your weechat!
tags: irc, linux, weechat