diff --git a/blog/feed.rss b/blog/feed.rss
index 30cf34d..bfb55ac 100644
--- a/blog/feed.rss
+++ b/blog/feed.rss
@@ -2,190 +2,89 @@
blog // ~benhttps://tilde.team/~ben/blog/index.html
a blog about tildes and other thingsen
-Mon, 01 Oct 2018 00:40:34 -0400
-Mon, 01 Oct 2018 00:40:34 -0400
+Thu, 15 Nov 2018 18:39:28 -0500
+Thu, 15 Nov 2018 18:39:28 -0500
-italy
+proactive redundancy
i just got back from a 10-day backpacking trip to italy and i'd like to share some of the photos i took!
+
after the fiasco earlier this week, i've been taking steps to minimize
+the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage.
-
the travel plan was rome -> venice -> florence -> naples -> pompei/vesuvius -> capri -> amalfi
+
the first thing that i set up was a handful of additional ircd nodes: see the tilde.chat wiki for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team.
-
this is the roman forum (with colosseum in the background) as seen from the palatine.
+
i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. host tilde.chat will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of {your,team,bsd,slash}.tilde.chat.
-
+
this creates the additional problem that visiting the tilde.chat site will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to debug. the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see host chat.freenode.net for the list of servers).
-
-]]>https://tilde.team/~ben/blog/italy.html
-https://tilde.team/~ben/blog/./italy.html
-~ben
-Thu, 20 Sep 2018 17:32:33 -0400
-
-utterances
-i'm not sure how to make any of the other services more resilient. it's something that i have been and will continue to research moving forward.
-
i somehow stumbled upon utterances today at lunch. (i think someone had it forked on their github page).
+
the other main step that i have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in rfc 1918.
-
no matter how i found it, i still decided to add it to my blog here with bashblog. utterances is a commenting system that leverages github issues. so, for example a comment on a post shows up on github like this.
+
i'd like to consider at least this risk to be mitigated.
-
now we just need to figure out if it can be pointed at a gitea instance like tildegit. might be time for a PR!
-]]>https://tilde.team/~ben/blog/utterances.html
-https://tilde.team/~ben/blog/./utterances.html
+]]>https://tilde.team/~ben/blog/proactive-redundancy.html
+https://tilde.team/~ben/blog/./proactive-redundancy.html~ben
-Wed, 05 Sep 2018 21:34:13 -0400
+Thu, 15 Nov 2018 18:39:26 -0500
-no more google
+november 13 post mortem
not sure if this is appropriately tagged, but i didn't feel like making a new
-one.
+
we had something of an outage on november 13, 2018 on tilde.team.
-
i figured i should probably get some notes down about moving off google.
+
i awoke, not suspecting anything to be amiss. as soon as i logged in to check my email and irc mentions, it became clear.
-
to start, i'll get a list of the things i was able to easily replace:
+
tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.
google drive => syncthing (with a persistent node running on my personal vps)
-
+
+
We have indications that there was an attack from your server.
+Please take all necessary measures to avoid this in the future and to solve the issue.
+
-
i'm still using:
+
at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.
-
-
gplay music/youtube
-
google maps (open streetmap isn't good enough to replace it)
-
google photos - but this is going to be replaced long-term with syncthing
-
+
when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the webmail to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!
here, i launch in to full debugging mode: what command was it? who ran it?
+
search ~/.bash_history per user was not very successful. nothing i could find was related to net or map. i had checked sudo grep nmap /home/*/.bash_history and many other commands.
+
at this point, i had connected with other ~teammates across other irc nets (#!, ~town, etc). among suggestions to check /var/log/syslog, /var/log/kern.log, and dmesg, i finally decided to check ps. ps -ef | grep nmap yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for ~fosslinux.
+
i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police nmap when it isn't scanning on every port?
+
after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that ~fosslinux had only run nmap for addresses in the 10.0.0.0/8 space. the 10/8 address space is intended to not be addressable outside the local space. how could hetzner have found out about a localhost network probe!?
+
finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.
+
+
it's definitely time to research redundancy options!
-]]>https://tilde.team/~ben/blog/no-more-google.html
-https://tilde.team/~ben/blog/./no-more-google.html
+]]>https://tilde.team/~ben/blog/november-13-post-mortem.html
+https://tilde.team/~ben/blog/./november-13-post-mortem.html~ben
-Tue, 14 Aug 2018 23:36:05 -0400
+Tue, 13 Nov 2018 20:20:33 -0500
-upsides of new dns nameservers
+quote of the day
-
no more google
-
no more google
-
automated certbot validation for letsencrypt wildcard certs!! no more manual TXT records every three months!
-
-]]>https://tilde.team/~ben/blog/upsides-of-new-dns-nameservers.html
-https://tilde.team/~ben/blog/./upsides-of-new-dns-nameservers.html
-~ben
-Tue, 14 Aug 2018 15:05:38 -0400
-
-dns shenanigans post-mortem
-let's start by saying i probably should have done a bit more research before
-diving head-first into this endeavor.
-
-
i've been thinking about transferring my domains off google domains for some
-time now, as part of my personal goal to self host and limit my dependence on
-google and other large third-party monstrosities. along that line, i asked for
-registrar recommendations. ~tomasino responded
-with namesilo. i found that they had $3.99 registrations
-for .team and .zone domains, which is 1/10th the cost of the $40 registration
-on google domains.
-
-
i started out by getting the list of domains from the google console. 2 or 3
-of them had been registered within the last 60 days, so i wasn't able to
-transfer those just yet. i grabbed all the domain unlock codes and dropped
-them into namesilo. i failed to realize that the dns panel on google domains
-would disappear as soon as it went through, but more importantly that the
-nameservers would be left pointing to the old defunct google domains ones.
-
-
i updated the nameservers as soon as i realized this error from the namesilo
-panel. some of the domains propagated quickly. others, not so much. tilde.team
-was still in a state of flux between the old and new nameservers.
-
-
in a rush to get the dns problem fixed, and under recommendation from several
-people on irc, i decided to switch the nameservers for tilde.team and tilde.zone
-to cloudflare, leaving another layer of flux for the dns to be stuck in...
-
-
of the five domains that i moved to cloudflare, 3 returned with a dnssec error,
-claiming that i needed to remove the DS record from that zone. d'oh!
-
-
i removed the dnssec from those affected domains, so we should be good to go
-as soon as it all propagates through the fickle beast that is dns.
-
-
-
-
-
-
-]]>https://tilde.team/~ben/blog/dns-shenanigans-post-mortem.html
-https://tilde.team/~ben/blog/./dns-shenanigans-post-mortem.html
-~ben
-Tue, 14 Aug 2018 15:03:49 -0400
-
-lxd networking and additional IPs
-now that tilde.team is on a fancy-shmancy new dedi server, i've tried to get a secondary IP address
-assigned to a lxd container (which i plan to use for my personal stuff). lxd shows that the secondary
-IP is being picked up by that container, but i'm still seeing the host machine's IP as the external
-address.
-
-
i'm not sure how i'll need to configure the network settings on the host machine (now that we're running
-ubuntu 18.04 and it uses netplan for configs and not /etc/network/interfaces). another confusing thing is
-that the main config in /etc/netplan says that the network config is handled by systemd-networkd...
-
-
at least i have through the end of the year when my current vps runs out to get this up and running.
-
-
ping me on irc or email if you have experience with this.
-
-
-
-
-
-
-]]>https://tilde.team/~ben/blog/lxd-networking-and-additional-ips.html
-https://tilde.team/~ben/blog/./lxd-networking-and-additional-ips.html
-~ben
-Thu, 26 Jul 2018 15:34:50 -0400
-
-dotfiles
-finally got around to updating my dotfiles to use gnu stow.
-i adapted ~tomasino's makefile
-for use with the configs that i'm keeping with it.
-
-
now i just need to figure out why my ssh config doesn't copy/symlink my config to ~/.ssh when it
-already exists.
-
-
-
@@ -195,21 +94,31 @@ already exists.
-]]>https://tilde.team/~ben/blog/dotfiles.html
-https://tilde.team/~ben/blog/./dotfiles.html
+]]>https://tilde.team/~ben/blog/quote-of-the-day.html
+https://tilde.team/~ben/blog/./quote-of-the-day.html~ben
-Sun, 22 Jul 2018 19:26:26 -0400
+Tue, 23 Oct 2018 13:04:08 -0400
-bashblog and your gopherhole
+thought of the day
i've created a repo for the tilde.team customizations to bashblog.
+
why do they tell us to use the stairs in case of fire? shouldn't we be using a fire extinguisher?
@@ -222,21 +131,31 @@ bashblog and your gopherhole
-]]>https://tilde.team/~ben/blog/bashblog-and-your-gopherhole.html
-https://tilde.team/~ben/blog/./bashblog-and-your-gopherhole.html
-~ben
-Sun, 22 Jul 2018 11:44:03 -0400
+]]>https://tilde.team/~ben/blog/thought-of-the-day14302.html
+https://tilde.team/~ben/blog/./thought-of-the-day14302.html
+ben
+Tue, 23 Oct 2018 13:04:08 -0400
-more drone photos
+thought of the day
i finally got my drone out this summer to take some more pics!
+
everything in the universe either is or isn't a potato.
@@ -249,23 +168,31 @@ more drone photos
-]]>https://tilde.team/~ben/blog/more-drone-photos.html
-https://tilde.team/~ben/blog/./more-drone-photos.html
-~ben
-Sun, 15 Jul 2018 23:15:46 -0400
+]]>https://tilde.team/~ben/blog/thought-of-the-day2227.html
+https://tilde.team/~ben/blog/./thought-of-the-day2227.html
+ben
+Tue, 23 Oct 2018 13:04:08 -0400
-tildeverse.org
+Thought of the Day
since the last time i wrote a post here, i've registered the tildeverse.org domain and started moving some services over that were already intended for tildeverse use.
+
“Arguing with religious people – It’s like playing chess with a pigeon; no matter how good I am at chess, the pigeon is just going to knock over the pieces, crap on the board and strut around victorious” – Anonymous
@@ -278,10 +205,84 @@ tildeverse.org
-]]>https://tilde.team/~ben/blog/tildeverseorg.html
-https://tilde.team/~ben/blog/./tildeverseorg.html
-~ben
-Sun, 15 Jul 2018 23:09:22 -0400
+]]>https://tilde.team/~ben/blog/thought-of-the-day22873.html
+https://tilde.team/~ben/blog/./thought-of-the-day22873.html
+ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+thought of the day
+wherever you go, there you are
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+]]>https://tilde.team/~ben/blog/thought-of-the-day27904.html
+https://tilde.team/~ben/blog/./thought-of-the-day27904.html
+ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+Thought of the day
+things are not what they appear to be. nor are they otherwise.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+]]>https://tilde.team/~ben/blog/thought-of-the-day.html
+https://tilde.team/~ben/blog/./thought-of-the-day.html
+ben
+Tue, 23 Oct 2018 13:04:08 -0400
tilde.team news
https://tilde.team/~ben/blog/tildeteam-news.html
https://tilde.team/~ben/blog/./tildeteam-news.html~ben
-Wed, 13 Jun 2018 15:07:45 -0400
+Tue, 23 Oct 2018 13:04:08 -0400
-white pride vs black pride
+tildeverse.org
What White Nationalists Don't Get
+
since the last time i wrote a post here, i've registered the tildeverse.org domain and started moving some services over that were already intended for tildeverse use.
-
A common and seemingly reasonable argument for white pride or white nationalism is: why cant I be proud of my culture?
Well, you can. Always have been able to. We have Irish pride celebrations, we have German drinking festivals, we have Serbian food festivals. Any European culture you can think of has multiple organizations in North America dedicated to taking pride in their heritage and NO ONE gives them shit for it.
-
-
But, you see, when you start talking white pride, that's not a culture. That's a skin color. There is no white culture, never was. There is no pan-European culture, never was. Europe is a continent, not a culture or ethnicity.
-
-
Now, some of you are probably about to go, but wait! Black pride! How is that okay? Well, easy. Go find a black person and ask them if their ancestors were slaves. When you find one who says yes, proceed to ask them what country in Africa were your ancestors from? Do you know what their answer will probably be? I don't know. This is because their culture was taken from them. It was beaten out of them. They were enslaved, Christianized, and then white washed. The one unifying feature they have as a people is that history of slavery and that history of being black. They cant have Liberian pride, or Congolese pride, or insert African country pride because they have no fucking idea where their ancestors came from other than the broad region of West Africa.
-
-
Meanwhile us white people can often trace our ancestors to specific cities and regions. I can trace my mothers maiden name to a single fucking village in Ireland. I know where I came from. I don't have white culture, I have Irish culture.
-
-
So that's why white pride makes you an asshole but black pride actually makes sense.
@@ -351,378 +344,49 @@ white pride vs black pride
-]]>https://tilde.team/~ben/blog/white-pride-vs-black-pride.html
-https://tilde.team/~ben/blog/./white-pride-vs-black-pride.html
+]]>https://tilde.team/~ben/blog/tildeverseorg.html
+https://tilde.team/~ben/blog/./tildeverseorg.html~ben
-Wed, 07 Mar 2018 16:49:51 -0500
+Tue, 23 Oct 2018 13:04:08 -0400
-phoenix
+upsides of new dns nameservers
inspired by oodsnet, (and my pull request to add darkmode),
-i started to create my own tilde.team fork (now forum.tilde.team).
-
-
the first step was to switch out the css to the tilde.team standard and update the classes for bootstrap.
-once i got it going and integrated with the tilde.team linux auth service, i asked other tildeans for input and suggestions.
-
-
~micaiah was interested in helping, but also wanted to learn a new language and/or framework, so we decided to start over,
-recreating the entire forum with elixir/phoenix. we'd discussed elixir previously, but never had a
-convincing use case to force us to learn it.
-
-
the project is live, with the source code on github.
-
-
the thing that i'm most impressed with is the speed of the erlang runtime :D
-
-
check out these response times. sub-millisecond!?!?!
i somehow stumbled upon utterances today at lunch. (i think someone had it forked on their github page).
+
no matter how i found it, i still decided to add it to my blog here with bashblog. utterances is a commenting system that leverages github issues. so, for example a comment on a post shows up on github like this.
+
now we just need to figure out if it can be pointed at a gitea instance like tildegit. might be time for a PR!
-]]>https://tilde.team/~ben/blog/otm.html
-https://tilde.team/~ben/blog/./otm.html
+]]>https://tilde.team/~ben/blog/utterances.html
+https://tilde.team/~ben/blog/./utterances.html~ben
-Thu, 15 Feb 2018 13:33:16 -0500
-
-quote of the day
-Be Alert! - the world needs more Lerts.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-]]>https://tilde.team/~ben/blog/quote-of-the-day.html
-https://tilde.team/~ben/blog/./quote-of-the-day.html
-~ben
-Tue, 13 Feb 2018 09:55:06 -0500
-
-webassembly
-according to this post on the mozilla blog,
-we will be able to compile wasm as it streams into the browser in much the same way that images are decoded. this is a huge game changer from the current paradigm
-of loading javascript completely before being able to parse and compile it.
-
-
JavaScript code is much more expensive, byte for byte, than an image, because of the time spent parsing and compiling it.
It's possible to parse and compile wasm as fast as it comes over the network, which makes it much more like an image than JavaScript code.
you can now use gh:username/repo as the remote in place of git@github.com:username/repo, which is much shorter and easier to type many times!
-
-
git clone gh:benharri/learngit
-
-
there are many other use cases for the ssh_config file. for example, here is my config for the tilde machine for easy ssh connections.
-
-
-Host tilde
-HostName tilde.team
-User ben
-
-
-
then use ssh tilde to start a new ssh session. this also works with scp: try something like this scp file.txt tilde:workspace/. in place of scp file.txt ben@tilde.team:workspace/.
-
-
the ssh_config file is super useful. check man ssh_config for a full list of options!
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-]]>https://tilde.team/~ben/blog/cold.html
-https://tilde.team/~ben/blog/./cold.html
-ben
-Fri, 05 Jan 2018 09:49:42 -0500
-
-8values
-not that i'm very surprised by this, but i took the quiz again (after losing my results url) and got these results.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-]]>https://tilde.team/~ben/blog/loading.html
-https://tilde.team/~ben/blog/./loading.html
-ben
-Thu, 21 Dec 2017 16:09:45 -0500
+Tue, 23 Oct 2018 13:04:08 -0400
vr
https://tilde.team/~ben/blog/vr.html
https://tilde.team/~ben/blog/./vr.htmlben
-Mon, 18 Dec 2017 13:36:04 -0500
+Tue, 23 Oct 2018 13:04:08 -0400
-net neutrality vote today
+webassembly
Everybody! We only have UNTIL TOMORROW to fight the FCC & the repeal of #NetNeutrality! Repealing Net Nutrality would result in an unequal access to online content including research, social/political organizing, and personal media. It would also allow powerhouse companies providing internet to charge more for regular quality internet, and charge certain users more than others.
-HERE'S A WAY TO ACT - takes less than a minute.
-
-
-
Go to gofccyourself.com
-(the shortcut John Oliver made to the hard-to-find FCC comment page)
-
Click on the 17-108 link (Restoring Internet Freedom)
-
Click on "express"
-
Be sure to hit "ENTER" after you put in your name & info so it registers.
-
In the comment section write, "I strongly support net neutrality backed by Title 2 oversight of ISPs."
-
Click to submit, done. - Make sure you hit submit at the end!
-
-
-
Copy and paste this into your own status update!
-Seriously, this is simple and so important. Do it.
according to this post on the mozilla blog,
+we will be able to compile wasm as it streams into the browser in much the same way that images are decoded. this is a huge game changer from the current paradigm
+of loading javascript completely before being able to parse and compile it.
+
JavaScript code is much more expensive, byte for byte, than an image, because of the time spent parsing and compiling it.
It's possible to parse and compile wasm as fast as it comes over the network, which makes it much more like an image than JavaScript code.
@@ -815,145 +466,10 @@ Seriously, this is simple and so important. Do it.
-]]>https://tilde.team/~ben/blog/net-neutrality-vote-today.html
-https://tilde.team/~ben/blog/./net-neutrality-vote-today.html
-ben
-Thu, 14 Dec 2017 06:26:55 -0500
-
-hey dere bub!
-if you haven't checked it out yet, give my new podcast a listen!
-
-
-
-
-]]>https://tilde.team/~ben/blog/hey-dere-bub.html
-https://tilde.team/~ben/blog/./hey-dere-bub.html
-ben
-Wed, 13 Dec 2017 19:12:51 -0500
-
-pan galactic gargle blaster
-short's brewery released another batch of their pan galactic gargle blaster imperial IPA. i had one last night and was very excited to have it once again.
-
-
-
“The Hitch-Hiker's Guide to the Galaxy also mentions alcohol. It says that the best drink in existence is the Pan Galactic Gargle Blaster, the effect of which is like having your brains smashed out with a slice of lemon wrapped round a large gold brick.”
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-]]>https://tilde.team/~ben/blog/thought-of-the-day2227.html
-https://tilde.team/~ben/blog/./thought-of-the-day2227.html
-ben
-Sun, 03 Dec 2017 13:47:36 -0500
+]]>https://tilde.team/~ben/blog/webassembly.html
+https://tilde.team/~ben/blog/./webassembly.html
+~ben
+Tue, 23 Oct 2018 13:04:08 -0400
where to find me elsewhere on the web
https://tilde.team/~ben/blog/where-to-find-me-elsewhere-on-the-web.html
https://tilde.team/~ben/blog/./where-to-find-me-elsewhere-on-the-web.htmlben
-Tue, 28 Nov 2017 16:22:42 -0500
+Tue, 23 Oct 2018 13:04:08 -0400
-blog update
+white pride vs black pride
sorry for the blast of posts. i recreated them from a wordpress blog on motd.org.
+
What White Nationalists Don't Get
+
+
A common and seemingly reasonable argument for white pride or white nationalism is: why cant I be proud of my culture?
+
+
Well, you can. Always have been able to. We have Irish pride celebrations, we have German drinking festivals, we have Serbian food festivals. Any European culture you can think of has multiple organizations in North America dedicated to taking pride in their heritage and NO ONE gives them shit for it.
+
+
But, you see, when you start talking white pride, that's not a culture. That's a skin color. There is no white culture, never was. There is no pan-European culture, never was. Europe is a continent, not a culture or ethnicity.
+
+
Now, some of you are probably about to go, but wait! Black pride! How is that okay? Well, easy. Go find a black person and ask them if their ancestors were slaves. When you find one who says yes, proceed to ask them what country in Africa were your ancestors from? Do you know what their answer will probably be? I don't know. This is because their culture was taken from them. It was beaten out of them. They were enslaved, Christianized, and then white washed. The one unifying feature they have as a people is that history of slavery and that history of being black. They cant have Liberian pride, or Congolese pride, or insert African country pride because they have no fucking idea where their ancestors came from other than the broad region of West Africa.
+
+
Meanwhile us white people can often trace our ancestors to specific cities and regions. I can trace my mothers maiden name to a single fucking village in Ireland. I know where I came from. I don't have white culture, I have Irish culture.
+
+
So that's why white pride makes you an asshole but black pride actually makes sense.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+]]>https://tilde.team/~ben/blog/white-pride-vs-black-pride.html
+https://tilde.team/~ben/blog/./white-pride-vs-black-pride.html
+~ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+4k gaming with a gtx1080ti
+i recently picked up a gtx1080ti on newegg (and a 4k monitor earlier in the summer on prime day). i can't stop playing the witcher 3. even though it's a couple years old, it just looks so good. plus, the story and gameplay are incredible as well. i find myself dreaming about the game and longing to play it when i'm not. i'll have to say it is definitively the best game i've ever played.
+
+
some of the other games that i'm looking forward to exploring more of in 4k are:
+
+
+
destiny 2
+
prey
+
overwatch (not that this will look insanely good, it will just be super silky smooth)
@@ -1037,26 +613,23 @@ blog update
-]]>https://tilde.team/~ben/blog/blog-update.html
-https://tilde.team/~ben/blog/./blog-update.html
-ben
-Mon, 27 Nov 2017 20:48:10 -0500
+]]>https://tilde.team/~ben/blog/4k-gaming-with-a-gtx1080ti.html
+https://tilde.team/~ben/blog/./4k-gaming-with-a-gtx1080ti.html
+tildeman
+Tue, 23 Oct 2018 13:04:08 -0400
-thought of the day
+8values
wherever you go, there you are
-
-
@@ -1074,10 +647,10 @@ thought of the day
-]]>https://tilde.team/~ben/blog/thought-of-the-day27904.html
-https://tilde.team/~ben/blog/./thought-of-the-day27904.html
+]]>https://tilde.team/~ben/blog/8values.html
+https://tilde.team/~ben/blog/./8values.htmlben
-Mon, 27 Nov 2017 16:58:50 -0500
+Tue, 23 Oct 2018 13:04:08 -0400
antiwitze
https://tilde.team/~ben/blog/antiwitze.html
https://tilde.team/~ben/blog/./antiwitze.htmlben
-Mon, 27 Nov 2017 16:58:17 -0500
+Tue, 23 Oct 2018 13:04:08 -0400
-thought of the day
+bashblog and your gopherhole
why do they tell us to use the stairs in case of fire? shouldn't we be using a fire extinguisher?
+
i've created a repo for the tilde.team customizations to bashblog.
+
+
+
+
+
+
+
+
+
+
+
+
+]]>https://tilde.team/~ben/blog/bashblog-and-your-gopherhole.html
+https://tilde.team/~ben/blog/./bashblog-and-your-gopherhole.html
+~ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+blog update
+sorry for the blast of posts. i recreated them from a wordpress blog on motd.org.
+
+
i plan on posting some random thoughts here from time to time. thanks for checking in.
@@ -1156,84 +758,10 @@ thought of the day
-]]>https://tilde.team/~ben/blog/thought-of-the-day14302.html
-https://tilde.team/~ben/blog/./thought-of-the-day14302.html
+]]>https://tilde.team/~ben/blog/blog-update.html
+https://tilde.team/~ben/blog/./blog-update.htmlben
-Mon, 27 Nov 2017 16:56:36 -0500
-
-Nonsense
-I hole-hardedly agree, but allow me to play doubles advocate here for a moment. For all intensive purposes I think you are wrong. In an age where false morals are a diamond dozen, true virtues are a blessing in the skies. We often put our false morality on a petal stool like a bunch of pre-Madonnas, but you all seem to be taking something very valuable for granite. So I ask of you to mustard up all the strength you can because it is a doggy dog world out there. Although there is some merit to what you are saying it seems like you have a huge ship on your shoulder. In your argument you seem to throw everything in but the kids Nsync, and even though you are having a feel day with this I am here to bring you back into reality. I have a sick sense when it comes to these types of things. It is almost spooky, because I cannot turn a blonde eye to these glaring flaws in your rhetoric. I have zero taller ants when it comes to people spouting out hate in the name of moral righteousness. You just need to remember what comes around is all around, and when supply and command fails you will be the first to go. Make my words, when you get down to brass stacks it doesn’t take rocket appliances to get two birds stoned at once. It’s clear who makes the pants in this relationship, and sometimes you just have to swallow your prize and accept the facts. You might have to come to this conclusion through denial and error but I swear on my mother’s mating name that when you put the petal to the medal you will pass with flying carpets like it’s a peach of cake.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-]]>https://tilde.team/~ben/blog/nonsense.html
-https://tilde.team/~ben/blog/./nonsense.html
-ben
-Mon, 27 Nov 2017 16:55:29 -0500
-
-Thought of the Day
-“Arguing with religious people – It’s like playing chess with a pigeon; no matter how good I am at chess, the pigeon is just going to knock over the pieces, crap on the board and strut around victorious” – Anonymous
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-]]>https://tilde.team/~ben/blog/thought-of-the-day22873.html
-https://tilde.team/~ben/blog/./thought-of-the-day22873.html
-ben
-Mon, 27 Nov 2017 16:54:07 -0500
+Tue, 23 Oct 2018 13:04:08 -0400
Christian Morgenstern – “verkehrte Welt”
]]>https://tilde.team/~ben/blog/christian-morgenstern---verkehrte-welt.html
https://tilde.team/~ben/blog/./christian-morgenstern---verkehrte-welt.htmlben
-Mon, 27 Nov 2017 16:52:49 -0500
+Tue, 23 Oct 2018 13:04:08 -0400
-Joe on Sporty-ball-z
+cold
For every nerd calling something sportsball there needs to be a jock that walks into a library and yells “WHAT’S UP WITH ALL THESE WORD BURGERS”
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+]]>https://tilde.team/~ben/blog/cold.html
+https://tilde.team/~ben/blog/./cold.html
+ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+dns shenanigans post-mortem
+let's start by saying i probably should have done a bit more research before
+diving head-first into this endeavor.
+
+
i've been thinking about transferring my domains off google domains for some
+time now, as part of my personal goal to self host and limit my dependence on
+google and other large third-party monstrosities. along that line, i asked for
+registrar recommendations. ~tomasino responded
+with namesilo. i found that they had $3.99 registrations
+for .team and .zone domains, which is 1/10th the cost of the $40 registration
+on google domains.
+
+
i started out by getting the list of domains from the google console. 2 or 3
+of them had been registered within the last 60 days, so i wasn't able to
+transfer those just yet. i grabbed all the domain unlock codes and dropped
+them into namesilo. i failed to realize that the dns panel on google domains
+would disappear as soon as it went through, but more importantly that the
+nameservers would be left pointing to the old defunct google domains ones.
+
+
i updated the nameservers as soon as i realized this error from the namesilo
+panel. some of the domains propagated quickly. others, not so much. tilde.team
+was still in a state of flux between the old and new nameservers.
+
+
in a rush to get the dns problem fixed, and under recommendation from several
+people on irc, i decided to switch the nameservers for tilde.team and tilde.zone
+to cloudflare, leaving another layer of flux for the dns to be stuck in...
+
+
of the five domains that i moved to cloudflare, 3 returned with a dnssec error,
+claiming that i needed to remove the DS record from that zone. d'oh!
+
+
i removed the dnssec from those affected domains, so we should be good to go
+as soon as it all propagates through the fickle beast that is dns.
@@ -1362,10 +974,38 @@ Joe on Sporty-ball-z
-]]>https://tilde.team/~ben/blog/joe-on-sporty-ball-z.html
-https://tilde.team/~ben/blog/./joe-on-sporty-ball-z.html
+]]>https://tilde.team/~ben/blog/dont-be-a-coconut.html
+https://tilde.team/~ben/blog/./dont-be-a-coconut.htmlben
-Mon, 27 Nov 2017 16:51:42 -0500
+Tue, 23 Oct 2018 13:04:08 -0400
+
+dotfiles
+finally got around to updating my dotfiles to use gnu stow.
+i adapted ~tomasino's makefile
+for use with the configs that i'm keeping with it.
+
+
now i just need to figure out why my ssh config doesn't copy/symlink my config to ~/.ssh when it
+already exists.
+
+
+
+
+
+
+
+
+
+
+
+
+]]>https://tilde.team/~ben/blog/dotfiles.html
+https://tilde.team/~ben/blog/./dotfiles.html
+~ben
+Tue, 23 Oct 2018 13:04:08 -0400
fun words in german
https://tilde.team/~ben/blog/fun-words-in-german.html
https://tilde.team/~ben/blog/./fun-words-in-german.htmlben
-Mon, 27 Nov 2017 16:50:40 -0500
+Tue, 23 Oct 2018 13:04:08 -0400
-Thought of the day
+git remotes with ssh aliases
things are not what they appear to be. nor are they otherwise.
-
-
you can now use gh:username/repo as the remote in place of git@github.com:username/repo, which is much shorter and easier to type many times!
+
git clone gh:benharri/learngit
+
there are many other use cases for the ssh_config file. for example, here is my config for the tilde machine for easy ssh connections.
+
+Host tilde
+HostName tilde.team
+User ben
+
+
then use ssh tilde to start a new ssh session. this also works with scp: try something like this scp file.txt tilde:workspace/. in place of scp file.txt ben@tilde.team:workspace/.
+
the ssh_config file is super useful. check man ssh_config for a full list of options!
@@ -1440,49 +1096,25 @@ Thought of the day
-]]>https://tilde.team/~ben/blog/thought-of-the-day.html
-https://tilde.team/~ben/blog/./thought-of-the-day.html
+]]>https://tilde.team/~ben/blog/git-remotes-with-ssh-aliases.html
+https://tilde.team/~ben/blog/./git-remotes-with-ssh-aliases.htmlben
-Mon, 27 Nov 2017 16:49:29 -0500
+Tue, 23 Oct 2018 13:04:08 -0400
-links to save for later
+hey dere bub!
cool pics
-gatsby ssg
-tachyons
+
if you haven't checked it out yet, give my new podcast a listen!
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-]]>https://tilde.team/~ben/blog/links-to-save-for-later.html
-https://tilde.team/~ben/blog/./links-to-save-for-later.html
+
+]]>https://tilde.team/~ben/blog/hey-dere-bub.html
+https://tilde.team/~ben/blog/./hey-dere-bub.htmlben
-Fri, 20 Oct 2017 16:18:28 -0400
+Tue, 23 Oct 2018 13:04:08 -0400
hi there
https://tilde.team/~ben/blog/hi-there.html
https://tilde.team/~ben/blog/./hi-there.htmltildeman
-Mon, 02 Oct 2017 16:11:13 -0400
+Tue, 23 Oct 2018 13:04:08 -0400
-4k gaming with a gtx1080ti
+italy
i recently picked up a gtx1080ti on newegg (and a 4k monitor earlier in the summer on prime day). i can't stop playing the witcher 3. even though it's a couple years old, it just looks so good. plus, the story and gameplay are incredible as well. i find myself dreaming about the game and longing to play it when i'm not. i'll have to say it is definitively the best game i've ever played.
+
i just got back from a 10-day backpacking trip to italy and i'd like to share some of the photos i took!
-
some of the other games that i'm looking forward to exploring more of in 4k are:
+
the travel plan was rome -> venice -> florence -> naples -> pompei/vesuvius -> capri -> amalfi
-
-
destiny 2
-
prey
-
overwatch (not that this will look insanely good, it will just be super silky smooth)
this is the roman forum (with colosseum in the background) as seen from the palatine.
+
+
+]]>https://tilde.team/~ben/blog/italy.html
+https://tilde.team/~ben/blog/./italy.html
+~ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+Joe on Sporty-ball-z
+For every nerd calling something sportsball there needs to be a jock that walks into a library and yells “WHAT’S UP WITH ALL THESE WORD BURGERS”
+
@@ -1589,8 +1213,452 @@ hi there
-]]>https://tilde.team/~ben/blog/4k-gaming-with-a-gtx1080ti.html
-https://tilde.team/~ben/blog/./4k-gaming-with-a-gtx1080ti.html
-tildeman
-Mon, 02 Oct 2017 16:09:26 -0400
+]]>https://tilde.team/~ben/blog/joe-on-sporty-ball-z.html
+https://tilde.team/~ben/blog/./joe-on-sporty-ball-z.html
+ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+links to save for later
+cool pics
+gatsby ssg
+tachyons
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+]]>https://tilde.team/~ben/blog/loading.html
+https://tilde.team/~ben/blog/./loading.html
+ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+lxd networking and additional IPs
+now that tilde.team is on a fancy-shmancy new dedi server, i've tried to get a secondary IP address
+assigned to a lxd container (which i plan to use for my personal stuff). lxd shows that the secondary
+IP is being picked up by that container, but i'm still seeing the host machine's IP as the external
+address.
+
+
i'm not sure how i'll need to configure the network settings on the host machine (now that we're running
+ubuntu 18.04 and it uses netplan for configs and not /etc/network/interfaces). another confusing thing is
+that the main config in /etc/netplan says that the network config is handled by systemd-networkd...
+
+
at least i have through the end of the year when my current vps runs out to get this up and running.
+
+
ping me on irc or email if you have experience with this.
+
+
+
+
+
+
+
+
+
+
+
+
+]]>https://tilde.team/~ben/blog/more-drone-photos.html
+https://tilde.team/~ben/blog/./more-drone-photos.html
+~ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+net neutrality vote today
+Everybody! We only have UNTIL TOMORROW to fight the FCC & the repeal of #NetNeutrality! Repealing Net Nutrality would result in an unequal access to online content including research, social/political organizing, and personal media. It would also allow powerhouse companies providing internet to charge more for regular quality internet, and charge certain users more than others.
+HERE'S A WAY TO ACT - takes less than a minute.
+
+
+
Go to gofccyourself.com
+(the shortcut John Oliver made to the hard-to-find FCC comment page)
+
Click on the 17-108 link (Restoring Internet Freedom)
+
Click on "express"
+
Be sure to hit "ENTER" after you put in your name & info so it registers.
+
In the comment section write, "I strongly support net neutrality backed by Title 2 oversight of ISPs."
+
Click to submit, done. - Make sure you hit submit at the end!
+
+
+
Copy and paste this into your own status update!
+Seriously, this is simple and so important. Do it.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+]]>https://tilde.team/~ben/blog/net-neutrality-vote-today.html
+https://tilde.team/~ben/blog/./net-neutrality-vote-today.html
+ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+no more google
+not sure if this is appropriately tagged, but i didn't feel like making a new
+one.
+
+
i figured i should probably get some notes down about moving off google.
+
+
to start, i'll get a list of the things i was able to easily replace:
+
+
+
+
+
+
+]]>https://tilde.team/~ben/blog/no-more-google.html
+https://tilde.team/~ben/blog/./no-more-google.html
+~ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+Nonsense
+I hole-hardedly agree, but allow me to play doubles advocate here for a moment. For all intensive purposes I think you are wrong. In an age where false morals are a diamond dozen, true virtues are a blessing in the skies. We often put our false morality on a petal stool like a bunch of pre-Madonnas, but you all seem to be taking something very valuable for granite. So I ask of you to mustard up all the strength you can because it is a doggy dog world out there. Although there is some merit to what you are saying it seems like you have a huge ship on your shoulder. In your argument you seem to throw everything in but the kids Nsync, and even though you are having a feel day with this I am here to bring you back into reality. I have a sick sense when it comes to these types of things. It is almost spooky, because I cannot turn a blonde eye to these glaring flaws in your rhetoric. I have zero taller ants when it comes to people spouting out hate in the name of moral righteousness. You just need to remember what comes around is all around, and when supply and command fails you will be the first to go. Make my words, when you get down to brass stacks it doesn’t take rocket appliances to get two birds stoned at once. It’s clear who makes the pants in this relationship, and sometimes you just have to swallow your prize and accept the facts. You might have to come to this conclusion through denial and error but I swear on my mother’s mating name that when you put the petal to the medal you will pass with flying carpets like it’s a peach of cake.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+]]>https://tilde.team/~ben/blog/otm.html
+https://tilde.team/~ben/blog/./otm.html
+~ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+pan galactic gargle blaster
+short's brewery released another batch of their pan galactic gargle blaster imperial IPA. i had one last night and was very excited to have it once again.
+
+
+
“The Hitch-Hiker's Guide to the Galaxy also mentions alcohol. It says that the best drink in existence is the Pan Galactic Gargle Blaster, the effect of which is like having your brains smashed out with a slice of lemon wrapped round a large gold brick.”
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+]]>https://tilde.team/~ben/blog/pan-galactic-gargle-blaster.html
+https://tilde.team/~ben/blog/./pan-galactic-gargle-blaster.html
+ben
+Tue, 23 Oct 2018 13:04:08 -0400
+
+phoenix
+inspired by oodsnet, (and my pull request to add darkmode),
+i started to create my own tilde.team fork (now forum.tilde.team).
+
+
the first step was to switch out the css to the tilde.team standard and update the classes for bootstrap.
+once i got it going and integrated with the tilde.team linux auth service, i asked other tildeans for input and suggestions.
+
+
~micaiah was interested in helping, but also wanted to learn a new language and/or framework, so we decided to start over,
+recreating the entire forum with elixir/phoenix. we'd discussed elixir previously, but never had a
+convincing use case to force us to learn it.
+
+
the project is live, with the source code on github.
+
+
the thing that i'm most impressed with is the speed of the erlang runtime :D
+
+
check out these response times. sub-millisecond!?!?!
i just got back from a 10-day backpacking trip to italy and i'd like to share some of the photos i took!
+
after the fiasco earlier this week, i've been taking steps to minimize
+the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage.
-
the travel plan was rome -> venice -> florence -> naples -> pompei/vesuvius -> capri -> amalfi
+
the first thing that i set up was a handful of additional ircd nodes: see the tilde.chat wiki for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team.
-
this is the roman forum (with colosseum in the background) as seen from the palatine.
+
i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. host tilde.chat will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of {your,team,bsd,slash}.tilde.chat.
-
+
this creates the additional problem that visiting the tilde.chat site will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to debug. the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see host chat.freenode.net for the list of servers).
i somehow stumbled upon utterances today at lunch. (i think someone had it forked on their github page).
+
we had something of an outage on november 13, 2018 on tilde.team.
+
+
i awoke, not suspecting anything to be amiss. as soon as i logged in to check my email and irc mentions, it became clear.
+
+
tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.
+
+
+
We have indications that there was an attack from your server.
+Please take all necessary measures to avoid this in the future and to solve the issue.
+
+
+
at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.
+
+
when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the webmail to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!
+
+
here, i launch in to full debugging mode: what command was it? who ran it?
+
+
search ~/.bash_history per user was not very successful. nothing i could find was related to net or map. i had checked sudo grep nmap /home/*/.bash_history and many other commands.
+
+
at this point, i had connected with other ~teammates across other irc nets (#!, ~town, etc). among suggestions to check /var/log/syslog, /var/log/kern.log, and dmesg, i finally decided to check ps. ps -ef | grep nmap yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for ~fosslinux.
+
+
i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police nmap when it isn't scanning on every port?
+
+
after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that ~fosslinux had only run nmap for addresses in the 10.0.0.0/8 space. the 10/8 address space is intended to not be addressable outside the local space. how could hetzner have found out about a localhost network probe!?
+
+
finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.
+
+
it's definitely time to research redundancy options!
no matter how i found it, i still decided to add it to my blog here with bashblog. utterances is a commenting system that leverages github issues. so, for example a comment on a post shows up on github like this.
-
now we just need to figure out if it can be pointed at a gitea instance like tildegit. might be time for a PR!
“Arguing with religious people – It’s like playing chess with a pigeon; no matter how good I am at chess, the pigeon is just going to knock over the pieces, crap on the board and strut around victorious” – Anonymous
let's start by saying i probably should have done a bit more research before
-diving head-first into this endeavor.
-
-
i've been thinking about transferring my domains off google domains for some
-time now, as part of my personal goal to self host and limit my dependence on
-google and other large third-party monstrosities. along that line, i asked for
-registrar recommendations. ~tomasino responded
-with namesilo. i found that they had $3.99 registrations
-for .team and .zone domains, which is 1/10th the cost of the $40 registration
-on google domains.
-
-
i started out by getting the list of domains from the google console. 2 or 3
-of them had been registered within the last 60 days, so i wasn't able to
-transfer those just yet. i grabbed all the domain unlock codes and dropped
-them into namesilo. i failed to realize that the dns panel on google domains
-would disappear as soon as it went through, but more importantly that the
-nameservers would be left pointing to the old defunct google domains ones.
-
-
i updated the nameservers as soon as i realized this error from the namesilo
-panel. some of the domains propagated quickly. others, not so much. tilde.team
-was still in a state of flux between the old and new nameservers.
-
-
in a rush to get the dns problem fixed, and under recommendation from several
-people on irc, i decided to switch the nameservers for tilde.team and tilde.zone
-to cloudflare, leaving another layer of flux for the dns to be stuck in...
-
-
of the five domains that i moved to cloudflare, 3 returned with a dnssec error,
-claiming that i needed to remove the DS record from that zone. d'oh!
-
-
i removed the dnssec from those affected domains, so we should be good to go
-as soon as it all propagates through the fickle beast that is dns.
now that tilde.team is on a fancy-shmancy new dedi server, i've tried to get a secondary IP address
-assigned to a lxd container (which i plan to use for my personal stuff). lxd shows that the secondary
-IP is being picked up by that container, but i'm still seeing the host machine's IP as the external
-address.
-
-
i'm not sure how i'll need to configure the network settings on the host machine (now that we're running
-ubuntu 18.04 and it uses netplan for configs and not /etc/network/interfaces). another confusing thing is
-that the main config in /etc/netplan says that the network config is handled by systemd-networkd...
-
-
at least i have through the end of the year when my current vps runs out to get this up and running.
-
-
ping me on irc or email if you have experience with this.
we had something of an outage on november 13, 2018 on tilde.team.
+
+
i awoke, not suspecting anything to be amiss. as soon as i logged in to check my email and irc mentions, it became clear.
+
+
tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.
+
+
+
We have indications that there was an attack from your server.
+Please take all necessary measures to avoid this in the future and to solve the issue.
+
+
+
at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.
+
+
when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the webmail to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!
+
+
here, i launch in to full debugging mode: what command was it? who ran it?
+
+
search ~/.bash_history per user was not very successful. nothing i could find was related to net or map. i had checked sudo grep nmap /home/*/.bash_history and many other commands.
+
+
at this point, i had connected with other ~teammates across other irc nets (#!, ~town, etc). among suggestions to check /var/log/syslog, /var/log/kern.log, and dmesg, i finally decided to check ps. ps -ef | grep nmap yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for ~fosslinux.
+
+
i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police nmap when it isn't scanning on every port?
+
+
after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that ~fosslinux had only run nmap for addresses in the 10.0.0.0/8 space. the 10/8 address space is intended to not be addressable outside the local space. how could hetzner have found out about a localhost network probe!?
+
+
finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.
+
+
it's definitely time to research redundancy options!
+
+
diff --git a/blog/november-13-post-mortem.md b/blog/november-13-post-mortem.md
new file mode 100644
index 0000000..420a09e
--- /dev/null
+++ b/blog/november-13-post-mortem.md
@@ -0,0 +1,31 @@
+november 13 post mortem
+
+we had something of an outage on november 13, 2018 on tilde.team.
+
+i awoke, not suspecting anything to be amiss. as soon as i logged in to check my email and irc mentions, it became clear.
+
+tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.
+
+> We have indications that there was an attack from your server.
+> Please take all necessary measures to avoid this in the future and to solve the issue.
+
+at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.
+
+when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the [webmail](https://mail.tilde.team) to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!
+
+here, i launch in to full debugging mode: what command was it? who ran it?
+
+search `~/.bash_history` per user was not very successful. nothing i could find was related to net or map. i had checked `sudo grep nmap /home/*/.bash_history` and many other commands.
+
+at this point, i had connected with other ~teammates across other irc nets ([#!](https://hashbang.sh/), [~town](https://tilde.town), etc). among suggestions to check `/var/log/syslog`, `/var/log/kern.log`, and `dmesg`, i finally decided to check `ps`. `ps -ef | grep nmap` yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for [~fosslinux](/~fosslinux/).
+
+i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police `nmap` when it isn't scanning on every port?
+
+after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that [~fosslinux](/~fosslinux/) had only run `nmap` for addresses in the `10.0.0.0/8` space. the `10/8` address space is intended to not be addressable outside the local space. how could [hetzner](https://hetzner.com) have found out about a localhost network probe!?
+
+finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.
+
+it's definitely time to research redundancy options!
+
+
+tags: post-mortem, linux, sysadmin
diff --git a/blog/proactive-redundancy.html b/blog/proactive-redundancy.html
new file mode 100644
index 0000000..4e6f3c0
--- /dev/null
+++ b/blog/proactive-redundancy.html
@@ -0,0 +1,73 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+proactive redundancy
+
+
after the fiasco earlier this week, i've been taking steps to minimize
+the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage.
+
+
the first thing that i set up was a handful of additional ircd nodes: see the tilde.chat wiki for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team.
+
+
i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. host tilde.chat will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of {your,team,bsd,slash}.tilde.chat.
+
+
this creates the additional problem that visiting the tilde.chat site will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to debug. the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see host chat.freenode.net for the list of servers).
+
+
i'm not sure how to make any of the other services more resilient. it's something that i have been and will continue to research moving forward.
+
+
the other main step that i have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in rfc 1918.
+
+
i'd like to consider at least this risk to be mitigated.
+
+
diff --git a/blog/proactive-redundancy.md b/blog/proactive-redundancy.md
new file mode 100644
index 0000000..4adc900
--- /dev/null
+++ b/blog/proactive-redundancy.md
@@ -0,0 +1,23 @@
+proactive redundancy
+
+after the [fiasco](november-13-post-mortem.html) earlier this week, i've been taking steps to minimize
+the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage.
+
+the first thing that i set up was a handful of additional ircd nodes: see [the tilde.chat wiki](https://tilde.chat/wiki/?page=servers) for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team.
+
+i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. `host tilde.chat` will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of `{your,team,bsd,slash}.tilde.chat`.
+
+this creates the additional problem that visiting the [tilde.chat site](https://tilde.chat) will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to [debug](https://tildegit.org/tildeverse/tilde.chat/issues/8). the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see `host chat.freenode.net` for the list of servers).
+
+i'm not sure how to make any of the other services more resilient. it's something that i have been and will continue to research moving forward.
+
+the other main step that i have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in [rfc 1918](https://tools.ietf.org/html/rfc1918).
+
+i'd like to consider at least this risk to be mitigated.
+
+thanks for reading,
+
+~ben
+
+
+tags: sysadmin, tilde
diff --git a/blog/tag_linux.html b/blog/tag_linux.html
index 8bdf2a6..9d022b8 100644
--- a/blog/tag_linux.html
+++ b/blog/tag_linux.html
@@ -24,41 +24,45 @@
not sure if this is appropriately tagged, but i didn't feel like making a new
-one.
+
we had something of an outage on november 13, 2018 on tilde.team.
-
i figured i should probably get some notes down about moving off google.
+
i awoke, not suspecting anything to be amiss. as soon as i logged in to check my email and irc mentions, it became clear.
-
to start, i'll get a list of the things i was able to easily replace:
+
tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.
google drive => syncthing (with a persistent node running on my personal vps)
-
+
+
We have indications that there was an attack from your server.
+Please take all necessary measures to avoid this in the future and to solve the issue.
+
-
i'm still using:
+
at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.
-
-
gplay music/youtube
-
google maps (open streetmap isn't good enough to replace it)
-
google photos - but this is going to be replaced long-term with syncthing
-
+
when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the webmail to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!
here, i launch in to full debugging mode: what command was it? who ran it?
+
search ~/.bash_history per user was not very successful. nothing i could find was related to net or map. i had checked sudo grep nmap /home/*/.bash_history and many other commands.
+
at this point, i had connected with other ~teammates across other irc nets (#!, ~town, etc). among suggestions to check /var/log/syslog, /var/log/kern.log, and dmesg, i finally decided to check ps. ps -ef | grep nmap yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for ~fosslinux.
+
i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police nmap when it isn't scanning on every port?
+
after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that ~fosslinux had only run nmap for addresses in the 10.0.0.0/8 space. the 10/8 address space is intended to not be addressable outside the local space. how could hetzner have found out about a localhost network probe!?
+
finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.
+
+
it's definitely time to research redundancy options!
now that tilde.team is on a fancy-shmancy new dedi server, i've tried to get a secondary IP address
-assigned to a lxd container (which i plan to use for my personal stuff). lxd shows that the secondary
-IP is being picked up by that container, but i'm still seeing the host machine's IP as the external
-address.
-
-
i'm not sure how i'll need to configure the network settings on the host machine (now that we're running
-ubuntu 18.04 and it uses netplan for configs and not /etc/network/interfaces). another confusing thing is
-that the main config in /etc/netplan says that the network config is handled by systemd-networkd...
-
-
at least i have through the end of the year when my current vps runs out to get this up and running.
-
-
ping me on irc or email if you have experience with this.
now that tilde.team is on a fancy-shmancy new dedi server, i've tried to get a secondary IP address
+assigned to a lxd container (which i plan to use for my personal stuff). lxd shows that the secondary
+IP is being picked up by that container, but i'm still seeing the host machine's IP as the external
+address.
+
+
i'm not sure how i'll need to configure the network settings on the host machine (now that we're running
+ubuntu 18.04 and it uses netplan for configs and not /etc/network/interfaces). another confusing thing is
+that the main config in /etc/netplan says that the network config is handled by systemd-networkd...
+
+
at least i have through the end of the year when my current vps runs out to get this up and running.
+
+
ping me on irc or email if you have experience with this.
we had something of an outage on november 13, 2018 on tilde.team.
+
+
i awoke, not suspecting anything to be amiss. as soon as i logged in to check my email and irc mentions, it became clear.
+
+
tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.
+
+
+
We have indications that there was an attack from your server.
+Please take all necessary measures to avoid this in the future and to solve the issue.
+
+
+
at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.
+
+
when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the webmail to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!
+
+
here, i launch in to full debugging mode: what command was it? who ran it?
+
+
search ~/.bash_history per user was not very successful. nothing i could find was related to net or map. i had checked sudo grep nmap /home/*/.bash_history and many other commands.
+
+
at this point, i had connected with other ~teammates across other irc nets (#!, ~town, etc). among suggestions to check /var/log/syslog, /var/log/kern.log, and dmesg, i finally decided to check ps. ps -ef | grep nmap yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for ~fosslinux.
+
+
i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police nmap when it isn't scanning on every port?
+
+
after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that ~fosslinux had only run nmap for addresses in the 10.0.0.0/8 space. the 10/8 address space is intended to not be addressable outside the local space. how could hetzner have found out about a localhost network probe!?
+
+
finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.
+
+
it's definitely time to research redundancy options!
after the fiasco earlier this week, i've been taking steps to minimize
+the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage.
+
+
the first thing that i set up was a handful of additional ircd nodes: see the tilde.chat wiki for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team.
+
+
i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. host tilde.chat will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of {your,team,bsd,slash}.tilde.chat.
+
+
this creates the additional problem that visiting the tilde.chat site will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to debug. the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see host chat.freenode.net for the list of servers).
+
+
i'm not sure how to make any of the other services more resilient. it's something that i have been and will continue to research moving forward.
+
+
the other main step that i have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in rfc 1918.
+
+
i'd like to consider at least this risk to be mitigated.
we had something of an outage on november 13, 2018 on tilde.team.
+
+
i awoke, not suspecting anything to be amiss. as soon as i logged in to check my email and irc mentions, it became clear.
+
+
tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.
+
+
+
We have indications that there was an attack from your server.
+Please take all necessary measures to avoid this in the future and to solve the issue.
+
+
+
at this point, i have no idea what could have happened over night while i'm sleeping. the timestamp shows that it arrive only 30 minutes after i'd turned in for the night.
+
+
when i finally log on in the morning to check mails and irc mentions, i find that i'm unable to connect to tilde.team... strange, but ok; time to troubleshoot. i refresh the webmail to see what i'm missing. it ends up failing to find the server. even stranger! i'd better get the mails off my phone if they're on my @tilde.team mail!
+
+
here, i launch in to full debugging mode: what command was it? who ran it?
+
+
search ~/.bash_history per user was not very successful. nothing i could find was related to net or map. i had checked sudo grep nmap /home/*/.bash_history and many other commands.
+
+
at this point, i had connected with other ~teammates across other irc nets (#!, ~town, etc). among suggestions to check /var/log/syslog, /var/log/kern.log, and dmesg, i finally decided to check ps. ps -ef | grep nmap yielded nmap on an obscured uid and gid, which is shortly established to belong to a container i had provisioned for ~fosslinux.
+
+
i'm not considering methods of policing access to any site over port 80 and port 443. this is crazy. how do you police nmap when it isn't scanning on every port?
+
+
after a bit of shit-talking and reassurance from other sysadmins, i reexamined and realized that ~fosslinux had only run nmap for addresses in the 10.0.0.0/8 space. the 10/8 address space is intended to not be addressable outside the local space. how could hetzner have found out about a localhost network probe!?
+
+
finally, after speaking with more people than i expected to speak with in one day, i ended up sending three different support emails to hetzner support, which finally resulted in them unlocking the ip.
+
+
it's definitely time to research redundancy options!
now that tilde.team is on a fancy-shmancy new dedi server, i've tried to get a secondary IP address
-assigned to a lxd container (which i plan to use for my personal stuff). lxd shows that the secondary
-IP is being picked up by that container, but i'm still seeing the host machine's IP as the external
-address.
-
-
i'm not sure how i'll need to configure the network settings on the host machine (now that we're running
-ubuntu 18.04 and it uses netplan for configs and not /etc/network/interfaces). another confusing thing is
-that the main config in /etc/netplan says that the network config is handled by systemd-networkd...
-
-
at least i have through the end of the year when my current vps runs out to get this up and running.
-
-
ping me on irc or email if you have experience with this.
now that tilde.team is on a fancy-shmancy new dedi server, i've tried to get a secondary IP address
+assigned to a lxd container (which i plan to use for my personal stuff). lxd shows that the secondary
+IP is being picked up by that container, but i'm still seeing the host machine's IP as the external
+address.
+
+
i'm not sure how i'll need to configure the network settings on the host machine (now that we're running
+ubuntu 18.04 and it uses netplan for configs and not /etc/network/interfaces). another confusing thing is
+that the main config in /etc/netplan says that the network config is handled by systemd-networkd...
+
+
at least i have through the end of the year when my current vps runs out to get this up and running.
+
+
ping me on irc or email if you have experience with this.
after the fiasco earlier this week, i've been taking steps to minimize
+the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage.
+
+
the first thing that i set up was a handful of additional ircd nodes: see the tilde.chat wiki for a full list. slash.tilde.chat is on my personal vps, and bsd.tilde.chat is hosted on the bsd vps that i set up for tilde.team.
+
+
i added the ipv4 addresses for these machines, along with the ip for yourtilde.com as A records for tilde.chat, creating a dns round-robin. host tilde.chat will return all four. requesting the dns record will return any one of them, rotating them in a semi-random fashion. this means that when connecting to tilde.chat on 6697 for irc, you might end up on any of {your,team,bsd,slash}.tilde.chat.
+
+
this creates the additional problem that visiting the tilde.chat site will end up at any of those 4 machines in much the same way. for the moment, the site is deployed on all of the boxes, making site setup issues hard to debug. the solution to this problem is to use a subdomain as the roundrobin host, as other networks like freenode do (see host chat.freenode.net for the list of servers).
+
+
i'm not sure how to make any of the other services more resilient. it's something that i have been and will continue to research moving forward.
+
+
the other main step that i have taken to prevent the same issue from happening again was to configure the firewall to drop outgoing requests to the subnets as defined in rfc 1918.
+
+
i'd like to consider at least this risk to be mitigated.
-
-
