mastodon postgres upgrade fun
howdy friends!
if you’re a mastodon user on tilde.zone (the tildeverse mastodon instance), you might’ve noticed some downtime recently.
here’s a quick recap of what went down during the upgrade process.
tools and services
inspired by tomasino's recent post, i'd like to detail some of the stuff that i rent and use.
general update
it's been quite a long time since i posted anything here on this blog.
i've been playing with gruvbox-css, decided to switch my blog to use it and finally realized it's been almost a year since i posted anything at all.
i don't have anything in particular to post today so i figured it would be good to give a general update since the last time i posted.
networking nonsense
i've recently been working on setting up drone ci on the tilde.team machine. however, there's been something strange going on with the networking on there.
raid nonsense
last week i did some maintenance on the tilde.team box. probably should have written about it sooner but i didn't make time for it until now.
proactive redundancy
after the fiasco earlier this week, i've been taking steps to minimize the impact if tilde.team were to go down. it's still a large spof (single-point-of-failure), but i'm reasonably certain that at least the irc net will remain up and functional in the event of another outage.
november 13 post mortem
we had something of an outage on november 13, 2018 on tilde.team.
i awoke, not suspecting anything to be amiss. as soon as i logged in to check my email and irc mentions, it became clear.
tilde.team was at the least inaccessible, and at the worst, down completely. according the message in my inbox, there hade been an attempted "attack" from my IP.
dns shenanigans post-mortem
let's start by saying i probably should have done a bit more research before diving head-first into this endeavor.
i've been thinking about transferring my domains off google domains for some time now, as part of my personal goal to self host and limit my dependence on google and other large third-party monstrosities. along that line, i asked for registrar recommendations. ~tomasino responded with namesilo. i found that they had $3.99 registrations for .team and .zone domains, which is 1/10th the cost of the $40 registration on google domains.
i started out by getting the list of domains from the google console. 2 or 3 of them had been registered within the last 60 days, so i wasn't able to transfer those just yet. i grabbed all the domain unlock codes and dropped them into namesilo. i failed to realize that the dns panel on google domains would disappear as soon as it went through, but more importantly that the nameservers would be left pointing to the old defunct google domains ones.
i updated the nameservers as soon as i realized this error from the namesilo panel. some of the domains propagated quickly. others, not so much. tilde.team was still in a state of flux between the old and new nameservers.
in a rush to get the dns problem fixed, and under recommendation from several people on irc, i decided to switch the nameservers for tilde.team and tilde.zone to cloudflare, leaving another layer of flux for the dns to be stuck in...
of the five domains that i moved to cloudflare, 3 returned with a dnssec error, claiming that i needed to remove the DS record from that zone. d'oh!
i removed the dnssec from those affected domains, so we should be good to go as soon as it all propagates through the fickle beast that is dns.
tags: linux, sysadmin, tilde, dns
lxd networking and additional IPs
now that tilde.team is on a fancy-shmancy new dedi server, i've tried to get a secondary IP address assigned to a lxd container (which i plan to use for my personal stuff). lxd shows that the secondary IP is being picked up by that container, but i'm still seeing the host machine's IP as the external address.
i'm not sure how i'll need to configure the network settings on the host machine (now that we're running ubuntu 18.04 and it uses netplan for configs and not /etc/network/interfaces). another confusing thing is that the main config in /etc/netplan says that the network config is handled by systemd-networkd...
at least i have through the end of the year when my current vps runs out to get this up and running.
ping me on irc or email if you have experience with this.
tilde.team news
hey hi hello!
it seems that i haven't written anything on my blog in quite a while...
time to fix that! i've been quite busy in the last month or so with a lot of new ideas an energy for tilde.team.
after rediscovering my account on tilde.town, i hopped in the irc there and my enthusiasm translated into a couple new members over here on the ~team.
our irc has been somewhat more active recently which is awesome:)
some of the new updates in the last month:
- tildegit (our own gitea instance)
- tildemail with postfix and dovecot for smtp/imap as well as local command line mail in mutt and alpine
tildeuser script wrapper with submission and approval flows- password auth disabled
i'd like to make use of our new mailserver, so shoot me some mail. i never get enough personal mail. it's all still privacy policy update notices. :(
see you soon!
hi there
welcome to my tildeblog
you are here.
i like to mess around with linux sysadmin shtuff. tilde.team is my baby. let me know if you want to get involved.
thanks for being awesome. keep it up.