Fixed a recent problem in which I set up the conditions for failure years ago.
My Linux PC at home has been having hardware issues. It’s been in service for 13-ish years and, electronically, it’s falling apart. It was kernel panicking while transferring files from my phone over USB3. After enough of this, I set about to diagnose and debug why it was panicking.
During this action (wherein I hulk-smashed some memory sticks and permanently damaged 2 of the 4 DIMM slots, like an asshole), other problems cropped up. For instance, NetworkManager appeared to have lost the network config, so I set it back up. “Oof, there might be file corruption! I need to find a new PC.” Also found some pictures that I had copied that were also corrupted (thankfully I had backups in a zip file). Huh! That’s concerning, because I also have a mail server on this machine.
So, I noticed later that I was unable to reach that mail server from my phone when I was out on my 4G connection. Using Wireshark, I found the TCP SYN packet was getting through the gateway and landing at the PC, but the PC wasn’t sending back a SYNACK. It was just dropping. Likewise, I couldn’t ssh into the machine from outside. Those packets were also being dropped.
I won’t go into the details of all the hacks and fixes and searches I tried for finding out WTF was happening. There were many hacks. Oh, so many. I should know this! Because I’m an expert at computering, I’ll have you know. Expert!
Dreamed about it this morning, how I’d look at
iptables and examine all the rules and try that angle. So I got up and bashed my forehead against the screen. Dumped the rules and pored over them all. It should work! But it didn’t.
Years ago, when reinstalling this PC after a crashed disk, I set the iptables INPUT chain to have the default rule DROP, so it would ignore all inbound traffic that’s not approved (don’t worry, the local LAN is in the whitelist). So, on a whim, I changed the default rule to ACCEPT. And you know what happened?
I could reach my mailserver.
So one of the rules is fucking me over.
:INPUT DROP [31:9840] -A INPUT -i lo -j ACCEPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -i enp3s0 -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -i enp3s0 -p tcp -m state --state NEW -m tcp -m multiport --dports 25,143,993 -j ACCEPT -A INPUT -i enp3s0 -p tcp -m state --state NEW -m tcp -m multiport --dports 80,443 -j ACCEPT
On their face, these rules are perfect and acceptable and should work.
But wait! What’s this?
root:/# ip a 2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
Where the fuck did
enp3s0 get off to? Ah goddammit. When the USB3 chipset disappeared along with its Southbridge root controller, the Linux kernel enumerated the Ethernet interface at a different name.
enp2s0, and suddenly my firewall rules no longer matched. Packets dropped.
I’m a dullard and a fool. I shot myself in the foot, but only after going for the long play. Why did I bind the rule to the interface name? Why not wildcard? I’m an idiot.
I removed the
-i interface matches from the iptables rules so they bind to the wildcard interface, and set the INPUT chain default rule back to DROP, and everything works again.
Really, this highlights the importance of getting off of this failing hardware before true corruption happens. Toward that goal, I bought a gaming PC from a friend (actually built in this decade), and I need to migrate to it this week while I design and purchase a new PC in the coming month now that xmas is over.
Dammit, y’all. God dammit.