For those who are keeping score, I’m a dumbass.
So, I have a file server at my home that’s running FreeNAS, which is based on FreeBSD, a popular Unix distribution. I have around 7.4 terabytes of storage capacity across 5 disks; that’s a lot of personal files. I bought a single 8TB disk for the purposes of backing up this mountain of data. The problem is that my methods of making exact copies of files to the disk were failing or taking too long to complete, so the backups were slow and not always good.
When you’re dealing with terabytes of data, everything’s slow and insufficient.
Well, the problem is that I was looking at the problem from the wrong point of view. On my personal workstation, which runs Linux and uses the ext4 filesystem, I have a stack of scripts that basically run “rsync” to make exact copies to a portable 2TB disk. This method is what I tried on the FreeNAS box. Due to the way rsync operates under the hood, especially on complex filesystems where there are lots of hardlinked files, it was occasionally consuming all of the free memory on the fileserver and eventually dying after a few hours of churning, sometimes with no files actually copied.
“There has to be a better way to do this,” I thought. “I guess I’ll split the disk up into smaller chunks and make scripts to do it that way.”
And then, like a strike of lightning, it hit me: this FreeNAS box isn’t running on an ext4 filesystem — it’s running ZFS. What feature does ZFS have that ext4 doesn’t? That’s right — snapshots. Futuristic, amazing, wonderful snapshots. And what can you do with these snapshots? That’s right, you can replicate them, bit-for-bit, to another disk or remote system. Woah.
Just slap me now.
So, I did what any sane person would do: use the available tools as intended. I made a snapshot. I then wiped the backup disk clean, and am currently replicating that snapshot to the backup disk. Then, once the initial snapshot is there, I can take incremental snapshots at later dates and push them to the backup disk in a significantly shorter amount of time.
Use what’s available to you. That’s the key lesson here. This is liberating. Let’s hope this works!
It seems minor, but installing the power cables for a ham radio in a car is a major project if you want it done right. After months of thoughts and tactics, I made good on my plan to install a 25 Amp circuit into my car to power my portable amateur radio equipment. I’ve been needing this capability for a while. It gives me another option for operating away from home, allowing me to drive to any convenient spot and make HF contacts, as well as allowing me to make VHF contacts while on the road. I’ll have radio power for as long as I have gas in the tank.
It took me a while to get around to doing it, but once I started, it was two evenings well-spent. Here’s the details. Continue reading
So just what are the Holy Algorithms?
They are the automatic schemes of man, human plans to model human behavior and optimize for some potential profit, either monetary gain or behavioral control.
Humans make the algorithms for use on humans. Never forget that. Computers are only the agents of mass scale.
How to resist? Simply know they are there. Never trust product reviews; those can be automatically generated. Links provided were given to you by someone with a vested interest in giving you those links. Product suggestions go to the highest bidder. The top 40 song was decided by the label that paid the radio station to play it. The political bombshell was leaked by someone seeking a political edge in the electorate. The post reshared was promoted by someone who was told to share it; a string of memetic code in the message told them to reproduce the message. Go ye therefore and make disciples of all the lands.
It’s not the computers at fault; they are only a tool of amplification. It is your fellow humans to blame. Everyone’s a huckster. Everyone’s a charlatan. Everyone’s a P.T. Barnum. Everyone has something to sell you, including myself.
“Psst, hey mister, would you like to buy a watch? No? How about a philosophical viewpoint?”
Recognize that fact and you can save yourself. Move randomly, act against the behavioral models, react in a way that goes against the models that expect some of you to react against the models — go laterally on occasion. Make the behavioral and economic theorists scratch their heads at the lowered percentage of hits. Make them doubt their measurement errors.
The Holy Algorithm seeks to rule all for fun and profit, but we are not required to submit and genuflect. We choose our chains.
It’s apparent to me now: the modern web is designed to give us just enough of what we’re looking for to require us to make an action to find out more. We’re being teased into providing input into the system.
Did someone post an image? Guess what: that image will be cropped with just enough of the bottom edge (where the final words of a meme live, for instance) hidden from view, so we’re forced to click the image to see the full thing. The crop size can be steeper if the bots detect words in the image. That click is an input that tells sites like Facebook that we obviously want more, that we have a stronger susceptibility to clickbait.
Make a post over five lines? Cropped and hidden behind a “Read More” link. The click is an input. The second “Read More” link after a paragraph is yet another input. There’s no loss if they show you the whole post at once; there’s no cost per byte out of the web server, so why do it? Feedback and input. You have to act if you want to read more. Having fewer words doesn’t make the infinite scroll any cleaner or streamlined. It’s the input.
Search on Google for a thing? You’re given a page full of results as paragraph snippets with just enough in each to lead you on to action. None of those snippets provides a shred of what you are looking for. You have to click. It’s like the bots interpret what you search for and explicitly exclude that from the snippets. Want the definition to a word? Unless Google is feeling benevolent, you’re given a pageful of results like “Find the definition of <word> at Dictionary dot com!” instead of seeing any part of the definition itself.
What are the algorithmic effects of all this? Nobody in the general public really knows. My hypothesis is that the ad impressions on the destination sites are mostly the reason. The value of a site’s ad space varies with the number of visitors inbound from affiliates, as well as the number of visitors who stick around for multiple pageviews.
And now many of the major websites are advertising companies at their core, Facebook and Google being the major operators, and it’s incumbent on them to add value to their own system by driving up their own traffic, putting sites that aren’t part of their affiliate networks in a lower priority for search results. The presence of beacons on affiliate sites allows them to track you when you’re looking at things for sale. After looking at cameras, for instance, you go back to Facebook or Google and you see the same camera you were just looking at in a sidebar ad, and you have no control over that. You can’t click it away.
So it’s not just the site operators and content editors making clickbait articles with misleading headlines and lede images, as I’ve previously complained about. It’s the algorithms feeding us shreds of interest to goad us into action. We’re being trolled, like bad uncles sending us on snipe hunts for their own enjoyment and profit.
This is reason 4,815 why I hate the modern web.