Numb Dive Thumb Drive Dumb Live

A few days after purchasing my 2012 Honda Civic, I’m beginning to learn a few of its quirks. The car comes stock with an audio unit that plays AM/FM, CD, Line-In, as well as USB support for thumbdrives and iPod. “How cool is that?” I asked. So I loaded up one of my spare thumbdrives with a miscellany of tunes from my collection, hopped in the car, plugged it in, and started scrolling through the list.

What I found was an unsorted collection that had neither rhyme nor reason in its sort order. Tracks from Tool were sorted before Combichrist. All the tracks on The Knife’s “Bleeding the Habitual” were completely out of order. The hell?

I did some digging around. Apparently, the MP3 playback feature sorts all of the files on the thumbdrive according to when they were copied onto the device. Yeah, you read that right. The thumbdrive must be formatted FAT32, an old hard disk format written by Microsoft before it made NTFS the default in all its OSes. If you open a command line window and change drives to one with a FAT32 format and then do ‘dir’, what you’ll find is a somewhat unordered directory listing. Why is that?

Under the hood, FAT32 stores references to all its files one-by-one in a data structure on the disk; when you add a file to a FAT32 directory, the reference to that file is appended to the end of that data structure. When software gets a directory listing, it will, in the very least, get an exact copy of this list. The indexes, you see, are exactly as written: one-by-one, appended to the end. If you want to sort the files on any kind of criteria such as modification date or alphabetical by filename, you must first read in this directory list and sort it yourself with your own software. This is very low-level stuff.

I can understand where the Honda firmware designers opted to sort files based on FAT32 directory entry; it’s cheaper, processor-wise and memory-wise, to query the directory structure and run with that instead of having to read it in and then create the data objects in memory to sort things alphabetically. It takes less time and memory for the radio to do so, and it requires less software engineering talent and money to write and test the code. However, it’s a rather lazy thing to do so, considering that even the smallest of modern embedded computer systems can have as much power and space as necessary if the engineer is crafty enough.

All told, I think it was a budgetary decision on Honda’s part; if their engineers can save $5 per unit for however many millions of units sold, then they’ve made their upper management very, very happy. Someone’s getting bonuses. The downside is that the end user gets a higher degree of confusion which is exactly the kind of thing nobody needs when they’re behind the wheel. To find their music in their collection, they’d have to move their eyes from the road to the screen to scrutinize page-by-page to find the music they want. This is a bad thing — distracted driving kills.

So. How do we work around this shortcoming? Well, luckily for us there is a Linux command-line utility called Fatsort. You can use it on any FAT32-formatted partition to rewrite all of the directory entries so that they’re in alphabetical order. A typical invocation is as follows:

fatsort -f -I -c /dev/sdg1

What this does is to apply an alphabetic sort while (-c) ignoring case, (-I) ignoring prefixes like “a” and “the”, and (-f) forcing the reorder on the mounted filesystem at /dev/sdg1. Your invocation will most likely differ.

After dumping all the music I could fit on the thumbdrive, I processed it with fatsort and plugged it into the head unit. Now all my artists, albums, and songs are sorted alphabetically so I can keep a mental map of where to scroll without taking my eyes off the road too much. I can now know with confidence that King’s X is just before KMFDM.

I wonder if Honda has an updated firmware ROM that fixes this sort of problem. One can only hope. Then again, asking a specialist mechanic at the dealership might be like talking to a wall. Not everybody pays attention to low-level details.

Transcode Is the Home of Count Flacula

Now that my CD collection’s ripped to FLAC, I had to filter through my entire MP3 collection and pull out all of the MP3 rips of said albums. That took a few hours, to say the least. Still have those copies, just in case, but after symlinking my FLAC folder into my music folder so my player software can pick up the new files, I really don’t need the old MP3 rips. But they’re there, y’know, just in case.

I use Rhythmbox for my music listening. It’s OK as a player, and it’s got some…idiosyncrasies, but it’s usable. Usually. In my early phase of ripping my collection, I examined Rhythmbox’s handling of FLAC content. It plays fine, edits tags fine. It will even transcode the FLAC if you transfer it to a portable media device. The selection of file format is automatic based on the player, but will default to MP3. In your preferences, under “Music” tab, use the dropdown to select your preferred format — this is usually for ripping CDs with Rhythmbox (ugh, use Sound Juicer instead), but also applies to transcoding for media players. I edited my MP3 profile to encode at 256KB instead of my old 192KB.

That leaves the final piece of the puzzle: sharing this collection with portable thumbdrives. According to some docs out there, you can have Rhythmbox automatically transcode when copying media to a flash thumbdrive by placing an empty file called .is_audio_player in the root folder of the device. Then, when you insert the thumbdrive, Rhythmbox will notice, check for the file, and act like it’s a portable media device if it sees the file. Then you can drag-drop the files to the drive, and everything will be automatically transcoded. It’s a cludge, but what the hell.

As an aside: I know the latest version of Ubuntu desktop ejected Rhythmbox and is now using Banshee as the default player. I assume it’s got similar functionality, but I’ve not tested it myself. I’m just not ready to jump into “Natty Narwhal” or whatever they’re calling it. Progress comes with clenched fists.

Diss Order

This question goes out to all you librarians and taxonomists in my readership.

I have a large music collection, and I make every effort to keep all of my file tags as clean, correct and complete as possible. This allows me to easily search my collection and drill down to the artist, album, and song I demand to hear at that moment. If you’ve ever seen my CD collection, you’re aware of my meticulous arranging and sorting by certain criteria. The same is with my electronic collection. Physically, I prefer to sort by artist, then by album release date. Electronically, I can sort by any taxonomy I so choose. Easy enough.

So, going back to my library research orientation in my first year in college, I learned that books are sorted by subject, then by the author’s name, last name first, and then by book title. If the first word of the title is an article like “a”, “an” or “the”, it is moved to the end of the title after a comma and the book is sorted appropriately. So if I were to search for H.G. Well’s “The Time Machine”, I would search in the fiction section under the author “Wells, H.G.”, then for “Time Machine, The” somewhere after “Ten Thousand Leagues Under the Sea”. This makes sense. If the author was a publishing group or authorship can’t be pinned to one author, then it would be sorted under the group name.

I do the same with my music collection. If the album is published under a band name, then it’s sorted alphabetically under the band name. Aerosmith would be sorted before Aphex Twin. But if the album is published under the artist’s name, then it’s sorted by the artist’s name, last name first. So Fiona Apple would be sorted between Aphex Twin and Apples In Stereo under Apple, Fiona. This makes sense.

When I first built my music collection, back when music software wasn’t so smart, I would manually put articles at the end of the album and song titles because the software could clump all of the “The”s together, etcetera. It’s a pain and a hassle to do it manually, and sometimes the “extra album info” features of the software would break because it didn’t recognize the album title, but I lived with it. Luckily, modern music software has gotten smart about the use of articles in the song and album titles and sorts appropriately in the music browser interface.

But the problem comes with the artist names, which I store in my particular, perfectly reasonable way according to my training and my experience with sorting physical media: last name, then first. However, the software doesn’t know that I’m listening to Fiona Apple; it searches the web for Apple, Fiona and finds no data to show. Just like the old days with the title articles, so it is with artist names.

So what do you guys do about this? Is saving the artist name as “Last, First” still viable, or is it a vestigial relic of an older technology? I like to sort by this method because it makes sense to me; I shudder to think about trying to scroll through a page of Michaels just to get to Michael Hedges. I don’t know, maybe doing the “Last, First” method is as obsolete in the electronic realm as typing double spaces after sentences; a complete necessity on technologies now rendered obsolete. Should I get modern and save artist names in the natural format? Most player software has a search box to allow me to jump straight to my desired songs, but I’d have to change my habits.

What’s your thought?

Crystal Clear

I suppose the downside to using FLAC as a codec for storing your music is that the file sizes are much, much larger than MP3. Based on my current statistics, each album will average around 340MB on disk, which seems like a lot but it’s not bad considering the Red Book Standard for CDs declares 700MB total capacity per disc.

Here’s a sample comparison between MP3 and FLAC using Rush’s album “Presto”. The MP3s were generated with the LAME encoder at 192Kbit, 44.1KHz, stereo. The FLACs were generated with the FLAC encoder, medium compression setting.

  • MP3: 73282 bytes (71.5MB)
  • FLAC: 342188 bytes (334.2MB)
  • Overall storage growth: 467%

That extra quality comes at a cost. However, with the dropping prices of large hard drives, storage space becomes inconsequential.

The second drawback of using FLACs instead of MP3s is one of hard drive performance. With the smaller MP3 files, the audio player can read in the entire file and cache it in memory instead of hitting the disk constantly for the next data block to decode. FLAC players, unless they’re written to use a larger block of memory to cache the larger file, will have to hit the disk constantly throughout playback. You may run into situations, as I have, where the player will run out of audio data to send to the speakers if you’re doing something that’s creating extra disk activity. Saving files, copying files, anything to do with adding work to the disk may crowd the music player’s file accesses so it has to stand in line to read the data. This can be overcome with faster disks, larger caches, or smarter music players.

All that being said, I’m glad I’m switching to FLAC. I’m actually hearing the music clear as a bell, just as it’s mastered to the actual CD. All the little nuances, the sonic fluttering in the background, and tiny little noises in the studio, it’s all there. And FLAC, since it’s a perfect copy of the CD material, retains the aural phasing and panning between stereo channels, so if the material’s recorded to “come out of the speakers”, then it comes out of the speakers. MP3 processes all this and crunches it down aurally into only the important pieces of the sound and drops the rest.

It’s good to hear my music again.

Catching FLAC

Last weekend, I began the slow, arduous process of re-ripping my entire CD collection into files easily playable on my computer. This time, instead of ripping into 192kbit MP3 with the LAME codec (like I did last time), I’m ripping them into FLAC. This has important implications.

First and foremost is that FLAC is lossless, meaning no data is thrown away between the transition from CD to the final sound file. MP3 is a lossy codec, and uses tons of statistical mojo to analyze the sound data of the CD and throw away the bits that your ears can’t hear, crunching the file size tremendously. The problem with this method is that you’re losing the quieter nuances of your music. FLAC’s strength is that it’s able to take the input waveforms and chop them up into similar, easy-to-compress chunks, making the file smaller than the original uncompressed form but on playback the audio is a perfect, exact copy of its original form.

Secondly, since FLAC doesn’t compress the file sizes as well as MP3 (with the obvious quality tradeoffs), the overall space needed to store my music collection has grown tremendously. Instead of storing an entire album in roughly 80 megabytes of space, it now takes an average of 350 megabytes. That’s a large bite to swallow, but with the falling prices of high-capacity hard drives, it’s nothing nowadays. Considering the audio CD format stores around 700 megabytes, that’s not so bad.

I’ve been meaning to do this, because even with my bad ears I can still sometimes hear the strange audio artifacts of the MP3 compression — called “sizzle” in the industry — when I’m listening to my stuff. After I ripped my first disc and gave a listen, I was shocked at the quality difference. There were little pieces of the sound, stuff from the studio, or the audience, or quiet stuff put into the mix, that I never knew was there after listening to the MP3-encoded form for years. The sound came out of my speakers; FLAC saves the exact same stereo phasing that’s mixed into the CD in the final file, and no amount of MP3 bitrate is going to capture that level of nuance. I’m shocked.

So last weekend, I bought a 1 Terabyte hard disk (that’s roughly 1,000,000 Megabytes), installed it, and started ripping the CDs on my shelf. Within two days, I had the shelf of CDs I’ve acquired since 2007; about 60 discs total. And then I cracked open the 120-pound crate of CDs that I’ve collected since my first disc in 1991. These were packed up at my last place, and I’ve just now gotten around to digging them out. I’m about 1/8th of the way through my entire collection, so I expect this to take a while.

When it’s all said and done, my hope is that I will never have to break out a CD again to get quality audio. The end FLAC files can be used as perfect copies to produce any sort of MP3, OGG, or next-generation compressed audio file for ease of portability. Any other use (like for listening at home), I can rely on the FLAC.