Well, damn. The old music workstation that I was going to repurpose into my radio desk workstation has a copy of Windows 7 which is now out of support. There goes that plan.

Screenshot of notice, "Your Windows 7 PC is out of support"

I mean, I can still use it, but why spend all the time and energy integrating onto a platform that’s beyond end-of-life? I think Windows 7 is one of the best versions they’ve done, but I have to get modern.

Le sigh.

Owncloud and Fail2ban Mismatch

For the past mumble years at home I’ve had an installation of Owncloud running, and one thing that’s been on my To-Do list is to put fail2ban in front of it to inconvenience any bad actors attempting to bruteforce their way into the Owncloud server (use random passwords, yo). I initially installed F2b on my OC box and made a first pass at configs, but nothing worked. Through multiple Linux, OC, and F2b upgrades, I kept revisiting this project, going through manpages, howtos, and forums, and reading everybody’s claims of success with their solutions.

Yet mine persisted in not working.

Unique to my situation is that my owncloud.log file lives in the root of my OC data partition instead of in /var/log, and it’s in the custom log format (default), not the syslog format. The typical log entry for a failed login is:

{"reqId":"l73w0AuwXQb2dbZCxngj","level":2,"time":"January 28, 2020 21:44:53","remoteAddr":"","user":"--","app":"core","method":"POST","url":"\/owncloud\/index.php\/login?user=asdf","message":"Login failed: 'asdf' (Remote IP: '')"}

For some reason, this was not matching any of my regex attempts.

To use F2b with OC, you must do some work. The notes in this document are valid for Debian Stretch (9, currently oldstable), and some modification may be necessary for other distros and versions. OC is 10.1.0, and F2b is 0.9.6.

F2b needs an extra pair of config files because it doesn’t ship with default support for OC:

/etc/fail2ban/filter.d/owncloud.conf :
failregex = "Login failed: '.*' \(Remote IP: '<HOST>'\)"
/etc/fail2ban/jail.d/owncloud.local :
enabled = true
port = 80,443
protocol = tcp
filter = owncloud
maxretry = 3
bantime = 10800
# note the custom logfile path here. fix for yours:
logpath = /mydata/owncloud.log

Many of the forum posts say that for OC versions 9 and up, the regular expression in failregex has changed, so I found a handful of versions, tried them all, and nothing matched. Nothing. Even created my own versions and minimal matching strings, and yeah, nothing. So tonight, high on coffee, I dug into the OC docs, F2b docs, PHP docs, Python docs, everything, and here are some of the things I learned:

  • Fail2ban uses Python regular expressions, so the format conforms to Python ‘re’ module rules
  • Regexes must include the <HOST> token (or some expanded version) or it will barf
  • There are no user-visible tokens to match datestamp; that’s all done behind the scenes internal to F2b
  • Only the lines that match the internal datestamp regex will be considered for your custom regex
  • If the regex includes the ^ anchor, that starts on the first non-whitespace character immediately after the internally-matched datestamp
  • This means F2b has some internal regexes to match datetime strings in a multitude of formats, none of which you can control
  • You can find these datestamp regex formats by using fail2ban-regex --verbose <logfilesample> <regex> like so:
root@ocbox:/etc/fail2ban/filter.d# fail2ban-regex -v /mydata/owncloud.log "<HOST>"
Running tests
Use   failregex line : <HOST>
Use         log file : /mydata/owncloud.log
Use         encoding : ANSI_X3.4-1968
Failregex: 0 total
|-  #) [# of hits] regular expression
|   1) [0] <HOST>

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [0] (?:DAY )?MON Day 24hour:Minute:Second(?:\.Microseconds)?(?: Year)?
|  [0] (?:DAY )?MON Day Year 24hour:Minute:Second(?:\.Microseconds)?
|  [0] Year(?P<_sep>[-/.])Month(?P=_sep)Day 24hour:Minute:Second(?:,Microseconds)?
|  [0] Day(?P<_sep>[-/])Month(?P=_sep)(?:Year|Year2) 24hour:Minute:Second
|  [0] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
|  [0] Month/Day/Year:24hour:Minute:Second
|  [0] Month-Day-Year 24hour:Minute:Second\.Microseconds
|  [0] TAI64N
|  [0] Epoch
|  [0] Year-Month-Day[T ]24hour:Minute:Second(?:\.Microseconds)?(?:Zone offset)?
|  [0] ^24hour:Minute:Second
|  [0] ^
|  [0] ^Year2MonthDay  ?24hour:Minute:Second
|  [0] MON Day, Year 12hour:Minute:Second AMPM
|  [0] ^MON-Day-Year2 24hour:Minute:Second

Lines: 278 lines, 0 ignored, 0 matched, 278 missed
[processed in 0.81 sec]
Missed line(s): too many to print.  Use --print-all-missed to print all 278 lines

This verbose output was the thread that unraveled this whole mystery. See what’s wrong here? Of course you don’t. I didn’t. Until I did. The datestamp in the OC logfile is January 28, 2020 21:44:53, which matches none of the built-in datestamp formats. Therefore, no matter what regex I provide, nothing will ever, ever match. It must be in a matching format such as Jan 28, 2020 11:28:41 PM.

Y’feel me?

Thankfully, OC has a logdateformat workaround in the config.php file. There are already two other edits to make, so this is just one extra.

/var/www/owncloud/config/config.php :
# point to your logfile
    'logfile' => '/mydata/owncloud.log',
# set to your server's timezone (use PHP tzdata rules)
    'logtimezone' => "America/Chicago",
# Default logdateformat is 'F d, Y H:i:s', (date format) which doesn't match
    'logdateformat' => 'M d, Y h:i:s A',

You have to configure OC to log with the local timezone and use a custom datestamp format so F2b matches. Then make the changes to F2b so it all just works.

To enable F2b for OC:

  • Save your config.php; OC automatically picks up the changes.
  • Create and save the two F2b configs
    • /etc/fail2ban/filter.d/owncloud.conf
    • /etc/fail2ban/jail.d/owncloud.local
  • Restart F2b. service fail2ban restart
  • Then tail -f your OC logfile while attempting to make a few bad logins in the OC web interface. Note the new entries on each failure.
  • Once you hit the threshold for failed logins, F2b will trip and create the iptables rules to block your test IP for web traffic (80/tcp, 443/tcp).
  • Verify with iptables-save or fail2ban-client status owncloud:
root@ocbox:~# fail2ban-client status owncloud
Status for the jail: owncloud
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     3
|  `- File list:        /mydata/owncloud.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     1
   `- Banned IP list:
  • unban the IP with: fail2ban-client set owncloud unbanip

This post is half sharing-of-knowledge and half bragging. If you’re stuck, try fixing the busted Owncloud config.php. I feel like I should open a bugfix request with the Owncloud team, or open a bugfix request with Fail2ban to ask for another datestamp format to be added. Either/Or. Since F2b doesn’t ship with OC support, they’ll probably poop on that idea and shovel it back. Either way, I’m done, and this problem is fuckin’ Resolved.

Warming Trends

Wow, yeah. I was just reading a post from June 1999 where I was bemoaning my newfound gainful unemployment. In it, I had this wonderful gem:

I would love to make a career out of my life-long hobby, computer science, and get paid for it. I’d be much happier. There’s a lot to be said for “quality of life.”

Happiness is immensely relative, and you don’t know you’re in it until you look backward across the span of time. I’m not immensely happy these days, even after winning this anticipated career in computer science. But, I will say that compared to those innocent, doe-eyed days of 1999, I’m much better off now in 2020.

As much as I yawp and yarn for them halcyon days of yore, I cannot truthfully say that my trajectory is anything but toward a better life than all my yesterdays. I’m thankful for the opportunities that have been granted, and for those I’ve created. A year after that post, I had finished my first paid web-development gig, and it taught me what I needed to know to land my first tech job in Austin. Everything else has been incrementally upward since then.

I hope we all have our own seasons of increase.