Todd's TechnoGeek Journal

The shingle over my door says "Ye Olde Securitie Shoppe." Well, not really, but it would if they'd had security consultants back in the middle ages. Supposedly, that means I can track down suspicious network behavior at the drop of a packet. Alas, real life is much more complicated.

Back when I was having problems with Charter (I say that like problems with Charter are *ever* over), one of the concomitant problems I was experiencing was the inability of the LAN devices to get valid DHCP leases from the Netgear router, even when I *knew* the router was up and running properly. By sequentially removing devices from the network, I learned that taking my mail/web server offline allowed all other devices to get a lease. Trying to get or renew a lease with the box online resulted in a variety of nonsensical behavior, such as receiving DHCP offers from the router without accepting them, or timing out while waiting for a DHCP server response. Very strange.

Running a packet sniffer showed that, at periodic intervals, some process on the mail/web server would begin spewing millions of tiny SYN packets to an outside web address. The packet traffic would essentially flood the lame-o Netgear router, overflow it's connection tables, cause it to drop packets, and essentially take the darn thing offline. Stranger and stranger.

Trial and error showed that it was Apache that was causing this behavior. Apache couldn't even be stopped normally; I had to manually kill all the child processes and renew the IP address for it to stop.

The logs showed nothing, and packet captures were pretty inconclusive. I couldn't spot any obvious triggering behavior.

I reinstalled Apache, and set up a rotating packet capture log, so hopefully I'll be able to dissect the behavior in more detail if it recurs. Best practice, of course, says I should assume the box is compromised and take it offline to rebuild, but--since this is the real world--I have to pick my battles. Without more obvious signs of an intrusion, I'm going to chalk this one up to a corrupted application rather than a DDoS with my box as a zombie slave.

Let's hope time doesn't prove me wrong. :)

So, after three solid weeks of difficulties, it looks like my most recent drama with Charter is over. Let's hope so.

How did we get here? Well, for starters, call centers are designed around the concept of "customer elimination." The idea is that, if all your customers just shut up and go away, the company will flourish. Of course, the customer elimination strategy works best when the customers shut up and go away, but still continue to pay for services they aren't getting.

Charter spent weeks trying this strategy on me. Of course it failed, because I'm an ornery cuss who happens to have a better-than-average grasp of how things work. So, when they kept telling me that there was no problem, I kept ratcheting up the volume *and* the number of calls per day until they agreed to fix the problem.

Today, after three weeks, they finally sent a new tech out who promised not to leave until the problem was resolved. We tested lines, we swapped BNC cables, we rotated modems, we rerouted CAT-5 patch cords, and generally tried everything in the book to identify the issue.

It turns out that, whatever else was going on, I somehow managed to aquire TWO bad modems in a row. The old Surfboard modem, which has been quietly blinking away for three years, was no longer reliably bridging traffic. And against all odds, the brand-new Linksys modem which I'd just bought from Office Depot had the exact same problem! How weird is that?

Both modems showed up as fully synchronized, and the cable company insisted that the modem had aquired an IP address. Unfortunately, both modems still refused to transmit TCP/IP traffic reliably.

The Charter guy finally got another tech out here to swap out my Charter-provided (and apparently defective) Surfboard modem for a new Ambit model. So far, that seems to be working like a charm.

I am very grateful that the technician I dealt with today was responsible and intelligent enough to work with me until the issue was resolved. I'm still angry that the company he works for kept practicing their black arts of customer elimination on me, though.

Maybe it's not too late to investigate the wireless ISPs in town.

Today has been a bad day in geekdom. I've been experiencing high--and sometimes total--packet loss inside my own LAN. That's never a good sign.

Part of the problem is that components have been booted, rebooted, and swapped around while trying to troubleshoot cable modem issues with Charter. So, after a few fruitless hours of trying to identify the problem, I opted for a full and complete network reset.

I powered off all switches, hubs, servers, and workstations. I left them all off for about five minutes, and then booted them up one by one. First, the cable modem; next, the router. After that, each server is turn was brought up.

Now, I'm still having the occasional weirdness with DNS resolution to the rest of the Internet, but at least the LAN appears to be stable. Thank goodness!

Monopolies are fairly predictable. They are set up to provide you with the lowest level of service for the highest customer charges, and you never have any recourse.

The bright lights at Charter have been telling me since last Wednesday that there was a "temporary" network problem in my area, but assured me that technicians were hard at work on the problem. Fast forward to today, though, and a very different picture emerges.

Apparently, five days of 47-98% dropped packets is not sufficient to rouse the somnolent worker bees over at Charter from their sloth. No, in fact, the "outage board" doesn't even register an outage in my area. So, off to the land of useless tech support I go.

On my third try today, I get connected to some genius who tells me that they can't support me since I'm not running Windows XP. I tell them it's a network problem; I don't need OS support. But no, the rule in the land of slothful monopolies is that you must run an approved OS, or they won't even talk to you. Never mind that I can assure them that layer two connecions work fine; it's just the IP layer that's affected.

Fine, fine...Heaven knows I have nothing better to do than dig out a Billyware virus-attractor--uh, I mean "Windows laptop"--and allow some underpaid script-reading moron to tell me how to renew an IP address at an MS-DOS command prompt in order to determine that something just *might* possibly be wrong on their end.

So, five days, two Internet chats, and three phone calls later, the eager beavers at Charter will be rushing right out "sometime on Tuesday" to figure out how they managed to screw up this time. Boy, I might have been angry about the whole thing, but luckily each and every employee thanked me woodenly "for choosing Charter." Now *that* clearly shows their customer-focused zeal!

Well, it looks like I violated the cardinal rule of geekdom ("Back it up regularly, for Arioch hungers!"), and my previous journal has gone to the great bit-bucket in the sky. *sigh* I had some really geektastic stuff in there, too.

Still, like technology, life continues to move forward, so it's with minimal regret that I'm reviving the blog using some new blogging software. So far, I like it.

NanoBlogger has some nice features: it's written in bash, it has no weird dependencies, and it is very easy to set up multiple blogs with it. Heck, I couldn't even manage to get a *single* instance of Wordpress running under Debian, and Blosxom wouldn't work out of the box, either.

Anyway, to misquote a certain haunted young lady, "Weeeeeeeee're back!"