Ahhh, Christmas, a time for food, family, and of course, gifts. Some of my friends this year, however, are giving me gifts that cost me money. These gifts come in the form of mobile phone text messages, which are not included in my cell phone plan. T-Mobile charges me 50 cents a pop for every one sent and received. Many of my friends have sent me Christmas greetings the past few days via text message, and I can just see the ogres at T-Mobile jumping with glee as my bill skyrockets. Oh well. At least I have friends.

If you’ve been at Illinois Tech this week, you’ve had the absolutely wonderful opportunity to see what carnage a Denial of Service attack against a core DNS server can create. Now, I honestly don’t know much about network infrastructure or the mechanics behind a Denial of Service attack; frankly, my passion is software, not hardware, so I’ve never sought out the knowledge. (Unfortunately, HowStuffWorks.com doesn’t have anything specific on DoS. They do, however, have articles on routers, firewalls, and “Internet Infrastructure,” for those that want to learn more about the hardware that makes the Internet work.) But despite my lack of knowledge, this past week has been an excellent chance to experience the effects a DoS can cause.

A DoS destroys your ability to use the network – much moreso than I ever thought possible. For my own sake, I am going to write a little bit about what I’ve seen and make some conclusions about what went wrong, and even try to suggest possible solutions that we can implement to solve things in the future. But before we start, a little disclaimer. The network people here at IIT are not the most apt people in the world, but they’re not the most inept either. We do have some truly brilliant people around here, but frankly, they don’t get recognized or listened to nearly enough. So before you start railing on the fact that if you were a network engineer here, this never would have happened, let me say this: the main problem I have seen this past week is not a lack of technical experience, knowledge, or anything like that; the problem is a lack of communication. I cannot stress that enough. So let’s get started.

On Sunday evening, Felix, one of our network engineers, noticed an inordinate amount of traffic to random ports and stuff, and suspected a worm or virus had made its way onto the network. In an effort to contain the attack, he blocked the MAC addresses of 160 computers with the highest traffic on the network at the time, effectively banning all network traffic from those people. The thought was that this would diagnose the problem – block the addresses, watch things go back to normal, then go through the PC’s that were blocked and figure out what worm or virus had caused the problem, fix it, and reopen the network.

Problem was – blocking the addresses didn’t fix the problem. In fact, it got worse. The DNS server started going down, and then finally went kaput altogether. This effectively knocked out internet access for all of campus. The internal network still worked, and some cached DNS entries still resolved all right, but most people were locked out. Here at the Support Desk we got back up and running for awhile by switching to an external DNS, but we couldn’t exactly tell the whole campus to do that, because it wouldn’t have solved the root of the problem. Eventually, because of the bogus traffic that was being spit out from our network and our DNS server, our own ISP blocked our DNS server.

This is where things started to go downhill – fast. Felix provided us with the list of blocked MAC addresses initially, but by the time the DNS went down, which was late Monday afternoon, that list was out of date. On top of that, because of the rampant random network issues, we couldn’t tell a customer what was wrong when they called. They might be blocked, they might not. It might just be the random DNS problems, it might not. We had still not been provided with a process through which people could get checked and unblocked if necessary. Thus, problem number one:

1) We at the Support Desk were not provided with information, sketchy or otherwise, to tell people when they called.

So we started telling people whatever we knew – there were network issues, there were some people with blocked MAC’s, we didn’t know when things would be back up. It is useful to remember that at this point, OTS had no idea what the problem was (the DNS server was the problem, but it had been brought down by the DoS attack, which hadn’t been identified yet), and no information had been sent out to the IIT community. Hence, problem number two:

2) OTS was extremely slow at getting public information out to the IIT community, and when they did, provided it via unreliable methods (email, mainly).

Yup, most information went out in the form of emails, which is certainly a valid form of communication, but it shouldn’t be the only one. But more on suggestions later…

As the week dragged on, and more and more customers became agitated from a lack of information and a seeming lack of action on our part, things just got worse. Apparently, there was much debate about the actual cause of the problem was. Virus scans and other searches had revealed nothing on many of the blocked PC’s, except a small VBS worm that remained under most virus scanner’s radar. One camp held that there was just a problem with the DNS server - something was wrong with it. The other camp held that we were under an attack. One – it’s our fault, two – it’s not our fault (at least, not directly).

It was suggested at this point that we remove the DNS server and replace it with a fresh one, which would make the choice between the two theories simple. But, for some unknown reason, this was resisted for quite some time. Eventually, a new DNS server was put in. It lasted 30 minutes. It was clear we were under attack. Thus, point number three:

3) When DNS went down, it should have been replaced immediately. That would have made it clear from the beginning that we were under attack.

During all of this hullabaloo, we at the Support Desk remained ignorant of any information regarding the problem, its causes, and what OTS was doing to resolve it. The calls we received were getting angrier and angrier as final’s week loomed closer and they still couldn’t get to their course websites, or do research, or browse the net to relax, or anything. In addition, we didn’t have an updated MAC block list, which had been changed (at one point it got to 400 MAC’s) several times at this point. Points numbers four and five:

4) The Support Desk was not provided with up-to-date information (such as updated MAC block lists) that would have assisted them in diagnosing the problem(s), and cut down the workload on other OTS divisions, and maybe, just maybe, would have left customers feeling a little happier about the state of things.

5) When OTS Communications went out, they were overly vague and seemed to take forever to write. Communications went out with information that seemed out of date as a result.

Well, those are the main points. The core problem was a lack of communication and information-sharing amongst departments. If the Support Desk is a voice for OTS, then we have to know what to say, and we simply weren’t provided with any information. Also, (and this really bugs me) the OTS communications just told people to call the Support Desk if they had a problem, presumably to get more information, but we didn’t have anything to tell them. So these were the problems, now what about solutions? Here are my thoughts, for what they’re worth.

Provide the Support Desk access to information such as MAC block lists. The info has to be up-to-date, otherwise it’s not really useful.

This seems to be a no-brainer to me. If we’re going to be sending out emails that say, “Call the OTS Support Desk for assistance in troubleshooting your network access,” then the Support Desk has to have up-to-date information about it. If people are going to be calling with their MAC’s, we can’t be looking at a list over a week old. In fact, it was just this morning that we finally got the updated list, after it had grown to over 400 people and shrunk back down to about 25.

Make one person responsible for communicating with the students.

By this, I don’t mean that one person should be taking calls with problems. That’s the Support Desk’s job. What I mean is that one person should write the communication emails, and should be responsible for getting accurate information from the appropriate people to send out. The process would be as follows: Network goes down… Communicator calls Networks, finds out problem X is the cause, gets an ETA… Communicator sends out email information, and makes arrangements to notify via other means. See below. The more I think about this, someone at the Support Desk should have this responsibility. But it should only be one person.

Communication should be done through a variety of means.

In this instance, email was, and usually is, the primary form of communication. But we need to send out voice mail to faculty, put up flyers in MSV and SSV, and, dare I say it, hold a press conference. This isn’t always necessary, but in this case, it would have been nice if students could have gone to a public meeting one evening in the MTCC auditorium or something, heard a brief statement about what the problem was and what action we were taking, and had a chance to ask questions and get some straight answers. People who aren’t in the dark are less likely to make up their own explanations for why something went wrong. Ending speculation can help end angry customer calls.

I guess that’s really all I have. This week has been trying for all of us here at OTS, but I hope that maybe the staff here will learn from this experience and make it better the next time this happens. But of course, I am just a lowly peon here, so what do I know?

By the way, for anyone that cares: The problem was caused by some (like 8-10) PCs on campus getting hacked (externally, internally, who knows) and participating in a DoS attack against the DNS server. The problem now seems to have been contained, and about 25 people still remain blocked. Yeah, DoS sucks, but it is kinda cool to realize that 8-10 machines can destroy a network connection serving a university of 3000 people. Kinda cool.

I started looking at some open-source software and came across Blender, a cool-looking 3D modeling program. On their homepage, I noticed this image (it may no longer be on their page):

Now what’s wrong with this picture? Well, the designer of the image was not very careful when selecting the words to use. The words “than” and “then,” while differing only by one letter, are vastly different in meaning. Then is most commonly an adverb, but can sometimes be a noun or adjective (time- related). Than, on the other hand, is a conjunction, used to compare things. I am better at English than you are, for example. So when you want to compare things, as the person making this advertisement image apparently wanted to, you want to use than.

I think the confusion regarding this stems from the use of “then” as a time- related adjective or noun. For example, “I was there then,” or “the then director of OTS” are valid uses of the word, which confuses people when they want to compare things, especially in time-related constructs, such as “more than before.” Then implies a previous state of being, so previously – then - there were less than there are now. This, I think, is the root of the confusion. If you really want to get your head spinning, try this: “There were more employers there then than previously.” I think that’s grammatically correct, but many people seem to get this confused. It could also be that they are often pronounced similarly if not exactly the same, so drill yourself on it if you’re having problems remembering.

Since I have noticed some common grammatical and language-related mistakes at work, in the IPRO, and just on blogs and websites in general, I think I might start a regular column on here with common problems people have with similar sounding, yet different meaning words. After all, I am the master of all that is English. And I wrote a novel.

In case you haven’t noticed, my novel is done. Well, at least the 50,000 words are done, so I am a NaNoWriMo winner. I still need to edit it, and with some artistic help from Joe Parry, I will add some illustrations and lay it out in InDesign or something to make a nice PDF booklet. I am also going to write up an essay on the experience of NaNoWriMo and my influences for the book. But that’s in the future. I know at least one or two people have been following along throughout the month, so now is the time for you to let me have it… I will be putting some polls up soon for people that have read the novel, but if you want to compliment me or criticize me for anything in the book, comment in this post or send me an email. I really want feedback… seriously. It’s the only way to improve.

Well, here’s the last of it. I added some new material to previous chapters, so the additions are listed here along with the chapter in which they go. The PDF has been updated with all the new material.

Chapter 29: Epilogue

Angela was bobbing up and down in her chair almost uncontrollably as Melissa came down the stairs into the kitchen. Lawrence , of course, sat reading a paper next to her, ignoring her completely as she knocked plates, cups, and other random items on the table to the floor with her movement.

“Angela, settle down and eat your brea kfast!” Melissa knew it wouldn’t happen. The five-year-old looked down in disgust at the strange mixture of foodstuffs on her plate. The toast was now soggy from spilled apple juice, and the remaining bits of egg swam in a pool of juice as well. Angela turned her nose up at the plate primly and crossed her arms across her chest.

She was pouting at the reprimand. Melissa rolled her eyes at her daughter’s silliness and flipped on the small television in the room, pushing down two slices of brea d in the toaster as the picture faded in.

“Can we watch cartoons, mommy?”

“No, we cannot,” Melissa responded fla tly. The question was asked each morning, and the answer always remained the same. But Angela nev er gave up; one day her mother would relent, and she would get to watch cartoons while she ate breakfast… But today is not that day, Melissa thought to herself.

Lawrence had his morning paper and cup of coffee, Melissa had her morning newscast and toast. It was their routine; the sole moment in the day that she would not vary – it was her tether to the world and to her own sanity. No matter what mischief Angela would cause throughout the day, it was all bearable if she’d had her toast and caught up with the world.

She sat down on the chair to Lawrence ’s right, ignoring her daughter’s continued pleas for cartoons. “Angela, I said no! Settle down and be quiet!” Angela resumed her sulking, but at least it got quiet.

She had just missed the weather report, but she was in time for the morning headlines. The newscaster droned on with little emotion through the long list of headlines, and Melissa listened only passively until something caught her attention.

“ Twenty-three-year-old McAllister Park resident Joel Mendocino is dead this morning, the victim of a car accident at the Franklin-Niles intersection late last night. Mendocino, who had recently returned from an overseas trip, had left the Elston Memorial hospital where he was being treated for a near-fatal gunshot wound he received earlier in the day. His father was the vehicle’s driver, and is listed in stable condition at Elston Memorial.”

“ Lawrence , listen to this! This poor kid lived around here!” Lawrence looked over his paper and murmured an acknowledgement, then resumed reading his paper. Melissa focused intently on the television as the news anchor continued.

“In related news, police were able to discover an illegal “fight club” of sorts that is believed to be the source of a rash of unexplained abductions and beatings of homeless individuals that has been plaguing the city of late.

“In a statement released early this morning, Detective Angus Cobb credited Mr. Mendocino’s identification of his attackers as the crucial clue that led the detectives to a supposedly abandoned warehouse in the ind ustrial area of the city.

“Several participants in the “fight club” were apprehended by city police officers, and it is expected that the ringleader will soon be located.”

The news anchor seemed to be enthralled by the story as it continued on.

“But the story doesn’t end there, folks. Mike Turner, a former Copeland Advertising executive, was shot and killed by detectives on the scene when he refused to discontinue his beating of a mentally-challenged homeless man, and instead charged towards police officers.

“It is believed that Mr. Turner was also responsible for the brutal beating and death of his wife’s lover, Brandon Lloyd, a story that we brought to you yesterday on the 6 o’clock newscast.

“Police are confident that the criminal’s behind these recent activities will soon be rounded up, but remind all city residents to report any suspicious activity immediately and exercise common sense when moving around the city at night.”

Melissa was almost sad to realize that the story had come to an end. It was more interesting than the plot to one of the afternoon soap opera’s that Lawrence berated her for watching. Who knew so much could happen one day in their city?

“Makes you wonder sometimes, doesn’t it honey?”

“Hmmm?” Lawrence murmured, looking over his newspaper again.

“Makes you wonder how many people you see and interact with each day will be dead tomorrow. It’s kind of sad, really.”

Lawrence looked back at his paper. “I suppose. But I don’t worry too much about it. After all, what do they have to do with us?”

Melissa stood and ran her dishes under the warm water at the kitchen sink. She wouldn’t think about Joel much after that. Lawrence was right, after all; what did it all have to with her?

Chapter 3

Joel shrugged as she moved as quickly as she could to the opposite side of the train, sat down, and eyed him warily. Whatever.

While he had traveled in Asia , he’d been amazed at the hospitality and friendliness of nearly everyone he met, including the tourists with which he’d had the opportunity to converse. All travelers there seemed to be inexplicably linked; they al shared that undeniable experience that was Asia , and that experience drew them together. That experience, combined with the basic human need to communicate, at least occasionally, with people from similar backgrounds and worldviews, produced a level of camaraderie with some of his new acquaintances that Joel had seldom had with friends from home.

Two, in particular, Sean and Pang, had become very close to him during the three days he’d known them. They had met outside a market while he was staying in Thailand for a week. Sean had been excited at spotting someone with the same color skin as his own, and Pang, who’d spent a year studying at a university in the States, was glad to have a Westerner with which she could converse. Pang and Joel hit it off immediately; they both had an affinity for analytical thought, much to Sean’s dismay, and spent many of their nights chatting about philosophy, art, literature and even mathematics, late into the night.

Sean played the dumb oaf, but behind his goofy exterior was a curious, sharp young man with a razor sharp wit. Together, the three of them had traveled the city high and low, seeking out coffee shops, bars and pubs, and sampling every piece of the Thai nightlife that Pang could think to show them. It was hard for Joel to believe that they’d only been together for three days before they’d split paths. He looked forward to calling them when he got the chance; their ongoing relationship was an opportunity to relive their past experience vicariously through each other, and it was an opportunity that Joel was thankful for.

But it didn’t stop there. The Asian people he’d met, from Sumatra to Thailand , were friendly, eager to please, and very rarely had any desire for profit, monetary or otherwise. On several occasions, while he’d carried his bags to and from a taxi, a random man off the street would run up to assist him, and many times, refused payment. Their payment was his smile and gratitude, and in the wake of their unabashed kindness, Joel felt obligated to at least attempt to repay them by spreading that kindness. It was just too bad no one could believe he was just trying to be helpful.

As the train began to move towards its downtown destination, Joel found an empty seat and sat down, peering out the window at the cars passing on the highway alongside him.

Chapter 8

The path along State Avenue to the tower was relatively clear; it was so late in the morning, a majority of people had already made it to their jobs. He eyed the homeless men on the corner warily. Why the hell didn’t they get jobs? They were always standing there, begging for change, occasionally offering a useless trinket or piece of shoddy journalism up in exchange.

But even worse than the beggars were the ones with the squeegees. They attacked his car every single time he drove to work, splashing his windshield with their vile liquid and doing a piss-poor job of cleaning it off. They always left his car in worse shape than when they started, and they expected to get paid! Bull shit! The mere thought of them soured his entire mood considerably, if it were even possible for him to be more pissed off at this point.

His previous physical activity had left his body reeling, and he was unable to move faster than a brisk walk towards the tower. He was interrupted several times by beggars along the way, huddling against the looming stone buildings for shelter from the wind. He ignored than each time, yelling whatever obscenities surfaced to his mind first, until finally he slapped at the hands of the last one, knocking the small collection of nickels and dimes from the man’s hand. He smiled cruelly as the man cried out and dropped to his knees to reclaim the coins that would serve as his dinner for the day. That would show the bastard.

He continued as fast as he could along State Avenue, ding his best to ignore the legs’ increasing cries of anguish as they suffocated, until the doors of the Tower stood before him, peering down at him with a foreboding gaze. He pushed his way through the revolving doors and ignored the pleasant greetings of the security guard as he headed straight to the elevator.