If you haven't figured what this section will be about, I will refer you back to the passages containing the asterisks, ampersands and dollar signs. Yes, this is dedicated to the brown, sticky, smelly stuff. You can avoid it, if you don't make the same mistakes as I do, If you can predict the unpredictable. I had different encounters with the brown stuff, some more serious than others, and it will strike you when you expect it least (aaahhh the nice train ride).
Let's get real, accidents do happen, but that is not valid ground to excuse everything. The backup incident should never have happened. I should have had at least a week old backup (even that would have been great). Other brown encounters happened like this:
- When version 4.x came out, I resisted the temptation to go and install it on all my PC base. After all, my current 3.x offered me all the protection and stability I wanted, and McAfee would continue to supply up to date .dat files for 3.x for the coming months. That would give me all the time I needed to play with 4.x to see what new features/headaches this would procure me. So I get the installation files, install the new version on my PC, and to my great pleasure, it kept my previous configuration. That meant that it still remembered to look at \\servername\updates for the current patches. So I dutifully put the bone where my dog expects it, then I click Autoupdate, and there it was, what I always wanted. Time to go to lunch.
Brown stuff happens most of the time just because of stupid things. When I came back from lunch, I realized it was the day that I proceeded with the updates. All of a sudden, no antivirus works at all, all versions 3.x being updated with incompatible 4.x .dat files. I would have thought that the software would have provided some kind of failsafe to prevent this, but apparently not. By the time I realized what I have done and figure out a way to get out of this mess quick before it gets worst, I came up with (you guessed it) a batch file that would overwrite no matter what the current .dat files with version 3.x .dat files. A few magic phone calls later, I have everyone rebooting and having my fix applied. Except that now support calls are now sent to phone-tech-center who tries his best (instead of directly to me, like in the past), but fails to grasp the nature of the problem (I should have thought to mention them right away about my bug and my current work on it, this would probably would have blocked the calls). So a few users had the time to reach phone-tech-support and complain about a non-working anti-virus software. So the techs dutifully proceeded over the phone to install version 4.x on client's PC. That means that when my batch file hits these machines, it will put incompatible version 3.x .dat files on a 4.x installation. Same problem, reverse.
At time like this, it is very valuable to be on the floor. You make a batch file to stop a gap, you enable it and then you get to see it in action right in front of your eyes only by looking at the cubicles. When you see one that stands apart from the rest of the machines, you know something's wrong. I actually resolved the reverse problem by manually desintalling (only 3 machines) version 4.x, and re-installing version 3.x, for homogeneity's sake. But I had learned a valuable lesson.
- Another time I had brown stuff, it happened again during lunch time (I guess it doesn't matter how many people is logged at that time, but who; I wish I had a lab to make experiments on these things instead of the real network). So I didn't realize *again* that it was upgrade day today, even more so because I wasn't using it at the time. But I was still making tests on my machine with my test server (which happened to be also the real server for antivirus distribution, since this all started out as a side-project and evoluted to what it had became. So I accidentally copy test-purpose only configuration file with auto-upgrade enabled (I was investigating the random freeze bug) in the distribution directory. I realized my mistake a mere 10 minutes after, as I started seeing unexpected machines connecting to the server. ooopps! Seems like some people actually logs on at 12:20-12:30 timeframe. Fix the file back, then take a look that's got the leak, call them back to make sure everything's running smooth. The first three calls confirm current installation of the software, but no problems arose. All right, I said to myself, this might turn out for good after all, free guinea pigs from the wild. Aaahh, the sad mistake that someone makes when he pats himself in the back before the completion of his task. Caller number four, which happened to be the last leaked guinea pig, turned out to be a real pig. His computer froze all right. I could still make it by killing the AVSHIELD.EXE task, which would release the CPU to perform the installation procedure. But to do so required patience that my interlocutor didn't seem to have:"OK, put your mouse over the small shield icon, and wait for the label to appear. This may take as long as 30 seconds. Then, right-click on it, then in another 30 seconds or so, a menu will pop-up, then click on Disable, then wait..." You get the picture. He couldn't endure that situation because he had files opened that he didn't save before going to lunch (that was HIS mistake, I have enough with mines already). So of course he's all hurried over the screen, going in a clicking frenzy when the last thing you want if you want to recover from this is to avoid a buffer overflow. He's brought the machine to a point where we couldn't even call the task list with Alt-Ctrl-Del. There's no choice left, power cycle.
The guy was pissed off at me because he lost some of his morning work. He considered this as an intolerable situation and threatened to get this further. After getting a reality check from one of my contacts on site, it turns out to be that this guy was of the brown-nosed type. The type who accepts gladfully a "promotion" to entry-level management, removing him from union, but with no pay rise or new responsibilities, actually doing the exact same work as before, but with unpaid overtime. The guy probably wished he could get a promotion by nailing me. I guess he failed, because I never heard of that incident afterwards. (By the way, even if I may sound harsh, I did not deal with this guy in the manner of the BOFH. But I have to admit it was tempting :-)BOFH:Bastard Operator From Hell, http://members.iinet.net.au/~bofh/ )
- Some places don't take security seriously, even if they think they do, and that is serious brown stuff. I have seen computer security personnel investigating on computer equipment theft who couldn't tell the difference between RAM, a CPU or a hard disk. I have seen personnel from same department that were specialized in viruses. They ran up a monthly report on the number of virus incidents. Any single file found with a virus counted as an incident. Known incidents were those reported by company employees to the computer security department (this was mandatory). It averaged around 50-150 incidents per months for the whole company (about 30 000 people). I actually never counted how many virus detections I had in my logs on a monthly basis, but I know it was easily in the 50-100 range on a regular basis (for about 250 users). I didn't report these cases to computer security dept. for two reasons: 1) I knew the files were cleaned even before I knew about it, and I was pretty quick at knowing about it; 2) it was obvious that my input alone would have doubled their figures, and they already seemed so busy at investigating deeply each infections that they would probably be overwhelmed by the numbers, added to the fact that there was not much more that they could do that I hadn't done before. The fact that my numbers compete with theirs even if I only have a small subset of the user base doesn't mean my network was attacked more often than others. This was due to poor knowledge about technology amongst the computer security people, and their lack of leadership in regards to virus protection.
Virus protection software was installed on every machine in the company, be it running DOS, Win 3.1, Win98, Win NT4, or OS/2 or other applicable. But it was the responsibility of each system administrator/desktop support departments to take care of keeping them updated, usually manually on each machine. This is inefficient and may leave some "forgotten machine in the corner" to quickly become outdated. At one point, it became company policy to handle the responsibility of updating antivirus software to the users themselves. Either they did it themselves, of if they weren't confident enough in doing it they could call to open a service ticket to get the machine updated by a tech support. Of course, tech support can't take on himself to update all machines in the office, because the ticket specifies only one. And he has to work on the other tickets too. So the few users who actually think about updating the software have to specify how many machines (usually only subsets of departments). Besides, default settings are applied, so even if the newest .dat files are available on the company intranet web site, the software don't actually go and pick it up itself, even if it is capable of that. In this kind of environment (entirely true), you get quickly outdated (useless) antivirus software, poorly configured (in many versions, default configurations specify to continue scanning on virus detection, without actually cleaning or isolating the file), and absolutely no way to have a clear picture of what's going on (poor logs on users machines, viruses that stay undiscovered for months, personnel who don't bother to report to them (I wasn't alone to do this), except that I was probably the one with the clearer picture). Had they taken the time to take care of this aspect by specifying configuration standards on the desktop configurations (implemented in part of a large company-wide PC upgrade), they would have both a clear idea of what's going on AND quicker investigations, which would leave them with time to apply to more useful tasks.
I actually reported to the computer security department the Worm.Explore.Zip incident, to fill them in on what to expect if they have to deal with it in other departments. I also gave them a glimpse of the kind of setup I had, in order to explain how I got to finding who was infected. That was a bit of risky, because they could figure out that I didn't report previous virus infections (tons of them actually, plenty for them to "investigate"). You can get slapped on your hands in these companies for actually doing too much a good job. They actually seemed impressed by it, and thanked me for the info about the virus, but no one actually got back to me to see how I set up my Web. I had showed my Web also to my new boss, and to her boss, and the boss that took it's place, along with some peers network administrators. They were all impressed and liked it, but nobody asked me how I did it or if it could be implemented on a larger scale (or any scale!). I guess that with a system like this, you can't charge as many man-months of work to the customer as with manual installs. Someone from security actually phoned me a while later, because they had my phone number from my Worm.Explore.Zip reporting. Since I actually knew the guy who had the .dat files on an intranet web server (the guy started this out of his own initiative), they wondered if I actually knew the guy who was taking care of *their* official way of distribution via mainframe TN3270 emulators. Apparently, the service wasn't running anymore for a couple of months and they wanted it back, since it was the only mean that could reach everybody in the company (some legacy machines were running on DOS/Banyan Vines, so they could download via TN3270 the updated DOS .dat files). Unfortunately for them, I didn't have a clue, since this was concerning arrangements that they made with an old mainframe employee who probably left the company before I was even hired there. Tough luck.
This is serious brown stuff because the people who hired these people think these guys are gurus. They rely on them to protect their bat and their balls, namely their data. I remember a time when I was on coop term in a federal agency, we had to deal with the then famed Stone virus. We scanned all our diskettes that could have been potentially infected, finding the sole copy of it on a manufacturer's boot diskette. Then we reported it to the RCMP who sent a "virus expert" on site. As soon as he gets in, I try to explain him what we had done so far, why we know this is the Stone virus, and what the damages were (relatively small, the best part of a day watching the hardware tech trying to install a bigger hard-disk that wouldn't boot, until we saw the message "Your PC is Stoned!"). But no, mister Big Agent had to make a show. So he takes a hex editor and proceeds to read the boot sector of the suspicious diskette. "Ah-Ah!, here it is!" he says joyfully while showing us the word STONE clearly stated in the middle of hexadecimal numbers. "I have found the virus!", he says, acting like he didn't take into consideration any word of what I had told him so far. Had he asked me before he checked the diskette, I could have told him that this virus is a boot-sector virus, that this is where it leaves his signature (read that in a magazine), and that this is how the virus scanner detected and identified the virus. I didn't see the need to actually go see by ourselves if it was really Stone. If these guys perform that kind of voodoo to impress their bosses, then I am wondering about the relative safety of North-American online businesses and public networks.
These guys are only good cops that know a bit on computers. (Make it clear that I am not talking here about real computer security experts finds and plugs holes in software as a hobby or people that goes up to even put cryptography in their spoken language ;-)) They actually caught the guy stealing computer parts, but only because the robber was astonishingly stupid. The robber stole 6 times in my cubicle (I had to work in a cubicle with all the equipment I had to deal with; at least the servers where in a closed ventilated room). They put a camera in the floor after the third time, and he saw them do it (he was doing night system monitoring, and was close from my cube). The fourth time, the camera wasn't running, the fifth time the lights were shut. Then he stole the PC next to his cubicle. The sixth time, he was on tape. When they first questioned him about that PC, he pretended he sat at his desk all night and that he heard nothing suspicious! In the words of Forrest Gump, "Stupid is as stupid does". He actually opened the case, took out all the parts and the motherboard out of the case, put it in his bag and to the car, back to the office, then *get this*, fold the metal case so it would fit in his bag then he went for another "cigarette break" to his car, and came back to the office to finish the night. He said that he did all this so we wouldn't notice that there would be an empty machine. Instead I noticed an unplugged monitor, keyboard mouse and network cable,all assembled as if there was actually a desktop under that monitor! I told you he was not Brainiac, and I insist that all this is 100% true.
9. Real-life crisis case study
11. The sad truth
Table of contents