It's a feeling that socks you right in the stomach, and just keeps pounding - you have been hit by a hacker.
Over the last seven years in which I have managed our company's systems, I have seen a growing number of people post to our forum, in the throes of discovering just that.
Recently, more of the intrusions being reported are of cloud-based services. But, over and over, the pattern is the same for system administrators - disbelief, denial, anger, embarrassment and action.
Panic, however, is the last thing you should do when hit by a breach. Here are the five steps I suggest to every sysadmin to fix the fall-out.
If an intruder has gained access to anything, chances are it is via a network. Your number one priority is to stop further harm, by disconnecting from that network.
The method of disconnection depends on the location of your server. For on-site facilities, you can literally unplug. But, if you are one of the many organizations now hosting in the cloud, you want to move it to an isolated network and turn it off.
Why disconnect? While you want to find and fix the vulnerability which let to the intrusion, for now, you simply need to shut the stable door before any more horses bolt. If your server carries mission-critical functions like payroll, things get a little different - that is why you should always separate functions.
You wish the hack had never happened. So grant your own wish by turning back time.
A hacker may have spoiled files, implanted data or inflicted other damage. The quickest route to un-doing his handiwork is to restore a backup. Pick the last known pristine moment and restore your data and software from there.
If you are confident to leave data in place but want to redeploy clean software - say, your web server - then you can spin up another instance of your server stack, configuration in tow, if you have used devops tactics to define your typical environment. Doing it this way beats configuring everything by hand each time an intrusion occurs.
Now you're up and running again, you want to find out who got in, where, how, what information may have been lifted and which of your users may be affected.
This step is somewhere between a post-mortem and detective work, but how you carry out that works depends on the nature of the breach.
By all means, comb through log files to see where something went awry. These can show you logins and other account activity. Examine which files were affected and when, check whether customers were affected through the sending of a phishing attack.
Be aware that, if a system was compromised, its log files may not be trustworthy; they may have been falsified, too.
Not only are you duty-bound to tell stakeholders whose data may have been affected by a breach - doing so may help stop the problem from spiraling out of control.
The best strategy for bearing bad news is transparency. When one service we use for cloud monitoring, Datadog, was recently hit by an attack, I ended up full of respect when staff were open about the issue and gave helpful suggestions on how to respond. Conversely, Dropbox's reluctance to confirm its historic hack has left a bad taste in the mouth.
The moment you know what happened, tell your users, and make clear recommendations on the actions required from them.
You can give users the full, running picture on the breach via a channel like blogs or Twitter, inviting users to subscribe in your initial email.
Like the loser in a broken-down relationship, a hack victim often finds himself wondering: "Could I have done anything different?" The answer is: "Yes." So, when you have sealed the system and covered the bases, take action to ensure the break-in can never happen again.
One of the main ways intruders gain access is via known vulnerabilities in pieces of software. So ensure you apply all patches and updates diligently. Remind internal users about security policies - "don't share your password liberally at the bar."
Consider restructuring your setup to make it harder for crooks to get in. Implement a DMZ setup using a proxy or load-bouncer, usually designed to simply manage traffic flow, meaning only your nominated proxy can talk with your server itself.
Even better, store log files on a remote server so that, even if a hacker crashes the party, they can't falsify the evidence.
It can seem like the end of the world when your company is attacked by a hacker. By remaining cool, calm and collected while taking stock of some of the above tips, you can both stop the attack in its tracks and ensure that you become a much tougher target in the future.
Phil Phillips is the director of DevOps at Experts Exchange, which connects you with people and information to solve problems.