A senior software engineering having a big interest in information security
I was in charge of managing a dedicated server running Debian 7, the server hosts multiple websites with email service, the server hosts also multiple instances of a critical web application, for a client who is running a business across different regions.
That day was a very important day, as the client was expecting his own client to turn up. The client wanted to make a demo on the application and show how they manage some business processes.
During that event, I got a phone call claiming client users not able to access the web application, I took the request as usual and started checking the filed issue, few seconds later I got another call about other users not able to access their mailbox, it is then I realized that a very nasty thing is happening and I'm in serious trouble.
I quickly figured out that I have made the worst mistake ever!
That day I was performing usual maintenance tasks on the server, freeing some disk space here and there. However, at some moment I deleted critical files that belong to different services like Postgres, Mysql, mail server...etc.
I didn't notice anything until I started receiving reported issues from the clients.
It was catastrophic in all corners
We lost three months of data as backups resided in a single place were I launched the deletion operation, there were no other copies of backups. Many services were surviving with what was left in RAM (I guess) and any respawned process was lethal for the corresponding service.
In the field, the client was badly embarrassed in front of his client, as he was cut off at the beginning of the demo.
Users started processing customer data manually using a pen and paper.
The recovering of the server was a "mission: impossible", we needed to reinstall everything from the scratch, but it was decided that the fastest recovery would be to migrate to another server with latest Debian version.
So I installed all the required services on the new server and restored the most recent backup, The business application was finally live.
I have learned a lot of lessons because of this disaster.
One can learn from other people's mistakes, but the best learning form is when you make the mistake yourself and experience the aftermath, It will make you tougher and wiser.
Create your free account to unlock your custom reading experience.