and ever since then we never look at a kitty the same way again. meow!
The beauty of optimism is — when you know of things that can go wrong, and you also know of ways to prevent them from going wrong, yet — you choose to gamble it all off and hope that nothing will go wrong!
Confusing? Edgy? Silly? — yes all of it and more.
Somewhere in Aug 2020, I & my team were working in frenzy on expanding our RedHat Ansible Playbooks repository. (if you are uninitiated into the world of Ansible — start here)
We had built & tested neat playbooks (human-readable code which allows us to define the desired states of our components) to install & configure MySQL, PostgreSQL & MongoDB onto (multi-flavor) Linux VMs. These could also create databases & users.
In order to test these — Two Classic Infrastructure Virtual Server — Public Instances on IBM Cloud were setup— one with CentOS Minimal install and the other with Ubuntu. Opted for default automatic private VLAN network interface. Felt too optimistic and skipped adding SSH keys to the server during creation (vulnerability-1 introduced!).
Next come security groups — An IBM™ Cloud Security Group is a set of IP filter rules that define how to handle incoming (ingress) and outgoing (egress) traffic to both the public and private interfaces of a virtual server instance.
While selecting from the default provided IBM security groups for the public & private interface of the provisioned devices — dabbled around as shown below and ended up with vulnerability-2. Terminology across different cloud providers always intrigues me :) , “all” does not imply “ingress+egress” rather it means all-ingress-only. Quite a wordy affair!
Our requirement comprised:
- Assigning these VMs as target hosts in Ansible Tower Inventory onto which multiple users from within company can execute the DB setup (& other) playbooks and connect to the created databases over internet via any IDE on their local machine. (Internet facing Database instances!)
- Use these machines to test the playbooks remotely during the authoring phase. And opted for the typical super user privileges and open access, wherever we were tempted to! Do not do that! :)
Let’s call it an experimental niche Lab setup— different from usual Dev or Prod environments.
Life was hunky & dory, till one fine morning — a cybersecurity red alert popped in, yikes!
We had left insecure internet facing MongoDB databases like sitting ducks.
The infamous ‘meow’ attack had happened! Though shocking, it was all a little amusing as none of our databases had any relevant or confidential data. Bazinga! (that moment when you find your wallet stolen, but it was your empty wallet).
Meow is a scripted bot attack that scrambles the data & indexes of victim Mongo databases by overriding it with numbers and appending the string ‘meow’. Attackers have an ironic sense of humor! Find out more about meow here.
Awakened by the jolt, it was time to secure the setup for our internet facing Mongo databases. Though, there are multiple options & levels of security, read MongoDB’s official security checklist here; but based on our unique need, cost considerations & smart recovery options — following were the steps to security.
1. Recreated the IBM Cloud Classic Virtual Server, this time added a pair of SSH keys and disabled root login. Wondering if choosing to recreate was a tough call? Well, not really. We were working on Ansible, remember?
So, we already have playbooks that automated all our subsequent configurations and setup. One-click redo. Recreation was preferable to avoid running into malware traces later on.
2. Back to Security Groups of IBM Cloud Classic Virtual Server. removed allow_all ingress on all ports.
Custom security groups with incoming ip-whitelisting though more secure & limits network exposure; was not preferable as our users would be accessing from local machines across globe. It would be unmanageable to whitelist user IPs on demand.
So, how would our in-house end users experimenting — access database instances from their local machine — IDE ? The core requirement!
A hassle free shortcut was SSH Tunnel with port forwarding.
Users could connect via encrypted ssh tunnel to the database hosting Virtual Server and forward traffic from local machine db port to Server side DB port.
For example: traffic from 27017 port of local machine would be forwarded to Server side mongo dB 27017 port, within a secure & encrypted SSH tunnel— thus allow IDEs to connect.
SSH tunnel is not a fool proof solution, it does have potential backdoor threats. But this approach in combination with prior tightly managed SSH keys — did increase our setup’s security.
3. Enable Access Control & Enforce Authentication on the Mongo DB instances from the Ansible Playbook.
Mongo DB does not enable access control/authorization by default (dated Oct 2020, current latest version — MongoDB 4.4).
So, in the mongo DB installation — Ansible playbook, the step to enable security.authorization was added.
This would ensure a user can access only the database resources and actions for which they have been granted privileges. Prior to it —a user could access any database and perform any action, yikes!
4. Configure TLS/SSL Transport layer encryption to Mongo DB — as defined here. However, after enabling this — connections made from Robo 3T (or RoboMongo), an open source MongoDB GUI were successful but MongoDB Compass GUI started behaving erratically.
And I did not want our users to be forced to pick one GUI over other for a post-playbook verification step. So, decided to skip this configuration from final assembly.
- - Other options like — ‘encryption at rest’ was not needed — as the databases were purely for use-and-tear-down. It was not necessary to secure the data at rest, however those with different business needs — may certainly do.
- - I did find Auditing feature interesting and it would have helped us secure our mostly transient Mongo databases better. But, this feature was only available in Enterprise Edition and Atlas (MongoDB-as-a-Service); and we were using free Community Edition.
Alrighty then — this was the real threat received :) from Meow attackers. Never paid any bitcoin, phew!
Lesson learnt: Stay safe even whilst experimenting!
Also published at https://medium.com/wabi-sabi-techexpress/when-the-meow-attack-on-our-mongodb-shook-us-86230d79c509
- Soul staring Cat & sitting cute ducks - Image source: openclipart.org ;
- ssh tunneling - Image from ssh.com;
- rest images - first hand captures & self created explainers from the experience.