The Story of a Mongo Collection

In my last company, we were having more than 70 small APIs and we were using MongoDB heavily. Apart from having ELK stack, we also had database level logging for all the incoming requests to some of the endpoints to track the requests and responses.

A few months ago, we were struck with a disaster, because of something foolish that was missed in the code reviews. One night we did a deployment and after doing the sanity checks on production, considering everything is fine, we all went home thinking that all is good.

After a few hours, I got a call from our DevOps team the website is down. After some digging, we found out that MongoDB is causing the issue; there is some newly created collection having huge amount of data.

I checked our codebase and it turned out that someone enabled the database level request logging to MongoDB for the “autocomplete” API call in search. So imagine 1500 users sitting on the website, searching for flights and each keystroke in the search is getting logged to the database.

The size of the collection had grown to around 600 GB and there was no space left on the server to store more data and the website went down. The quickest hack was to put the capping of 1 GB on the collection until the issue is resolved; we did that and the website was live again.