Edmodo outage – restored from backup

Edmodo hit a little rough patch this morning and we had to restore from last nights backup which was done about 5:00am CST 5-15-2009.

Here is what happened. We have two versions of Edmodo, one that everyone uses and one that is purely for development, but unfortunately they were sharing the same production database. Essentially our development version used the same database as our production version. We made some changes to the code on the development version that was bad and it automatically started deleting all the posts in Edmodo. Can you hear the panic set in? We ended up having to restore from last nights which, which finished restoring about 12:20pm this afternoon.

We are truly sorry to any new users and existing users that either signed up or posted anything between 5:00am CST and 12:20pm on CST 5-15-2009.

This has definitely a learning experience for us and we are taking measures to keep this from happening again.

1. We have already moved our development site off our production database to prevent this from ocuring in the future.
2. We will be looking at ways to be able to recover from a more recent time than a nightly backup.