Recent Outage (no more tears)

[chamber]

I'm not officially a member of the staff anymore, but none of the admins are on atm. We were (very likely) being maliciously attacked for an extended period of time. That attack strained the DB and caused some minor loss of data. About 12h of posts were lost. Hohum made the final call to take down the site because it was affecting other things on the same server, and he made the final call to put it back. There was no official word before now (and now included) for that reason.

To quote hohum:

so like, the site's down for over 24 hours and then it's back up and there's rollbacks and no official word?

We lost about 12 hours of posts. From 6am (when the backup was taken) to 6pm (when the site went offline).

Nobody knows for sure what happened. There's no definitive proof but there is a ton of circumstantial evidence which points to something malicious. Whatever happened caused the database to go into a tail spin and that's why we lost the posts.

We know about the PM issue. I'm not actively working on it. I don't think anyone else is up to work on it. It may or may not get done before I go to sleep but I'm not making any promises.

DO NOT TRY AND DELETE PMS YOURSELF TO FIX IT. IT WILL DELETE BOTH COPIES OF THE PM

The PM issue is hopefully fixed. If you notice any issues contact me immediately.

[Mr. Flay]

As chamber says, the PM thing should be fixed now (you may have to release PMs that are on hold, near the top of that screen).

I believe the thing where forum=20 was throwing a "page not available" error is fixed as well.

Disk space was a problem right after the recovery, but I think Kison has fixed that.

Glitches I am still aware of:

1. Email server appears to need a kick, as it is not sending activation/reactivation emails.
   
   
2. Wiki is throwing a database-connection error on some, but not all, pages. Probably just needs its database connection string updated.
   Fixed.
   
3. ACP/MCP stylesheets are glitching. MCP sometimes is missing its navigation sidebar.
   
   
4. We just hit a 504 error, which I'm mildly concerned about.
   Known; the slow attack is still ongoing, but being mitigated.

[Mr. Flay]

Email server should be back up and running fully once more.

[Mr. Flay]

All errors should now be fixed. If you have an alt/hydra that was not activated, let Kison or myself know.

[Mr. Flay]

A small selection of accounts (those made between the last backup and the time we restored the site) had an error which should now be fixed. If you had an alt/account that was still getting the "No forums" error, it should no longer happen. If it does, let me know.