So What Happened to CoursePlus Today (Aug 19)?
Today wasn't a good day to have problems with CoursePlus. There's rarely a good day to have problems with CoursePlus, but today was really not a good time: it's the end of the term, final papers were due at 5pm, and final exams are tomorrow. So CoursePlus first running slowly then becoming completely unavailable was incredibly frustrating to a lot of people.
Yesterday morning, we began to notice that CoursePlus was responding to clicks slowly. As the day progressed, it began to take longer and longer for pages to load in CoursePlus. Things got better overnight, but got worse as the morning progressed. By the time the afternoon rolled around, it was taking over 60 seconds (one minute!) for a single page to load. After that, it didn't take long for CoursePlus to go completely offline.
Information Systems had been monitoring the CoursePlus servers since yesterday and couldn't find a problem in the servers. The code which makes up the CoursePlus application hadn't been changed since June, and things were working just fine until a couple of days ago. As it turns out, the problem was in the database which powers CoursePlus. A small set of data, which is queried and filtered every single time you request a page in CoursePlus, had an index on it which let the database find information in that data set faster. The source of the problem was that this index on this key data set became corrupt. Querying this key data set normally takes 3-7 milliseconds, but once the index on that data became corrupt, that same query took 8-11 seconds. This was the source of the slowdown across all of CoursePlus yesterday and today.
The Information Systems team fixed this problem and CoursePlus began responding perfectly well by 3:30pm today.
Again, we know that this problem couldn't have come at a worse time. We know that this was incredibly frustrating for many people. We sincerely apologize for this, and we continue to work on making sure it doesn't happen again.