Many have inquired why the site was running so slowly this afternoon. Well, there was a double homicide shooting incident on the Virginia Tech campus today which resulted in tens of thousands of listeners flooding in to the audio portions of the site within a period of about five minutes. Virginia Tech is a highly technical community so they are more online than most.
The site was mentioned by students being interviewed on CNN, as well as numerous viral posts in the social media circuit (Facebook, Twitter etc), which drove a large amount of traffic to the site in a short period of time.
Luckily, we learned an important lesson a few months ago when CNN mentioned us during the east coast earthquake, so we put in place "throttles" to prevent too much traffic from taking the site down until we could provision new servers. On that day when CNN mentioned us on the air, so much traffic so quickly to RadioReference caused our database servers to crash under the load and we were down for over 45 minutes! Running a platform that can instantly see tens of thousands of people flocking to it out of nowhere can be challenging since it is difficult and costly to keep servers constantly provisioned and prepared for such traffic. But, the good thing is the site stayed up and running, albiet slowly, and we were able to provision new servers to better handle the additional traffic we were seeing.
For those that want the nitty gritty on what happened today - we have a proxy server that load balances requests across our Web servers and also throttles requests so if we hit a traffic spike like we did today, we won't overwhelm what is already running. We currently throttle to 2000 req/sec and we were maxed out for most of the afternoon. After the news hit we brought up 2 additional Web servers and a database replica server to handle the load and things are back to reasonable levels of performance.
With that said - for those of you that were having trouble getting your RR fix this afternoon, our apologies.