Important: Unofficial PBE Patch Notes for 12/8/2013 - S3 Championship Ward, Winter Wonder Lulu, Snow Day Singed & Snowstorm Sivir Splashes, Unofficial PBE Patch Notes for 12/7/2013 - Yasuo Login Screen, Snowstorm Sivir & Snow Day Singed Load Screens, Showdown is Back
Post-Incident Messaging: Lag On EUW
Originally Posted by Riot (View RedTracker Source)Hey guys,
I want to take the time to shine a light on our recent live service issues. This is going to be pretty dense but it's what you want and deserve so here it goes...
To start with. We are aware of the poor experience that European players have endured this past week. Since I cant change what's happened, I want to apologize and give some additional information on the reasons and scope of the monstrous lag on our servers hosted from Frankfurt.
As we pointed out before, the issue is actually related to the hardware problem our partner had earlier this week. This was related to an unstable piece of network equipment that failed over to back-up on several occasions and caused some periods of network instability and bad player experiences. We have been working with our partner for the last couple of days on a fix. This resulted in 2 separate maintenances and finally the faulty piece of hardware was replaced on Wednesday/Thursday night GMT. Initially, we thought that we had the problem fixed. However, the next day during peak hours, we continued to get reports of lag from players. We quickly got the team together and realized that we still had an issue.
We contacted our partner and they too started their own investigation. The problem only appeared during the peak times and worse yet affected only some of our servers spread out across the 4 environments (EUW, EUNE, TR, RU). EUW and EUNE were the most heavily affected platforms so we disabled ranked games because of the degraded state of the service. As our combined efforts continued, we examined every piece of the multiple circuits that connect us with to our partner and finally found an issue in one of them. It was a faulty link that had been accidentally enabled during the maintenance on Thursday and was the issue underlying all the issues over the last few days. This circuit has been disabled and has cleaned up the lag issues players had been experiencing.
Moving forward, our next step is to ensure that we can more effectively control the traffic coming into our data centers by plugging directly into the internet backbone. With the new Amsterdam data center (still on track for Q1 2014), were going to introduce RiotDirect a new dynamic routing system managed by Riot network engineers, where we will be responsible for every single aspect of our own traffic in Europe. Although, this will be essentially invisible to the player, the results will help provide a better quality of service for our players. In the meantime, during the upcoming week, were finalizing deals with multiple ISPs and will be getting them directly plugged into our data center. This will allow us to move some of our game servers away from existing connections and onto this new system. This plan should provide smooth holiday gameplay for our European players.
Again, I want to say that it truly troubles us when our players experience poor service. I promise you that we hear and feel your pain. We also commit to keeping you updated as our new systems and data center plans unfold so that you can see how we are trying to constantly improve and evolve the quality of our service for you.