Riot Direct Tour: Poland's Vanishing Data Packets

 

Pretend for a minute that you really enjoy playing football (probably not difficult). And you’re pretty good--you’re playing in a small but fiercely competitive league, and hopes are riding on your grade-A shooting. You take the ball past one defender, juke the next with ease, and you’re heading for the goal. You line up the perfect shot, one that will arc gracefully over the goalkeeper’s hopeless expression and nestle soundlessly into the waiting net. But that doesn’t happen; the ball suddenly rolls half a meter to the right, and you’re already committed to the kick. Your foot connects with nothing, ruining your balance. The opposing team laughingly steals the ball from you, and your teammates don’t stop them. They’re too busy shouting at you or, worse, shaking their heads in quiet shame.

This is what it’s like to play League with high packet loss. The reality of the game doesn’t match up with what you see on your screen, and you make decisions based on old information. Plenty of packet loss trouble happens fully on one end of the connection or the other; it’s either us or you. And sometimes there are solutions. Switching to a wired connection from a wireless one, for example, can go a long way in preventing an unintended tower dive.

But packet loss can easily happen because of the way the internet is built, which is a little outside of both of our realms of control. This was the case for hundreds of thousands of players in Poland, who despite maintaining strong connections and a visible path to the game servers, constantly experienced some of the worst packet loss in the region. For these players, with upwards of 5% packet loss (that’s a lot), kicking the ball was a form of roulette. And it sucks for those players; when packet loss is your constant companion, you stop trusting yourself to make plays. More than that, your team stops trusting you. Even the most forgiving allies will stop coming to gank your lane if you can’t consistently react, whether that’s your fault or not.

Riot Direct is a team formed to tackle just these sorts of cases--situations where the whole of the internet stands against a good game of League. We’ve talked before about what that looks like, but in many such cases, the problems are obvious. In North America, the team helped move the game servers from the west coast to Chicago, on the logic that putting servers in the middle of the country would make for a more consistent play experience all over (It did. Duh).

The team did do something similar in Europe. We’ve plugged into ten data centers across the continent, most recently in Sofia, Bulgaria. These improvements have drastically cut down the number of times a connection has to bounce before reaching our game servers:

 

Fact: Talon’s win rate increased by 2% for each fork we removed from the map. Source: Every ADC main’s promo series.

This work has been great for a whole lot of regions in eastern and central Europe, and our Sofia center gives us a great foothold to keep improving connections in south-eastern Europe. Poland should have been improving--but it wasn’t. What these players shared was a common ISP and several common cable pathways to our servers. No matter how fast we got them onto the Riot Direct highway, their connections were already shaky, and so they kept kicking in vain at footballs that never were.

Since catching traffic on the way to our server wasn’t working, we had to catch it at the source. We decided to set up in data center in Warsaw, Poland, where the ISP was based. Once we had a physical presence on their network, we could share internet traffic, and grab League players before they had a chance to ride the broken roads they traveled before.

Whenever we plug into a new data center, we expect to improve League traffic as a whole in the region and see small gains over a wide range. Warsaw was unique in that it was targeted at a poignant but localized problem for a (relatively) small number of players. We wouldn’t know for sure how effective it would be until we turned on the data center and started redirecting traffic. Mid-November of 2016 we did just that, and almost immediately, saw some results:

 

This graph tracks packet loss in Poland a few days before we activated Warsaw and a few days after. Percentage of packets lost is on the Y axis, time on the X.

For reference, 2% packet loss is roughly average for a network--you won’t see disruptions of play at that number. For affected Polish players, playing at peak times (when packet loss is highest), after we turned on Warsaw was like playing in the middle of the night. For the first time, you could play League without mashing buttons in the hopes that one or two commands make it through the noise. Playmakers could pick up Riven or Vayne and at last know what it means to feed without lag. Overall, this was an extremely positive change, not just for the affected players but for everyone who might team up with them.

Such a big change comes with risk; you can’t mess with the foundations of the internet without some collateral damage. This damage came in the form of several hundred players, whose connections previously ran through Warsaw on different ISPs. Before November, these players had reasonable connections, but the instant we turned on our data center, these connections were rerouted and performing worse. Significantly worse, in some cases.

Turns out these players were connecting through Riot Direct but, in a rare turn of events, their previous connection was actually more efficient. In order to fix it, we track each player’s connection manually and stop it from being redirected poorly. It’s a tedious but permanent solution, so once we fix a player’s connection they don’t have to worry anymore about being redirected. As of January, affected connections are back to normal, and we’re still working with various internet service providers in the area to see if we can make things even better.

League is a game of milliseconds, and the more consistent your connection, the better time you’re gonna have. The Riot Direct team is committed to finding ways to make this happen anywhere problems might otherwise be chalked up to “bad internet.” And while we can’t solve every issue this way, there is a certain domino effect. Every improved connection is another ally you can rely on to kick the ball when it counts.


3 months ago