Anatomy of a DDoS: what happened on 20 October, and what we changed overnight

At 00:37 UK time on Tuesday 20 October, both game servers were hit by a DDoS attack. Balkan went first and English a minute later: the world froze, a timer started counting, and then "connection timed out".

The short version

• Someone flooded both game servers with junk traffic, so much that real players' traffic could not get through.
• The servers never crashed. Characters, cars, money and items were safe the whole time. Players were knocked off because their game could not hear the server for a few seconds, not because anything broke.
• New protection went in the same night. When someone tried again two hours later, nobody even noticed.

What a DDoS is

Imagine the only road into the city. Normally it carries a steady stream of cars: everyone who is playing. A DDoS is someone sending thousands of empty lorries down that road at once. Nothing is wrong with the city, but nobody real can get in.

Everything you do in game is sent to the server as tiny messages called packets, and the server reads them one by one. Flood it with millions of fake packets and it spends all its time reading junk, while your real packets wait in the queue until your game gives up. "Distributed" means the junk comes from lots of places at once, which makes it much harder to block.

How big it was

On a normal night about 500 packets a second arrive at the servers. At 00:37 that jumped to 475,000 a second, almost a thousand times normal. The first wave stopped at 00:42, and a second, smaller wave hit the English server from 00:45 to 00:54.

Inside the attack

The junk packets were tiny, about 70 bytes each, and that was deliberate. The aim was not to fill the connection but to keep the server too busy checking millions of small packets to read anything real. The part of the server that reads incoming traffic went from almost idle to around 80% busy in a minute. The game itself was never overloaded; it just stopped receiving players' messages.

The two waves hid in different ways. Wave one came from a botnet: ordinary computers, routers and phones infected with malware and remote-controlled, almost certainly without their owners knowing. None of them had ever played on our servers. Wave two used fake addresses: in just 20 seconds it showed more than 135,000 different "senders", really a much smaller number of machines pretending to be lots of people. The attack also switched between servers to test whether the defences were the same everywhere.

What changed overnight

A bouncer on the door. When your game joins, it always does a quick knock first, a normal web connection, before any game traffic. A fake address cannot knock, because the server's answer goes to the real owner of that address, not to the attacker. The server now keeps a guest list of everyone who knocked properly. Game traffic from someone on the list goes straight in; everything else is thrown away at the door. Joining works exactly as before.

Speed limits and a block list. Every sender has a limit far above what any real player needs, the kinds of traffic attackers bounce off other servers are blocked, and the botnet machines from that night are on a permanent block list.

Alarms. Traffic is now recorded every minute and the network security team is alerted the moment it looks wrong: which server, whether the senders look fake and whether players are affected.

Locking the other doors. Everything on the servers that could be reached from the internet was checked, and anything that did not need to be open was shut.

Did it work?

Two hours after the main attack, a single machine started firing about a thousand packets a second at the English server. It had never played and never knocked. The bouncer turned away about 128,000 of its packets in two minutes, the players online saw no lag and no timeouts, and the sender went straight onto the block list.

What we learnt

• Measure everything. Recording traffic every minute turned "the server went down" into a clear timeline, and proved the new protection works.
• Check who is real, do not chase who is fake. Blocking addresses works against infected machines but not against made-up ones. Checking that a player is genuine works against both.
• Our own protection has a limit. A big enough flood can clog the road before it reaches the door, so the answer is filtering further out, on a much bigger network.
• Small tests come before big attacks. So the alarms stay on.

The post set out what came next: trialling a protection service built for FiveM servers, starting with Balkan; moving the filter to the very first point where traffic enters the server; and separate roads for the websites and the game servers.

DDoS attacks are a crime

In the UK, attacking a computer system so it stops working is an offence under the Computer Misuse Act 1990, with a maximum sentence of 10 years in prison. Across the EU, including the countries our Balkan community lives in, national laws make it a criminal offence too, and so do the US and most other countries. Paying for an attack from a so-called stresser or booter site is part of the attack. Police regularly take those sites down and go through their customer lists; under the international operation PowerOFF, people who used them have been warned, arrested and charged.

How you can help

If you are ever knocked off along with everyone else, open a ticket and tell us what time it happened, with your time zone. Times line up with the traffic records and tell us exactly what was going on.

This is a shorter version of a post on the PlexaRP blog. Read the full post on the PlexaRP blog.