Preventing DoS attacks

[eXDee]

Hey,
We run very many counterstrike servers with a linux game host. The server is 8 cores with 10gb of ram.
And they are being DoS attacked.
We have told the host to implement these iptables rules as found here https://forums.alliedmods.net/showthread.php?t=87794
iptables -A INPUT -p udp --dport 27015 -m length --length 28 -j DROP
iptables -A INPUT -p udp --dport 27016 -m length --length 28 -j DROP

For each of the ports our servers run on but they have done nothing. We've tried vdf mods, metamod script and none work.

We are confident its an intentional DoS attack because it affect servers one at a time, happens suddenly till the server empties, and then moves onto our 2nd server of the same gametype as players migrate there. Not server lag, its definitely intentional, the pattern is there and is designed to empty our servers, by a competing gametype provider (yes, sad i know).

As mentioned it affects 1 server at a time, the cores on the box are only at 50% usage each (checked this myself in ssh) and there is 4gb of ram free.

The lag basically causes the same actions to repeat over and over, the dying animation of players will repeat over and over, movement is incredibly jumpy and simply unplayable.

As for mods we are running eventscripts and sourcemod with several scripts/plugins under these but we have verified that none of these are causing it (or in that case being exploited). We are also confident that it is happening remotely, not from a player in the server.
Could something like this be as basic as UDP Flood?

Does anyone have any clue at all how to deal with these?

[comradek]

Are you sure the attacks are happening on 27015 and 27016? Try running "netstat -tulpan" the next time it happens to see what the active connections are. If it's consistently happening from one IP address, you could go as far as banning the IP in /etc/hosts.deny .

There might be some relevant information in /var/log/secure if it's say, an SSH flood. (DenyHosts is a very good tool for preventing that if that is the case)

[pengy]

Try to get Kigens Anti Cheat. This is usually caused by running 3 commands over and over again on the client, but kigens fixes it. (Don't ask me them.

[over_kill]

Do you have any way of monitoring traffic on your network (such as Wireshark)? The first thing I would do is monitor what is going on when the said "attacks" take place, just to verify that you are indeed being barraged with traffic and whether or not you are experiencing a DoS attack, from one source (probably), or a DDoS attack, multiple sources at once (less likely). If you are under attack, try to update all software, and then consult this site: http://www.linuxsecurity.com/content/view/121960/49/

It has some good information to help you prevent and fight DDoS attacks, but could work for DoS attacks as well.

You might also want to run an nMap on your network and close all unnecessary ports.
Another idea, if you are willing to invest in a firewall, Cisco makes a nice hardware firewall for a good price (imho). Cisco's ASA (Adaptive Security Appliance) 5500 Series Firewalls base model runs for around $400, I believe.

From Cisco:
Protect your Unified Communications infrastructure, call
control, endpoints, and applications from attack. The Cisco
ASA 5500 Series delivers market-leading Unified
Communications security services with the widest portfolio of
application inspection and threat prevention services for
voice/video - protecting against attacks such as protocol
fuzzing, denial of service (DoS), rogue phone callers, and
much more.

I am actually configuring one right now, the amount of security settings is impressive to say the least. It also has a large number of graphs and charts to monitor traffic, attacks and threats. It's pretty neat.

Another thing to keep in mind, whoever is doing this to you probably doesn't like you, and banning the attackers IP might not work. If they are running through a proxy (which I am assuming they will be if it is a DoS, unless they are really dumb), and they really hate you guys, they'll just come through with a new IP and continue where they left off.

If you are being attacked, good luck getting it fixed!

[comradek]

If I were Cisco, I'd DDoS people just to get them to buy my sweet firewalls

but seriously, that thing sounds pretty cool, over_kill

Good call on the proxy. IP banning probably won't be enough but it's still worth a shot.

[pengy]

Wow... all of you are talking about attacking the server directly, not just the game servers.... firewalls cant prevent this kind of attack, because the commands used generates it from the server, which causes everyone on that server to lag, but none of the others. You guys are overthinking this WAY too much. The game server is vulnerable to many attacks. Also since this is a game server hosting company, I am sure that they have ports closed, and its in a datacenter with a DDoS preventing firewall on all servers...

[eXDee]

We are running KAC and various client anti crash scripts. As i mentioned, we are confident it is occuring remotely, not from a client on the server. Therefore KAC is irrelevant anyway. The server can be empty after everyone leaving but myself, but it still happens (until they stop the attack)

There is a definite pattern, the lag occurs when our server begins to fill, until it empties. Then when the players move to our second server, the lag will follow, until everyone leaves the servers because their ping is 300+. Its 100% intentional, no doubt about that. I'll just say somehow we've managed to create enemies with another host which runs the same gametype.

Basicly the attack is on each game server at a time, ie their specific port. Therefore denyhosts is also irrelevant.

As for those iptables rules we added all 10 ports appropriately (not just 27015/16), had the host implement them, and they took no effect.

Im assuming this is some kind of UDP Flood on the specific port? Whatever it is, it obliterates all other traffic to that specific game server, but all others on the same box/ip are fine, until they get attacked.

Attached is a demo of what the result is on the server. This custom map will also be required: http://www.surf-infamous.com/cstrike/maps/ba_jail_summer_r_fix2.bsp.bz2 (couldn't include, too big)

[pengy]

map link doesnt work

[BehaartesEtwas]

is there any kind of messages in the game server logs? then you could try writing some appropriate fail2ban script to ban the IP address. of course this can only work if the game servers are directly attacked.

[eXDee]

pengy Wrote:map link doesnt work

Sorry, you have to copypaste it or it won't work.

Code:

http://www.surf-infamous.com/cstrike/maps/ba_jail_summer_r_fix2.bsp.bz2

Logs don't seem to reveal much but i'll take another look.

[eXDee]

Okay, we just changed hosts to a new one which is much larger, and runs windows servers, with higher security. They dont even allow dll's to be uploaded for mods.

And the problem is happening again, they are DoS'ing the servers until they empty. Now not only does it lag to death, on these new servers the server actually crashes after several minutes of lags, whereas the others didnt.

This is incredibly frustrating since there seems to be nothing we can do about it, and is now becoming a financial issue too. We're talking over 100 slots worth of servers.

And i can now 100% confirm its a remote exploit, i was in the server alone after it emptied, and it was still lagging before it crashed. No other clients. Its happening on windows and linux, and now another provider ive talked to says they are being attacked too. Lastly, when lag occurs server tickrate seems to still be high, cpu/ram usage doesn't increase.

edit: heres the response from our new host, after we requested assistance/possible use of packet monitoring software etc.

Quote:The problem is it's not really an attack on the machine, just the individual game server, so traditional DDOS protection on the network is nearly useless detecting it and stopping it. Typically game DOSes send slightly malformed game data packets which the game server accepts but then later causes it to crash or lag out. It's extremely difficult to detect or block this kind of attack from the network side. Packet dumps of just data going to your server might not help either, as a few malformed packets out of millions would be impossible to locate.

Your best bet might be the mod community and game admin community who may have more relevant advice for your specific game.

How on earth are they doing this?
Lastly, could a mod move this to general know i now it occurs on windows and linux?

[css]

We may have seen the same attack on our server.

There was flood of about 200 pkt/s, which is not much, with zero content. It was easy to block because the attack came from one specific IP. The IP was faked because the IP pointed back to software company doing anonymity software.

This is how you can try to find where the attack is coming from. When the attack is going on, shut down your server at port 27015. Immediately capture network traffic to port 27015. You should see very much network traffic only from the attacker. The block the IP.

There is one problem (which might not be real problem): The attacker can fake his IP in UDP packets. He can make it look like tha packets came from thousands different sources. It is impossible to block by specific IP. The only solution is to block certain length packets, which is what you have in your first post: iptables -A INPUT -p udp --dport 27015 -m length --length 28 -j DROP. The actual length or size of the packet may be different.

PS. In Linux you can use this command to see traffic to port 27015: tcpdump -s 1024 -vvv -XXX -n 'port 27015'. Be ready to push ctrl-c if you try it when your server is running. It will prints lots of data. Other option is to direct the output to file and examine file later.

[eXDee]

Thanks heaps, finally something that might possibly lead to a solution. I'll give it a go on the linux box as soon as i can.

[eXDee]

Gah, it seems only root can tcpdump. Looks like im going to have to catch the server tech online at the exact time -_-

[over_kill]

eXDee Wrote:Gah, it seems only root can tcpdump. Looks like im going to have to catch the server tech online at the exact time -_-

You could request access from the sys admin to the command for a limited amount of time via the "sudo" command, which allows you to run a command at another users privileges (usually root).

Or, you may want to try messing around with this program for CS:S servers. http://adminmarketplace.com/downloads.php?do=file&id=1

It appears you are not nearly alone with this problem.

Is this what your server ends up looking like??
http://www.youtube.com/watch?v=RnboOLAQDkA
That looks rough.

Also, are you running any form of Mani's? There are known exploits.
Are you running a Steam or non-steam server??

-O.K.