srcds DOS attack?

[jheiv]

I started logging the packet queue to my server after the 300th segfault (with a one minute cronjob logging netstat -tulpan):

All seems well...
Mon Oct 5 18:42:01 UTC 2009:
udp 1572 0 0.0.0.0:27015 4826/srcds_i686
Mon Oct 5 18:43:01 UTC 2009
udp 1880 0 0.0.0.0:27015 4826/srcds_i686

...until a minute later...
Mon Oct 5 18:44:01 UTC 2009
udp 106372 0 0.0.0.0:27015 4826/srcds_i686
Mon Oct 5 18:45:01 UTC 2009
udp 111548 0 0.0.0.0:27015 4826/srcds_i686
Mon Oct 5 18:46:01 UTC 2009
udp 111340 0 0.0.0.0:27015 4826/srcds_i686

Minutes later the server crashes.

The Recv-Q column jumped from 1880 (prior to that it sits between 2000 and 0) to more than 100k!

Does this seem right? I guess what I'm saying is, could this just be a normal side effect of a crashing server or could this be the cause of a crashing server?

Any ideas of how to look into this further?

Thanks for your time.

More of the report without timestamps:
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111572 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111012 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111464 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 109244 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111016 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110464 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110804 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110984 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111536 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111568 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111504 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110952 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 110740 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111536 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111504 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111016 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 616 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 988 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 648 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 308 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1296 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1880 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1232 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 616 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 308 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 340 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 0 3144 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1572 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1880 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 106372 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111548 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 111340 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 680 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1296 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1264 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1540 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 1668 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
udp 3824 0 0.0.0.0:27015 0.0.0.0:* 4826/srcds_i686
* SERVER CRASHED *

[Arjen]

Don't know ...but maybe this will help: http://www.sourceop.com/modules.php?name=Downloads

[mumbles]

not sure if this is an attack, but I can tell you that a UDP packet storm DOES shut down source networking. I cannot play source from school because of all the UDP noise. I wiresharked to find this out and that is the only valid explanation. Granted, on campus there are literally thousands of computers online at the same time using the same switches, UDP is dangerous because it is not a direct connection, UDP is especially dangerous as it does not guarantee delivery or time of delivery. The sender just kinda throws it out there and assumes that nobody will read it except the intended receiver. Now that my UDP==bad rant is over, if you flood UDP to a server's address, it will try to process each one and throw out the bad ones. This all takes time, which WILL cause DoS type symptoms. This could also crash the server as it will not receive any valid updates from any client for long enough to say everyone timed out. This also could definitely cause buffer overruns and many other issues depending on how the srcds handles this type of issue. So the short story is... these UDP packets are from an invalid address on the port that the server listens to. They are probably custom crafted to take up your server's time. I say this because they are targeted at the port your server listens to, but they are too large to be a client connection request.

[jheiv]

This was definitely a DoS attack. See (a tiny fraction of) the tcpdump capture:
01:29:54.215279 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18
01:29:54.215281 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 0
01:29:54.229257 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18
01:29:54.233254 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 0
01:29:54.234253 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18
01:29:54.234258 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 0
01:29:54.234259 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18
01:29:54.234261 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 0
01:29:54.234263 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18
01:29:54.234265 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 0
01:29:54.234267 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18
01:29:54.234269 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 0
01:29:54.253241 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18

This was blocked with a firewall rule, modified from one I found on this site:
iptables -A INPUT -p udp --dport 27015 -m length --length 0:28 -j DROP

So far it has stopped them, I've blocked the apparent source IP too but I guess its possible they're spoofing that since it's only UDP.

Anyone else seen this? Is my rule enough?

[css]

(10-06-2009, 12:07 PM)jheiv Wrote:  This was definitely a DoS attack. See (a tiny fraction of) the tcpdump capture:
01:29:54.215279 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18
01:29:54.215281 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 0
01:29:54.229257 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18
01:29:54.233254 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 0
01:29:54.234253 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18
01:29:54.234258 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 0
01:29:54.234259 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18
01:29:54.234261 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 0
01:29:54.234263 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18
01:29:54.234265 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 0
01:29:54.234267 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18
01:29:54.234269 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 0
01:29:54.253241 IP 96.19.63.51.64928 > 66.135.40.174.27015: UDP, length 18

This was blocked with a firewall rule, modified from one I found on this site:
iptables -A INPUT -p udp --dport 27015 -m length --length 0:28 -j DROP

So far it has stopped them, I've blocked the apparent source IP too but I guess its possible they're spoofing that since it's only UDP.

Anyone else seen this? Is my rule enough?

I've seen this. It looks remarkably the same I saw few months ago. I've now added your rule on my server too.

One suggestion to the iptables rule: make it log the attacker. It's along these lines:

Code:

iptables -N logattacker
iptables -A logattacker -j LOG
iptables -A logattacker -j DROP
iptables -I INPUT -p udp --dport 27015 -m length --length 0:28 -j logattacker

The second line with "-j LOG" could probably use some sort of logic to match only the first (few) packets so it doesn't fill up log files. Anyway, I have this on my server now.

If somebody else uses this, you can check whether anyone has tried something nasty by looking at the output of the following command:

Code:

iptables -L -v -n
Chain INPUT (policy ACCEPT 5484M packets, 836G bytes)
pkts bytes target     prot opt in     out     source               destination
    0     0 logattacker  udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:27015 length 0:28

There's 0 in the first column "pkts", so the rule hasn't matched any packet yet. When it does, the number is probably around 10000. Then it's possible to look at /var/log/syslog or /var/log/messages and find the log entries about the flood.

[jheiv]

That (logging) is a great idea.

Here's my iptable rules for just that:

Code:

-N logattacker
-A logattacker -j LOG --log-prefix "SRCDS: " --log-ip-options
-A logattacker -j DROP
-A INPUT -p udp --dport 27015 -m length --length 0:28 -m limit --limit 2/sec -j logattacker

The 2/second limit is arbitrary, really but it helps logging the >2k/second entries I was seeing.

As you can see, it picks it up on iptables -L -v -n just like you said it would:

Code:

Chain INPUT (policy ACCEPT 1737K packets, 88M bytes)
pkts bytes target     prot opt in     out     source               destination
   20   560 logattacker  udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:27015 length 0:28 limit: avg 2/sec burst 5

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 1737K packets, 88M bytes)
pkts bytes target     prot opt in     out     source               destination

Chain logattacker (1 references)
pkts bytes target     prot opt in     out     source               destination
   20   560 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           LOG flags 4 level 4 prefix `SRCDS: '
   20   560 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Here are samples from /var/log/messages:

Code:

Oct  6 04:59:48 tophat kernel: SRCDS: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=192.168.1.101 DST=192.168.1.101 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=14794 PROTO=UDP SPT=13953 DPT=27015 LEN=8
Oct  6 04:59:48 tophat kernel: SRCDS: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=192.168.1.101 DST=192.168.1.101 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=10548 PROTO=UDP SPT=45349 DPT=27015 LEN=8
Oct  6 04:59:49 tophat kernel: SRCDS: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=192.168.1.101 DST=192.168.1.101 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=16390 PROTO=UDP SPT=12023 DPT=27015 LEN=8
Oct  6 04:59:49 tophat kernel: SRCDS: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=192.168.1.101 DST=192.168.1.101 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=45433 PROTO=UDP SPT=43407 DPT=27015 LEN=8
Oct  6 04:59:50 tophat kernel: SRCDS: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=192.168.1.101 DST=192.168.1.101 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=7368 PROTO=UDP SPT=10498 DPT=27015 LEN=8
Oct  6 04:59:50 tophat kernel: SRCDS: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=192.168.1.101 DST=192.168.1.101 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=31757 PROTO=UDP SPT=41786 DPT=27015 LEN=8
Oct  6 04:59:51 tophat kernel: SRCDS: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=192.168.1.101 DST=192.168.1.101 LEN=28 TOS=0x00 PREC=0x00 TTL=64 ID=22003 PROTO=UDP SPT=8641 DPT=27015 LEN=8

(This was me testing on my local box - hence the 192.168 address)

And you can always cut out the source address for later processing:

Code:

cat /var/log/messages | grep SRCDS | awk '{ print $10 }'

Thanks css!

[BehaartesEtwas]

remember that source addresses of UDP are unreliable. any half-decent attacker would spoof the source address to avoid detection. if someone really wants do make a DoS attack he does not *want* any reply of the target server, so he can simply specify a wrong source address in the UDP packet. there is no protection against that.

on the other hand, if the source address is always the same it might indicate that this is not actually an attack but some mistake (e.g. a broken client).

[css]

Great. I've also updated my firewall rules to the limited logging style.

I also have DoS Attack Fixer [ http://www.sourceop.com/modules.php?name=Downloads&d_op=viewdownload&cid=9 ] plugin installed on my server. It blocks something (not exactly sure what) and there's command "daf_status" (and "daf_log"), which shows the attacker IP.

---

(10-06-2009, 07:29 PM)BehaartesEtwas Wrote:  remember that source addresses of UDP are unreliable. any half-decent attacker would spoof the source address to avoid detection. if someone really wants do make a DoS attack he does not *want* any reply of the target server, so he can simply specify a wrong source address in the UDP packet. there is no protection against that.

on the other hand, if the source address is always the same it might indicate that this is not actually an attack but some mistake (e.g. a broken client).

I think it requires quite skilled attacker to go with the spoofed UDP packets. Obviously it's easy to spoof UDP in theory, but unless there is ready-made software for "fake-my-IP-and-do-the-flood" then it's quite difficult to attack with faked IP address. Also I've noticed that some ISPs don't route spoofed UDP packets forward in their network.

Luckily even if the IP address is spoofed, jheiv's improved iptables rule will stop the attack (and log the spoofed IP address).

I bet if I ever see this kind of flood again it's definitely not from a broken client. How do you even break a game?

[jheiv]

(10-06-2009, 07:38 PM)css Wrote:  Great. I've also updated my firewall rules to the limited logging style.

I also have DoS Attack Fixer [ http://www.sourceop.com/modules.php?name=Downloads&d_op=viewdownload&cid=9 ] plugin installed on my server. It blocks something (not exactly sure what) and there's command "daf_status" (and "daf_log"), which shows the attacker IP.

---

(10-06-2009, 07:29 PM)BehaartesEtwas Wrote:  remember that source addresses of UDP are unreliable. any half-decent attacker would spoof the source address to avoid detection. if someone really wants do make a DoS attack he does not *want* any reply of the target server, so he can simply specify a wrong source address in the UDP packet. there is no protection against that.

on the other hand, if the source address is always the same it might indicate that this is not actually an attack but some mistake (e.g. a broken client).

I think it requires quite skilled attacker to go with the spoofed UDP packets. Obviously it's easy to spoof UDP in theory, but unless there is ready-made software for "fake-my-IP-and-do-the-flood" then it's quite difficult to attack with faked IP address. Also I've noticed that some ISPs don't route spoofed UDP packets forward in their network.

Luckily even if the IP address is spoofed, jheiv's improved iptables rule will stop the attack (and log the spoofed IP address).

I bet if I ever see this kind of flood again it's definitely not from a broken client. How do you even break a game?

There are a few linux utilities that are pretty powerful if you have root access -- I used one to test the firewall rule -- if you're interested in them for testing pm me, but I'd rather not make it easier for anyone malicious by posting. Linux would be necessary for IP spoofing since Windows prevents raw sockets, but if the attacker didn't care about hiding their IP, they could probably just google "windows packet generator" or something.

The funny this is that we were able to look through our server logs for the IP and we found the attacker logged in during the attack. Now, unless it was an admin on our server, no one else should be able to get at the IP addresses of other clients on the server AFAIK. Also, the players chat of "Hey guys, why is the server so laggy today?" gave it away to the server owner. Apparently he changed his IP to a slightly different one in the same block an rejoined. I forwarded all the logs to the abuse email, in case they're bored.

This firewall rule is a good way to drop all malformed packets, but I think you would need a game plugin (or I guess a shell script could work) to step in and be able to drop packets (or for a shell script, dynamically write iptable rules) for unconnected clients. The scenario I envision would be the same kind of flood except with a randomized souce IP and a larger packet size (one that can't be blocked by a simple rule without dropping legitimate game packets).

[css]

(10-07-2009, 03:15 AM)jheiv Wrote:  The scenario I envision would be the same kind of flood except with a randomized souce IP and a larger packet size (one that can't be blocked by a simple rule without dropping legitimate game packets).

I don't think the larger packet floods are possible. The server apparently does react somehow to the small packets, but I don't think that it can be flooded with any random content. It wouldn't make sense that there is such a flaw in the server. I can envision that there is bug that makes the server freeze when it's being flooded with these 0 / 18 byte sized packets, which are probably related to querying server information, but doing the same flood with larger packets seems too easy.

[lhffan]

Code:

-N logattacker
-A logattacker -j LOG --log-prefix "SRCDS: " --log-ip-options
-A logattacker -j DROP
-A INPUT -p udp --dport 27015 -m length --length 0:28 -m limit --limit 2/sec -j logattacker

This is very intressting. How do i alter the abowe so it also blocks tcp traffic on 27015 BUT allowes rcon acess from diff ip´s

And also the exct the same for 27035. Can i have one rule for both my servers. And how would that look like.[/align]

[BehaartesEtwas]

I think even standard netcat (a linux tool) allows source address spoofing, if not, a modified version will be available very easily. It's a little naive to think this is some kind of a show stopper... And btw: iptables won't be able to detect the flood if the source address is different for every packet. So I still believe it is easily possible just by writing a small shell script on linux to flood a server with UPD packets.
If this is harmful I can't tell. It will get harmful if the rate is high enough, but then you can flood basically any port on the server to achieve the same effect (i.e. saturation of bandwidth).

[jheiv]

Yeah, I agree -- this isn't going to stop someone very determined and moderately sophisticated, but it has stopped at least one person (who has tried again since).

I also agree that with a sufficiently large outgoing pipe, they could consume all of our bandwidth (to a colocated server), but we haven't seen that yet. Also, given the size of the packets they weren't going after taking up our bandwidth (I was able to ssh in with no noticeable lag) but rather exploit a vulnerability in the server mishandling (or trying to handle) small malformed packets.

In the latter case, the IP rule will block those packets -- in the case where someone is a) sophisticated enough and willing and b) has a large enough outgoing pipe to saturate our incoming pipe then we have another problem to look at -- but you have to admit, there is some merit to having this simple rule block packets that would otherwise crash your server.

[css]

(10-07-2009, 08:28 PM)BehaartesEtwas Wrote:  And btw: iptables won't be able to detect the flood if the source address is different for every packet.

The rule matches packets to port 27015 regardless of the source address. Any packet directed to port 27015 of size 0-18 (which the game itself doesn't use) are dropped. This has stopped the DoS style flood crashes this far.

Doing normal DoS attack with bandwidth saturation is completely different. There's no protection for that.

[jheiv]

Quick update, I screwed up the IP Tables I posted -- if you read that carefully, I'm actually limiting the jump from input to "logattacker". I really wanted to limit the logging.

Here are my updated rules:

Code:

-A INPUT -p udp -m udp --dport 27015 -m length --length 0:32 -j logattacker
-A logattacker -j LOG --log-prefix "SRCDS:ATTACK: " --log-ip-options -m limit --limit 2/sec
-A logattacker -j DROP

With the previous rules, you'll get a log of the attack, but it will do nothing to prevent it from crashing your server.