SRCDS performance with TF2 on Intel and AMD - Linux

[gdead]

First of all, this forum has been a fantastic resource for me for the last few weeks. I've been working on standing up a TF2 server for the clan I'm in and have spent a lot of time playing with hardware and software tweaks.

My initial hardware was a 2GHz celeron box (2GB RAM, IDE drives) on a 100Mb/s connection running Fedora 8. I tried to take a guess at how many slots the machine could handle based on other threads here. I was hoping for 10-16 TF2 slots. Well, if I was running an HL2DM server, maybe, but from what I can figure out TF2 is about the most processor intensive of the HL2-based multiplayer games.

Without any tweaks, the celeron box fell on it's face after about 6 players. Lag skyrocketed and the game became unplayable. After that experience, I read up on the forums and made the following tweaks:
- 1000Hz kernel
- Kernel compiled to be optimized for celeron
- 66 tick, 500FPS srcds

Even with these tweaks, the server alone was taking up 20% CPU, and every player that joined up was taking up another 10% CPU each. Again, by about 7 or 8 players, the box was totally cooked.

I removed that machine, and replaced it with a 2x dual core 2.0GHz Opteron with 4GB RAM and SATA disks. I know the extra cores won't make any difference, but at least they can handle the OS and its needs. This box has CentOS which has a tickless kernel, and I have it optimized for AMD processors. The results were better, but still not what I expected. Running at 66ticks and 450FPS, the server only takes up 1-2% at idle. Each player that connects adds about 5-6% CPU load on the core the process is pegged on (I'm launching it with taskset -c 1).

A 6v6 game on this machine is playable, tho the ping times creep up for some reason. FPS remains at about 350, which is all I really want anyway. Once we start getting to 14 -16 players, the box gets towards 80-90% CPU load on the one core and things (as expected) fall apart.

I've got a pair of 2.8GHz Opterons being shipped to me, which should hopefully get me up towards a 20 slot server. I guess my questions are the following:

- Are my performance numbers normal? I see a LOT of 24 and even hacked up 32 slot servers out there. I assumed that a 2.0GHz opteron could run at least a 24 slot server without breaking a sweat... but that's clearly not my case.
- If I was on Intel CPU's, would my situation be different? any idea on what a 2.8GHz Xeon would handle as far as slots and performance?
- What in the heck are ppl running to make 32 slot servers happen? I presume it's Intel hardware with 3.4GHz+ clocks, but even then I'm having a hard time seeing it.

Am I missing any simple tweaks? Or is this about normal. Thanks all

[HBS|Ryan]

try not using taskset, and try dropping the fps to 250

and what opterons are they?

[gdead]

I've run without taskset as well and haven't noticed a huge difference. I'll dump the Fps down and see what happens.

They're 2212 opterons (socket f)

[HBS|Ryan]

try compiling for 250fps

[gdead]

Will do. Just to be clear, I should get rid of the "tickless"ness of the config and instead hard code the kernel (and user_hz) to 250?

Thanks for your help Ryan.

[janne]

Yes, you should remove tickless-ness.
Yes, your performance numbers seem normal.
FPS at 250 or 1000 don't make a big difference.

My personal experience is that srcds runs alot better on Intel than AMD somehow.

[gdead]

A few observations. I was reading up on the realtime patches for RHEL/CentOS and one of the things they recommended regardless of if you were running RT or not was to kill irqbalance, esp if you're using taskset to hammer processes to procs. I did that, but don't have enough data to know for sure the difference.

Also, I just dropped in 2.8GHz Opterons. At first, they didn't seem to make a difference. Then I realized that the Opterons by default scale their clock speed. With 4 users on the server, it was running at 1GHz (the lowest available speed). I went into /sys/devices/system/cpu/cpu[whatever I had taskset it to] and hammered the CPU speed to 2.8GHz. Woof. That was cool. 4 users on 2fort, and the box was idling between 9 and 11% usage on the active core. That's more like it. A 24 slot server should be no problem for this machine now.

FYI: I am running at 250Hz and 250FPS and things seem just fine from a playability/hitreg perspective. We ran a scrim on the server and everything seemed great.

[mikkel]

There's something horribly wrong with something somewhere. You definitely have the raw processing power to be able to handle a 24 player server on your 2212s. I've got two 24 player TF2 servers on a debian setup running comfortably on a dual core Xeon 5110 at 1.6GHz with 2GiB of RAM.

[css]

Disable frequency scaling the next time you compile new kernel. You need it as much as you need dynamic ticks. Actually, neither of the features (should) do you any harm, but they won't do any good either - powerwise, that is.

Frequency scaling is obviously useless - unless you're environmentalist Anyway, it doesn't do any harm either, because you can set it to run on maximum speed - and then maybe at lower speed at night when there are fewer players to save the planet.

Dynamic ticks are less seemingly useless. You can use dynamic ticks, but still have 1000 tick kernel. Dynamic tick kernel uses the CONFIG_HZ option the same way as normal kernel. In addition to just limiting to CONFIG_HZ, the dynamic tick kernel can go over the defined value. CONFIG_HZ is merely lower limit for dynamic tick kernel. If you set CONFIG_HZ to 1000, then I doubt that the kernel will ever want to do more interrupts than that. Defining CONFIG_HZ to 100 with dynticks would be environmentalist's choice. Then the server would boost the tickrate to higher optimal value when there are many players on the server. Then when the server action cools down, the server tick rate - and the core - cools down too.

The kernel Hz and the server FPS are not related. The kernel Hz is so called "time slice", which is used to distribute computing time for processes. If the kernel Hz is for example 100 Hz, then one timeslice is 10 ms, which is 1000 ms / 100 Hz = 10 ms. Higher 1000 Hz value gets you to 1 ms time slices. It means - in theory - that the game server process can get CPU time as fast as in 1 ms if it's not currently getting any. Anyway, in my experience it's overkill, and because I've got web and mysql database running on my game server, I have Hz set to 100. If I have 1000 Hz kernel then the web and mysql database are causing very much overhead, which kills the performance. I have very busy mysql database server, so that's probably part of the reason for lower performance. Running 1000 Hz kernel with web and mysql may be OK if it's just normal clan website and forum or something similar. The main point is that kernel Hz does not limit server FPS! I have 100 Hz kernel running ~500 FPS 24 slot server.

So, you can run high performance server but still be environment friendly. Some GSP might want to take advice from this post. You can have the 1000 FPS written on a green leaf on your web site front page. How neat is that!


Here's where you find CPU frequency scaling in "make menuconfig".

Code:

Power management options  ---> CPU Frequency scaling  ---> [ ] CPU Frequency scaling

You can also check /usr/src/linux/.config (for the new kernel) or /proc/config.gz (for the current kernel) to verify the settings.

Code:

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
# CONFIG_CPU_IDLE is not set

Dynamic ticks are here:

Code:

Processor type and features  ---> [ ] Tickless System (Dynamic Ticks)

The same thing in .config or /proc/config.gz

Code:

# CONFIG_NO_HZ is not set

I recommend using "make menuconfig" to change the options because there might be some dependencies when enabling/disabling the features. "Menuconfig" will see those depencies and enable/disable required dependencies. Directly editing .config is probably deprecated anyway