Optimization Questions/How many clients...

[Tialys]

Our clan recently realized that our monthly cost of renting game servers individually was only a few dollars shy of a nice dedicated server, so we went ahead an got one. It's doing fine for the servers we wanted to run, and now we're wondering how far we can push it.

I'm not a complete noob at linux, however I'm not tremendously experienced at optimizing a server.

Stats:
Debian 32bit (Latest stable)
Quad Core Xeon x3220 2.4Ghz Processor
4GB Ram
250GB HDD Space

We're currently running:
3 DoD:S 500FPS 100Tick match servers (3 x 13)
1 DoD:S 300FPS 66Tick Pub (1 x 24)
1 CSS 500FPS 100Tick match server (1 x 11)
1 CSS 300FPS 66Tick Pub server (1 x 16)

We've had 2 of the match servers and the pub full at one point and had no issues (approx. 40-60% on one core).

How many more servers can we run on this box before we start to slow them down?

Also, does Hyperthreading really affect performance that much, and if so is there an easy tutorial for turning it off (I don't want to recompile the kernel unless I have to)?

Finally, what other things can I do to get the most performance out of this box?

[css]

Tialys Wrote:How many more servers can we run on this box before we start to slow them down?

Some say "one server per core". You might disagree based on your current experience.

Try to install Munin and my "Source dedicated server statistics" plugin for it. Then follow how the FPS changes during the day. As long as you get only ~50 FPS fluctuation everything should be okay.

Tialys Wrote:Also, does Hyperthreading really affect performance that much, and if so is there an easy tutorial for turning it off (I don't want to recompile the kernel unless I have to)?

Are you sure hyperthreading is even enabled on your system? See "cat /proc/cpuinfo" and calculate do you see four CPUs or more. If there's only four then ht is not enabled.

Hyperthreading most likely will decrease your server performance. Hyperthreading is meant for single-core CPUs to get better response times for GUI stuff. In a server environment there's too much overhead in the "multi-core emulation" aka. hyperthreading, so it's better be disabled.

Tialys Wrote:Finally, what other things can I do to get the most performance out of this box?

They say custom compiled 1000 tick kernel is the way to go. I've been there done that with zillion different configs, from dynticks to real-time kernels with all combinations. The best this far is not the most hi-tuned server, but sort of basic kernel - although the server does also other things than just the game server. I think some players might see some difference on some LAN server with hi-tuned 1000 tick server, but if you're going to run multiple servers on your box on a "high-latency" internet connections, then it's probably better to go little less-tuned server but steadily.

Another tip which I've seen posted on many forums is to assign one core per one server. You can do it like "taskset -c 1 ./srcds_run -game cstrike ...." and the system will run the game server using only the CPU core #1 (which actually is core #0 ).

In your case I'd probably try something little more complex (because you'll have multiple servers per one core anyway), so do it like "taskset -c 0,1 ./srcds_run -game dods ..." (run the first 3x13 servers like this) and then "taskset -c 2,3 ./srcds_run -game ...." for the other 3 servers. Then the three 500 fps dods servers will have cores 1 and 2 dedicated for them and cores 3 and 4 will run the other servers.

The idea in this is that the cores don't need to do so called "context switches" so much. One process will run much longer on one core, so there's less overhead in the internal processing when the processes are divided between the cores. If you run all the servers without assigning them to certain processors, then the processes will be run on all cores and every time the process is changed to another core, there will be little overhead in copying process memory etc. Most likely this is yet another myth that it'll slow things down, but you never know

[Tialys]

I'm actually willing to believe the core switching causes a slowdown. I've noticed I get slight bits of choke from our servers and it seems to correspond with a change between cores. It by no means affects gameplay, but psycologically it bothers the people I'm running this for. I'll try to assign them to cores like you said and see if that takes care of the minor choke.

Also, I have a stock kernel, but it will allow me to run 1000FPS (apparently) do I still need to recompile (I vaguely remember fiding out my kernel is at 100hz, but I'm not sure).

Thanks!

[css]

Tialys Wrote:I'm actually willing to believe the core switching causes a slowdown. I've noticed I get slight bits of choke from our servers and it seems to correspond with a change between cores.

The processes probably switch cores about 10-100 times a second, so it's most likely not noticeable ever But you're correct about the psychological aspect. I too have seen the "wow this is 100 tick server hitreg is so good" effect on players with ping around 80. Complete BS but that's what players believe.

Tialys Wrote:Also, I have a stock kernel, but it will allow me to run 1000FPS (apparently) do I still need to recompile (I vaguely remember fiding out my kernel is at 100hz, but I'm not sure).

You can check your kernel by doing:

zcat /proc/config.gz |grep -i hz

The output is something like:
# CONFIG_NO_HZ is not set
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100

BUT this is not all. In some instructions (ie. Nitoxys' post somewhere in steampowered.com's forum) you'll also need to configure the "user space programs" (ie. all programs) to run in 1000 Hz environment. This is accomplished by modifying param.h in Linux's kernel files. Otherwise the user space is run always in 100 Hz mode.

Code:

Linux's kernel directory/include/asm-x86/param.h:
#ifdef __KERNEL__
# define HZ             CONFIG_HZ       /* Internal kernel timer frequency */
# define USER_HZ        100             /* .. some user interfaces are in "ticks" */
# define CLOCKS_PER_SEC (USER_HZ)       /* like times() */
#endif

There you see that hard-coded "USER_HZ 100". Change it to 1000 and re-compile if you want the server to be really 1000 Hz. None of the stock kernels have this.

Nitoxys also tells to modify another file "linux/jiffies.h" and add another HZ comparison there, but I don't see it necessary because the HZ doesn't go over the 1536 that's already defined there.


If you want the same effect, but without any downsides, enable the 1000 Hz option in the kernel's menuconfig and the re-compile. Then dream off like thousands others you have 1000 FPS kernel with 1000 FPS game server. That's easier. The effect is the same.

I'd be interested to see your game server FPS statistics. Installing Munin + srcds plugin is easy on Debian. I've got installation instruction made especially for Debian. See http://css.setti.info/code/munin-srcds/ and see if you're interested.