Running 10+ instances of L4D2 Dedicated Server

[yumyudai]

Hello,

I currently run 10 instances of L4D2 dedicated server on the following machine:
CPU: AMD Phenom II 1055T (6 cores @ 2.8Ghz)
RAM: 16GB DDR3 RAM
HDD1: 500GB 7200RPM SATA II HDD (for OS)
HDD2: 1TB + 2TB 5400RPM SATA II HDD bonded together with LVM (srcds is here)
NIC: Intel CT 1Gbit(Primary) + Realtek RTL8111/8168B bonded together with Balance TLB method
OS: Fedora 14 x86_64

I have done minimum installation of Fedora 14 and disabled iptables/ip6tables and SELinux.
Also, the kernel timer frequency is set at 1000Hz.

Optional CPU features such as AMD Cool'nQuiet and TurboBoost are disabled as well.

I was able to run maximum of 16 instances on this machine however since the updates 2 or 3 months ago, I can only run about 9 to 10 instances of the server without lags.
(Lag as in the game becomes like lerp 0ms even though you are playing at lerp 100ms)
Sometimes, it lags even with 10 servers running.
The server's CPU usage is 300% out of 600% max (so 50% of total capacity) at peak and RAM is 12GB+ cache so more than enough.
So the question is, why does it lag when the server capacity is no where near max?

Here is a CPU usage graph of the server:


The server has Metamod:Source 1.8.5, Sourcemod 1.4.0 and Confogl 2.2.2 installed.
I have the following set in server.cfg:

Code:

sv_pure 2
sv_cheats 0
vs_max_team_switches 9999
sv_consistency 1
sv_pure_kick_clients 1
sv_voiceenable 1
sv_alltalk 0
sv_log_onefile 0
sv_logbans 1
sv_allow_lobby_connect_only 0
sm_cvar fps_max 0

sv_minrate 20000
sv_maxrate 60000
sv_minupdaterate 20
sv_maxupdaterate 101
sv_mincmdrate 20
sv_maxcmdrate 101
sv_client_min_interp_ratio 0
sv_client_max_interp_ratio 2

My guess is one of the following four is causing the issue:
1. OS (Maybe Fedora 14 has bad compatibility with srcds?)
2. RAM(Maybe faulty RAM?)
3. Disk I/O Wait(Maybe disk is too slow and causes the server to lag on other servers' mapchange?)
4. Switching HUB(Maybe the switching hub is bad?)

Any hint you can give me is appreciated! Thanks!

[lhffan]

First you can change this

sv_minrate 20000
sv_maxrate 60000
sv_minupdaterate 20
sv_maxupdaterate 101
sv_mincmdrate 20
sv_maxcmdrate 101


to

sv_minrate 10000
sv_maxrate 30000
sv_minupdaterate 33
sv_maxupdaterate 33
sv_mincmdrate 33
sv_maxcmdrate 33

[yumyudai]

That will disable players from setting cl_interp values to lower values (< 0.05) though...

[realchamp]

Does Fedora have an option to view CPU usage the same way as Ubuntu and Debian((h)top)?
If so that would be alot more usefull. The problem, which I think is the problem, is that it likes to run all the servers on the same core.

[yumyudai]

Yes, there is htop.

True, maybe I can try taskset and allocate CPU core to each srcds instance.

I ran memtest86+ on the server and it looks like something is bad:
<All four RAM modules (4GB DDR3 RAM * 4)>
Ran 3 times. 2 out of 3 passes came with errors. 1 with no error at all.



<Each RAM module (4GB DDR3 RAM)>
Not putting picture of each pass but each RAM module had no errors.
All RAM module tested on same RAM slot with 2 passes each.


I'm guessing there's a faulty RAM slot..

---

I forgot to mention in above post that I have left4dead2/downloads linked to a directory in /dev/shm to reduce disk I/O.

Code:

:/storage1/l4d/left4dead2/left4dead2$ ls -la | grep downloads
lrwxrwxrwx  1 root  root             22 Feb 13 02:46 downloads -> /dev/shm/l4d/downloads/

[realchamp]

Well try assigning them to specific cores. About two servers on each core.

[Dragineez]

I assume you're forking the start-up, not running each from a separate directory?