Slow map changes on css-server

[janne]

Hi all!

I have a problem with long delays(~10-20 sec) when changing maps on my css-server where nothing seems to happen.
I can see the cpu core hits 100% for the srcds process.

Usually, but not allways, the second time the map is loaded it goes alot faster, but problably because it's cached somewhere in memory.
I have tried to make a ram-disk for the maps, but I still get the delays when changing maps.

I have 3 cs:source servers on this machine and all have the same problem.
(only plugin i have is zblock 4.1)

I had similar problem on my previous server, but the delay was a bit shorter.

So, Is this normal or can something be done??


My spec:
Debian 4.0r3 with a customized vanilla kernel 2.6.22.1-rt9
(I've tried all kinds of kernel versions, but I find this to be the best when it comes to a have stable 1000fps css-server)

Intel Core 2 Quad Q6600 2.4GHz
Asus P5E-VM HDMI iG35 SATA Audio VGA GB-LAN mATX
Corsair TWIN2X2048-6400C5DHX 2x1024MB DDR2 PC2-6400 800MHz
2x (Software RAID1) Seagate Barracuda 7200.10 250GB SATA-II

[css]

janne Wrote:Usually, but not allways, the second time the map is loaded it goes alot faster, but problably because it's cached somewhere in memory.
I have tried to make a ram-disk for the maps, but I still get the delays when changing maps.

I have 3 cs:source servers on this machine and all have the same problem.

How have you created the ram-disk? If you use tmpfs as I've experimented, the file is put in the cache when it's read first time. After that it'll never leave the cache. So even with "ram-disk" you might experience slow start when the map is first loaded.

I've written instructions how to setup multi-server shared srcds content at http://forums.srcds.com/viewtopic/7335 . You seem like experienced server admin that could make some use of the more advanced stuff (ie. make all srcds content shared + RAM'd).

PM me or post here if you see the tutorial usable. We can then work on the smaller details and maybe get a neat script pieced together.

BTW. Janne seems like a Finnish name and noting that you wrote ram-disk with hyphen, I'd say you're a Finn by 95% certainty

[janne]

I did a small test from your tutorial using tmpfs, but it didn't improve anything. However, I only put the dir ./maps/ on the ram-disk.
Not even the slightest improvement tbh.
I like the idea about it though, especially now when ram is pretty cheap.

btw. I'm a Swede

[css]

janne Wrote:I did a small test from your tutorial using tmpfs, but it didn't improve anything. However, I only put the dir ./maps/ on the ram-disk.
Not even the slightest improvement tbh.
I like the idea about it though, especially now when ram is pretty cheap.

I did the same "only maps on RAM" test (I had only few maps RAM'd), and tbh I didn't notice any boost either. That's probably because the server and Linux do caching automatically too.

Anyways, I think it'd be good to test the concept in full. I can try to create a script, which links the whole srcds directory to tmpfs, except logs, SourceTV demo files and downloads directory. Then everything will be in Linux memory all the time. It'll definitely boost read times from the "disk", but it's whole another thing whether it boosts map loading times. I don't think the disk I/O operations are the most time consuming part when loading a map, but surely it won't hurt if it takes only 10 ms.

I'm also interested about other tweaks to make maps load faster. Others feel free to jump in and tell if you've got ideas how to make maps load faster.

[janne]

Now, I have tested a bit on my other server...
Core2Duo E8400 3ghz, 4gig ram PC6400 running Kubuntu 8.04 kernel 2.6.24-17-rt...
and it have the exact same "problem" with long delays on first map change.
Also here the CPU goes to 100%.

It seems that the cpu-usage is the real problem, and not the read speed from the hard drives.
I think that is why creating a ram drive doesn't help us much.


My setup with my 3 css-servers use the same common files on the server anyway, using symbolic links.
So when Linux once have cached maps into memory for one server, the other servers can use that cached map file too.

[css]

Why is it bad if the long loading time is only for the first time when the map is loaded? It happens once a day or so anyways. If you've got some weird restart system for the game server, then do "cat ./maps/* >/dev/null", which forces Linux to scan the files to memory/cache. If you combine that command with tmpfs you'll effectively just move all the contents of maps to memory/cache forever. After that there's no "first load" - except for the game server itself if it's restarted, but I don't think the game server stores anything in memory from the previous map after a map change, so it shouldn't matter whether it's first load or tenth load for a map.


Other options:

Have you tried tasksetting the servers to use only one core?

# Core 1
taskset -c 0 ./srcds_run -game ...

# Core 2
taskset -c 1 ./srcds_run -game ...

The effect should be that you'll get rid of unnecessary context switches. That should help because the game server won't make any use of the additional cores anyway. The downside of using multiple cores are the context switches. The less there are context switches the less there is overhead of running the processes. That's why setting the game servers to run on specific cores should boost performance, because the cores won't need to change the process they're running, but they can just keep on going without doing expensive context switching.

BTW. How long is your definition of long?

[vigor]

Taskset on most systems causes major FPS issues.

Run your server preferably around nice -10

You may want to examine your setup a little more to see if there is a glaring issue.

Based on your setup you should not be having issues depending on the load and resources available. But then again it's hard to say without knowing a little more.

[janne]

Takset dont help at all. Its just a waist of resources. Linux is doing an excellent work switching cores without noticeable affect. Especially if I, for example, should have more than one game server per core.

Nice however is good, and I am using it.

But I do not have any performance issues.
The only problem is load speeds on first time map changes.

Definition of long is about ~10-20sec

I tried "cat ./maps/* >/dev/null" when i tested the ram disk theory, but it didn't help at all.

[css]

janne Wrote:Definition of long is about ~10-20sec

Seems normal. Are you trying for world record?

[janne]

css Wrote:

janne Wrote:Definition of long is about ~10-20sec

Seems normal. Are you trying for world record?

When the map once is cached it only takes 2-3 seconds to change the map while cpu remain at normal usage...
...so 20 seconds seems like ages, and 100% cpu usage anoying.
Thats what I'm trying to fix.

And Yes; I am trying to have the best css server in the world.