Best kernel?

[Peter_Pan123]

do someone know why do the vanilla kernel 2.6.31 produce 0,0,0 load ? normally with other kernel i get load by increasing cpu load. with this one i get (same with 2.6.30) no load ..... same with 40% CPU load...???

[BehaartesEtwas]

you will probably know my answer ;-) the cpu load isn't something worse looking at. it doesn't reflect any actual load (how ever you want to define it in some useful way) for game servers...

[Peter_Pan123]

yes i know ... but i want to understand why? normally if the system is under cpu load you have a load between processes but why is the kernel different ?!?! :-(

---

BehaartesEtwas: you asked for Petar_Pan: did you adjust the priorities correctly (else RT patches are contra-productive, I agree...)? > answer: yes i use the vanilla with chrt.

is there something do you wanted to tell me?

[BehaartesEtwas]

(11-05-2009, 06:19 PM)Peter_Pan123 Wrote:  yes i know ... but i want to understand why? normally if the system is under cpu load you have a load between processes but why is the kernel different ?!?! :-(

---

it's not the kernel, it's the game. the methods of measuring cpu load in the linux kernel is probably not built to work under the special conditions... probably the RT patches will make things "worse" as the scheduling gets much more complicated.

still, even if you would know the cpu load exactly, you couldn't gain much from that information. much more important that the quantity of cpu usage would be the time structure of the cpu usage. In fact some of this information can be obtained using the debug options of the kernel... (e.g. you can get a histogram of the wakeup times, basically giving the "jitter" in time of the frames). but you can also look at the fps, gives more or less the same information (but more indirectly)...

Quote:BehaartesEtwas: you asked for Petar_Pan: did you adjust the priorities correctly (else RT patches are contra-productive, I agree...)? > answer: yes i use the vanilla with chrt.

is there something do you wanted to tell me?

I meant with the RT patches, not on a vanilla kernel...

[Peter_Pan123]

what do you think is better? using rt patch and set the chrt with the rt optin for RT (dont know the scheduler .... but its not the default) or using the default scheduler with a vanilla kernel.

[BehaartesEtwas]

in my experience RT patch + RT scheduling (that is that what you set with chrt ^^) is best. but that doesn't mean you could find something better for your machine... try it out.

[Monk]

RT is kind of fictitious on x86, too much latency from interrupts

The linux rt patches are mostly only designed to reduce the latency from the scheduler by adding a bunch of mutexes in specific codepaths, some atomic ops here, some there..

The best kernels are 2.4 series, as nanosleep will busywait in a loop with a udelay().

[Peter_Pan123]

are the kernels from 2.4 save? exploids e.g. or did you mean best kernels for rt patches?

[ulx]

(11-04-2009, 06:21 PM)BehaartesEtwas Wrote:  btw: 2.6.31.4-rt14 looks promising as well, but I have it running only for less then 24h...

I'm running exactly this kernel incl. patch as well since several weeks meanwhile - works like a sharm.

[Peter_Pan123]

(11-10-2009, 06:28 PM)ulx Wrote:  

(11-04-2009, 06:21 PM)BehaartesEtwas Wrote:  btw: 2.6.31.4-rt14 looks promising as well, but I have it running only for less then 24h...

I'm running exactly this kernel incl. patch as well since several weeks meanwhile - works like a sharm.

works like what?

[ulx]

(11-10-2009, 06:44 PM)Peter_Pan123 Wrote:  

(11-10-2009, 06:28 PM)ulx Wrote:  

(11-04-2009, 06:21 PM)BehaartesEtwas Wrote:  btw: 2.6.31.4-rt14 looks promising as well, but I have it running only for less then 24h...

I'm running exactly this kernel incl. patch as well since several weeks meanwhile - works like a sharm.

works like what?

... like a charm, sorry for the typo.

[BehaartesEtwas]

(11-08-2009, 07:45 PM)Monk Wrote:  RT is kind of fictitious on x86, too much latency from interrupts

The linux rt patches are mostly only designed to reduce the latency from the scheduler by adding a bunch of mutexes in specific codepaths, some atomic ops here, some there..

RT patches are designed to reduce latencies for both scheduling and interrupts. especially they reduce the impact of interrupts on scheduling and eliminate some unwanted effects like priority inversion.
it actually works, you can use the latency measurement infrastructure of the kernel debugging to see this. wakeup latencies actually get down to some microseconds instead of several 10th to 100th microseconds on vanilla kernels. this is very crucial for stable fps.
if some other kernel make more stable fps it is more like an accidental effect. it works only under certain conditions and most certainly without *any* other load (e.g. from webservers etc.).
I started using the RT patches for doing audio recordings in a time when computers wern't really much faster then actually necessary for this. RT patches were the only option to get wake-up latencies *reliable* down to achieve a gap-less recording.
remember, realtime is about determinism, not about performance. meaning, RT will guarentee you a certain maximum latency. that is what is important also for game servers. if you create a slight overhead with this it is the trade-off you have to take, but that does not matter at all.
extreme example (just to demonstrate what I mean): lowering the fps from 950 to 900 fps is acceptable if you eliminate drops down to ~100fps.
also remember measuring the fps using on-board methods (i.e. stats command / fps meter) is biased unfortunatly to high fps. you don't see all fps drops. so in fact you might fake more stable fps if you actually increase certain latencies... (because the stats command doesn't get executed during high latencies)

(11-08-2009, 07:45 PM)Monk Wrote:  The best kernels are 2.4 series, as nanosleep will busywait in a loop with a udelay().

you shouldn't recommend such old kernels. they are not maintained anymore, so there might be security issues. also they might not run on new machines, even some older 2.6 kernels make problems e.g. with ICH10 chipsets...

[Peter_Pan123]

"you shouldn't recommend such old kernels. they are not maintained anymore, so there might be security issues. also they might not run on new machines, even some older 2.6 kernels make problems e.g. with ICH10 chipsets..."

Thats why i asked for. i thought they are still working ob fixes and security things on it (2.4).

the problem with an rt patch is that you cant host more then 1 gs each cpu, without drops. i tried it ... or should i use and other chrt scheduler? or the normal one from your tutorial? there are a lot like sched_other sched_xx ...

[BehaartesEtwas]

in my experience it works even better than vanilla kernels with 2 or 3 servers per cpu... like described in my tut...

[Monk]

(11-11-2009, 12:25 AM)BehaartesEtwas Wrote:  

(11-08-2009, 07:45 PM)Monk Wrote:  RT is kind of fictitious on x86, too much latency from interrupts

The linux rt patches are mostly only designed to reduce the latency from the scheduler by adding a bunch of mutexes in specific codepaths, some atomic ops here, some there..

RT patches are designed to reduce latencies for both scheduling and interrupts. especially they reduce the impact of interrupts on scheduling and eliminate some unwanted effects like priority inversion.
it actually works, you can use the latency measurement infrastructure of the kernel debugging to see this. wakeup latencies actually get down to some microseconds instead of several 10th to 100th microseconds on vanilla kernels. this is very crucial for stable fps.
if some other kernel make more stable fps it is more like an accidental effect. it works only under certain conditions and most certainly without *any* other load (e.g. from webservers etc.).
I started using the RT patches for doing audio recordings in a time when computers wern't really much faster then actually necessary for this. RT patches were the only option to get wake-up latencies *reliable* down to achieve a gap-less recording.
remember, realtime is about determinism, not about performance. meaning, RT will guarentee you a certain maximum latency. that is what is important also for game servers. if you create a slight overhead with this it is the trade-off you have to take, but that does not matter at all.
extreme example (just to demonstrate what I mean): lowering the fps from 950 to 900 fps is acceptable if you eliminate drops down to ~100fps.
also remember measuring the fps using on-board methods (i.e. stats command / fps meter) is biased unfortunatly to high fps. you don't see all fps drops. so in fact you might fake more stable fps if you actually increase certain latencies... (because the stats command doesn't get executed during high latencies)

(11-08-2009, 07:45 PM)Monk Wrote:  The best kernels are 2.4 series, as nanosleep will busywait in a loop with a udelay().

you shouldn't recommend such old kernels. they are not maintained anymore, so there might be security issues. also they might not run on new machines, even some older 2.6 kernels make problems e.g. with ICH10 chipsets...

The problem you don't seem to understand is that no matter how low of latency to push to userland applications that run in ring3, the game server binaries are broken. They are unoptimized, buggy, and compiled with fPIC which clobbers the %ebx register, plus they are 32bit only. They use archaic APIs (gettimeofday) plus SSE2 support is/was broken, a great number of other things.

In order to have high, stable FPS the following needs to happen:
- Coarse gettimeofday()
- Minimal jitter from nanosleep/tsleep/usleep() (the last calls the first)
- Minimal scheduler latency (SCHED_RR or SCHED_FIFO)

The first one is not really hard to implement, the 2nd one depends on the hardware, what scheduler youre running (CFS or the default, or even the staircase scheduler)

Youre 'idler' program makes the FPS stay stable because of preemption latency. When a CPU is idle, there's wakeup latency, but if you peg 99% at lower priority (SCHED_OTHER) than (SCHED_RR/FIFO) there will be hardly none at all. When a CPU is idle, a HLT is issued and it waits for the next interrupt to wake up to see if there's work to do, but if it's not idle that's one more step it doesn't have to do.

You cannot have stable FPS on idle machines with low load due to the aforementioned issues, plus you cannot have nanosecond nanosleep latency.

I did not recommend to anyone to use older kernels, I was stating the best one for Linux is 2.4 (just before they removed the busywait stuff)