Left 4 Dead 2- Server/Client performance "Jerky"

[FurySupreme]

Let me get the obvious stuff out of the road first.

Basic Info:
OS: Windows Server 2003, Standard Edition (32 bit) running on a Xen Center 5.5 Virtualization
Processor: Intel Quad-Core Q8600 2.66 GHz (High Precision Event Timer enabled)
RAM: 2 GB (But can easily be set upwards to 3 if required.)
Game: Left 4 Dead 2

Startup Batch Script:
@echo off
C:\HLDS\left4dead2\srcds.exe -console -game left4dead2 -ip xxx.xxx.xxx.xxx -port 27015 -clientport 20005 -nohltv -pingboost 2 -tickrate 66 +exec server.cfg +map c5m1_waterfront +hostip xxx.xxx.xxx.xxx

Server Config File:
hostname Fury's House of Fun
rcon_password "xxxxxxxx"
sv_password "xxxxxxxx" // only needed to make your server private.
sv_contact "your@email.com"

hostport 27015
sv_lan 0 // enable LAN = 1; disable LAN = 0
sv_region 255
sv_allow_lobby_connect_only 0 //enable/disable Lobby connections 1/0

mp_disable_autokick 1 //disables autokick=1
sv_allow_wait_command 0
sv_alltalk 1
sv_alternateticks 0
sv_clearhinthistory 0
sv_consistency 1
sv_pausable 0
sv_voiceenable 1
sv_cheats 0
sv_forcepreload 1
sv_pure_kick_clients 0

sv_log_onefile 0
sv_logbans 1
sv_logecho 0
sv_logfile 1
sv_logflush 0
sv_logsdir logs

The issue:

So far when we've been hosting Left 4 Dead 2 servers (haven't tried with one yet) we've been experiencing "Zombie Stutters" where you'll have about 10 frames of jerky movement, a moment of smooth movement, and then jerky movement again. (Ditto for special infected. We had a tank having a fit once...)

Windows Media Player's running in the background as a frame booster and it has High Priority set in Task Manager. Despite this, the problem still exists. Changing maxupdaterate and cmdrate marginally help, though the initial problem still exists. Server FPS tops out at 29.1 fps with Media Player running, and 28 at idle.

At first I thought it was the Virtualization that was doing it but now after testing other servers (Killing Floor, Battlefield 2) and not experiencing the same issues, I'm wondering if it's a L4D2 issue. Testing has been performed by myself on a LAN and a friend connecting in from an outside IP. Traditionally, we've never noticed this jerkiness before unless one of us was running a listen server during a horde event.

So I'm stumped. If anyone's got any ideas, I'd muchly appreciate it.

[Mooga]

The server is running on a VM. SRCDS has issues with running on virtual hardware, something to do with the CPU. Both KF and BF2 run on different engines that might handle virtualization better.

[FurySupreme]

I was kind of hoping it wouldn't be that. I seem to recall one or two other people running VM Source servers without issue on here. To hell with it. I'll nuke the punk and see what happens.