Players Getting Disconnected

[PAL-18]

Basically what's happening is players are being disconnected randomly. They get the lost connection to server and after 30 seconds they get disconnected. This doesnt happen to all players, it only happens to certain players and it doesnt affect every player in the server when it happens.

It's my own dedicated server at a datacenter. I have 4 other game servers servers running on it and none of the others suffer from the problem.

BASIC INFO
Server OS: Ubuntu Server 12.04 LTS 64Bit
Processor: Intel® Core™ i7-2600K (3.4Ghz Quad Core, Hyperthreading/Turbo disabled)
Ram: 16GB
Game(s): Team Fortress 2
Start Up Command: -console -game tf -ticrate 66 -pingboost 3 -secure -ip 205.204.84.177 +port 27015 +maxplayers 25 +exec server1.cfg -pidfile /home/******/tf2/orangebox/hlds.pid
Admin Mods: Metamod, Sourcemod

NETWORK AND BANDWIDTH INFO
(This is only needed if you are having a slot-count, bandwidth, or connection issue.)
Bandwidth: 100Mb
External IP: 205.204.84.177

[Mike]

Server.cfg has anything? Firewalls?

[PAL-18]

Nevermind, found out the issue is just to 2 specific people ... who both happen to use the same internet.