Lagg on LAN Server

[Dude]

I have a server that goes through to the internet and I connect to it through LAN, I get extreme lagg spikes every so often. How do I stop this from happening?

I don't think specs have to do with it because other people on the server are fine.