If you want to test to see if the problem affects you, have someone from outside your network add your 3 servers to their favorites list.
Have them refresh the favorites list every 2 seconds. They should NEVER time out. If they do at all, your NAT table is filling up and throwing out the request.
If you do a google on "Actiontec NAT table", you'll see all sorts of problems caused by this, some not even related to Steam. If you're not seeing the issue on your 20/5 connection, consider yourself very lucky as Actiontec's 1k entry NAT table is a serious limiting factor for a lot of people. But on the plus side, there's a workaround at least
Have them refresh the favorites list every 2 seconds. They should NEVER time out. If they do at all, your NAT table is filling up and throwing out the request.
If you do a google on "Actiontec NAT table", you'll see all sorts of problems caused by this, some not even related to Steam. If you're not seeing the issue on your 20/5 connection, consider yourself very lucky as Actiontec's 1k entry NAT table is a serious limiting factor for a lot of people. But on the plus side, there's a workaround at least