Could it be that if more than one connection attempts are being made to the server at one time, it brings down the ports?
We might have more than one threads trying to connect to those ports at the same.