Hi,

that's an issue with Silverlight. The problem is that if the silverlight engine does not receive the clientaccesspolicy.xml file the fist time it tries to download it, then it will never try again to download it assuming that it has no permission to connect (you can see that if the server is down the exception thrown is security related).
A workaround could be to have some aliases for the server machine and then, if a connection fails due to security problems, try with a different alias.

HTH