The Adapter Remoting Infrastructure assumes a scenario in which the Remote Server running the Remote Adapter is a totally independent process and may run on a different machine. Hence, it is not launched by the Server.

I confirm that the Server startup is delayed until all the Remote Adapters have been found.
However, by using a "Robust Proxy Data Adapter", you can overcome this limitation
(see DOCS-SDKs\sdk_adapter_remoting_infrastructure\conf\sock ets(robust) in the distribution package).
In this case, if a Remote Data Adapter is missing, it is treated as an existing Data Adapter which provides no updates at all. This must be compatible with your front-end behavior.