Resolved
Due to a network configuration problem our RTX-6000 instances aren't properly peering with the network. In order to fix this the hosts had to be rebooted. If you have a VM on these instances your data is safe but any running jobs were terminated. The hosts should be back online now.