Design a compute scheduler that allocates CPU, GPU, memory, and related resources to user-submitted machine learning jobs across multiple heterogeneous clusters.
Users submit jobs with requirements such as container image, command, CPU count, GPU count and type, memory, expected runtime, priority, retry policy, and placement constraints. Each cluster may have a different hardware inventory, current utilization, network topology, accelerator generation, and failure domain.