Added API to get multi model deployment config

[lu-ohai]

Added API to get multi model deployment config

Input/Output format

• The input format for multi model deployment config is listed as below

{
    "shape": [
        "VM.GPU.A10.2",
        "VM.GPU.A10.4",
        "BM.GPU.A100-v2.8",
        "BM.GPU.H100.8"
    ],
    "configuration": {
        "VM.GPU.A10.4": {
            "parameters": {
                "VLLM_PARAMS": "--trust-remote-code --max-model-len 60000"
            },
            "multi_model_deployment": [
                {
                    "gpu_count": 1,
                    "parameters": {
                        "VLLM_PARAMS": "--trust-remote-code --max-model-len 32000"
                    }
                },
                {
                    "gpu_count": 2,
                    "parameters": {
                        "VLLM_PARAMS": "--trust-remote-code --max-model-len 32000"
                    }
                }
            ]
        }
    }
}

• The output response format is listed as below

{
    "deployment_config": {
        "model_ocid_1": {
            "shape": [
                "BM.GPU.A10.4",
                "BM.GPU4.8",
                "BM.GPU.L40S-NC.4",
                "BM.GPU.A100-v2.8",
                "BM.GPU.H100.8"
            ],
            "configuration": {
                "BM.GPU.A10.4": {
                    "parameters": {
                        "VLLM_PARAMS": "--enforce-eager --max-num-seqs 16 --max-model-len 65536"
                    },
                    "multi_model_deployment": [
                      {
                          "gpu_count": 1,
                          "parameters": {
                              "VLLM_PARAMS": "--trust-remote-code --max-model-len 32000"
                          }
                      },
                      {
                          "gpu_count": 2,
                          "parameters": {
                              "VLLM_PARAMS": "--trust-remote-code --max-model-len 32000"
                          }
                      }
                  ]
                },
            }
        },
        "model_ocid_2": {
            "shape": [
                "BM.GPU.A10.4",
                "BM.GPU4.8",
                "BM.GPU.L40S-NC.4",
                "BM.GPU.A100-v2.8",
                "BM.GPU.H100.8"
            ],
            "configuration": {
                "BM.GPU.A10.4": {
                    "parameters": {
                        "VLLM_PARAMS": "--enforce-eager --max-num-seqs 16 --max-model-len 65536"
                    },
                    "multi_model_deployment": [
                      {
                          "gpu_count": 1,
                          "parameters": {
                              "VLLM_PARAMS": "--trust-remote-code --max-model-len 32000"
                          }
                      },
                      {
                          "gpu_count": 2,
                          "parameters": {
                              "VLLM_PARAMS": "--trust-remote-code --max-model-len 32000"
                          }
                      }
                  ]
                },
            }
        }
    },
    "gpu_allocation": {
        "VM.GPU.A10.4": {
            "models": [
                {
                    "ocid": "model_ocid_1",
                    "gpu_count": 2
                },
                {
                    "ocid": "model_ocid_2",
                    "gpu_count": 2
                }
            ],
            "total_gpus_available": 4
        }
    }
}

Notebook

• No possible gpu allocations savailable

• No primary model id provided

• Primary model id provided (id ending with jwsq) and it gets the maximum gpu count.

[-actions]

📌 Cov diff with main:

📌 Overall coverage:

[-actions]

📌 Cov diff with main:

📌 Overall coverage:

[mrDzurb]

Hi @lu-ohai, can you add more description into the PR? Also add the test and validation details. Share what is the expected input data and what would be the output, just provide a couple of use cases.

[VipulMascarenhas]

overall, the get_multimodel_compatible_shapes API might be a very slow operation given that we have to get the config file from object storage for each model. On average, the get_deployment_config API call takes 4-6 seconds for each model. It could result in a bad experience if user select 2-3 models and waits for 10-15 seconds only to see a message saying the combination is not feasible. We can probably cache the result for each model so that the subsequent calls are faster. Or we can send parallel async requests to fetch multiple configs, instead of reading them sequentially. Some testing will be required to confirm what optimizations may be required.

cc: @mrDzurb

[mrDzurb]

overall, the get_multimodel_compatible_shapes API might be a very slow operation given that we have to get the config file from object storage for each model. On average, the get_deployment_config API call takes 4-6 seconds for each model. It could result in a bad experience if user select 2-3 models and waits for 10-15 seconds only to see a message saying the combination is not feasible. We can probably cache the result for each model so that the subsequent calls are faster. Or we can send parallel async requests to fetch multiple configs, instead of reading them sequentially. Some testing will be required to confirm what optimizations may be required.

cc: @mrDzurb

Totally agree. We should use both technics caching and threadpool.

4-6 seconds to read a file from OS bucket, this is insane :)

[lu-ohai]

@mrDzurb @VipulMascarenhas Based on the testing, fetching configs from three model ids takes roughly 5-6 microseconds, so wondering under which case it takes get_deployment_config api 6 seconds to complete. I think we can add the cache layer the followup pr if needed.

[darenr]

Very nice code

[VipulMascarenhas]

lgtm 👍