Enable AQUA SDK & CLI to Deploy Fine-Tuned LLMs in Multi-Model Deployment #1175
Conversation
📌 Cov diff with main: 📌 Overall coverage: |
📌 Cov diff with main: 📌 Overall coverage: |
| @@ -311,6 +312,12 @@ def create_multi( | |||
| # "Currently only service models are supported for multi model deployment." | |||
| # ) | |||
| is_fine_tuned_model = Tags.AQUA_FINE_TUNED_MODEL_TAG in source_model.freeform_tags | |||
There was a problem hiding this comment.
Could you add some comment what we are doing here?
ads/aqua/finetuning/entities.py Outdated
| @@ -179,3 +185,42 @@ class CreateFineTuningDetails(Serializable): | |||
| class Config: | |||
| extra = "ignore" | |||
| def extract_base_model_ocid(aqua_model: DataScienceModel) -> Tuple[str, str]: | |||
There was a problem hiding this comment.
The name of the function is a bit confusing, it says that it extracts the base model ocid, but returns name and ocid.
Also I think it would be better to create model/utils.py module and move this logic there. Same for the set_finetune_env_var.
ads/aqua/finetuning/entities.py Outdated
| env_var.update({"FT_MODEL": f"{fine_tune_output_path}"}) | ||
| return env_var |
There was a problem hiding this comment.
looks like we are already modifying the original env_var, probably we don't need to return anything form this function?
Could you add more description in the PR? It would be also helpful if you put in the details how the new config will look like. |
📌 Cov diff with main: 📌 Overall coverage: |
📌 Cov diff with main: 📌 Overall coverage: |
📌 Cov diff with main: 📌 Overall coverage: |
📌 Cov diff with main: 📌 Overall coverage: |
| @@ -157,6 +157,8 @@ class AquaMultiModelRef(Serializable): | |||
| Optional environment variables to override during deployment. | |||
| artifact_location : Optional[str] | |||
| Artifact path of model in the multimodel group. | |||
| fine_tune_artifact : Optional[str] | |||
There was a problem hiding this comment.
I think it would be better to be more specific, according to the comment, this is more about fine_tuned_weights_location, right?
| return config_source_id, model_name | ||
| def set_fine_tune_env_var(aqua_model: DataScienceModel, env_var: Optional[Dict[str,str]], model: Optional[AquaMultiModelRef] = None) -> None: |
There was a problem hiding this comment.
I'm not sure if we really want to combine two flows in one scope. Do we really need to setup theFT_MODEL env variable for multi-model deployment case? If not, I would prefer to keep single-model and multi-model deployment logics separately.
📌 Cov diff with main: 📌 Overall coverage: |
This PR aims to add support for using fine-tuned models in a Multi-Model deployment.
When deploying a model group w/ a fine tuned model