Description
When running the make evaluate
command with the specified parameters, an error occurs indicating that the VLLMModelConfig
initializer received an unexpected keyword argument.
Steps to Reproduce:
- Set up the environment as per the project's guidelines.
- Execute the following command:
make evaluate MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B TASK=math_500
Expected Behavior: The evaluation process should complete without errors, utilizing the specified model and task parameters.
Actual Behavior: The following error message is displayed:
TypeError: VLLMModelConfig.__init__() got an unexpected keyword argument ''
Analysis: Upon investigation, it appears that the MODEL_ARGS
variable in the Makefile
's evaluate
target may include an empty $(PARALLEL_ARGS)
variable, leading to consecutive commas and resulting in an empty string being passed as a keyword argument.
Proposed Solution: Modify the MODEL_ARGS
assignment in the Makefile
to conditionally include $(PARALLEL_ARGS)
only if it's not empty:
MODEL_ARGS="pretrained=$(MODEL),dtype=float16$(if $(PARALLEL_ARGS),,$(PARALLEL_ARGS)),max_model_length=32768,gpu_memory_utilization=0.8"
This change ensures that no extra comma is introduced when $(PARALLEL_ARGS)
is empty, preventing the unintended empty string argument.