Skip to content

Error in make evaluate: TypeError: VLLMModelConfig.__init__() got an unexpected keyword argument '' #163

Open
@mattdepaolis

Description

@mattdepaolis

When running the make evaluate command with the specified parameters, an error occurs indicating that the VLLMModelConfig initializer received an unexpected keyword argument.

Steps to Reproduce:

  1. Set up the environment as per the project's guidelines.
  2. Execute the following command:
make evaluate MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B TASK=math_500

Expected Behavior: The evaluation process should complete without errors, utilizing the specified model and task parameters.

Actual Behavior: The following error message is displayed:

TypeError: VLLMModelConfig.__init__() got an unexpected keyword argument ''

Analysis: Upon investigation, it appears that the MODEL_ARGS variable in the Makefile's evaluate target may include an empty $(PARALLEL_ARGS) variable, leading to consecutive commas and resulting in an empty string being passed as a keyword argument.

Proposed Solution: Modify the MODEL_ARGS assignment in the Makefile to conditionally include $(PARALLEL_ARGS) only if it's not empty:

MODEL_ARGS="pretrained=$(MODEL),dtype=float16$(if $(PARALLEL_ARGS),,$(PARALLEL_ARGS)),max_model_length=32768,gpu_memory_utilization=0.8"

This change ensures that no extra comma is introduced when $(PARALLEL_ARGS) is empty, preventing the unintended empty string argument.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions