QUESTION: How to provide dynamic values to settings sources

[danizen]

I have a scenario where I want to use pydantic settings to make it easier to work with AWS Glue job arguments, which look to the script like command-line arguments. How can I provide dynamic values in settings_customize_sources?

When the Glue Job is run from within an AWS Glue workflow, the workflow parameters take precedence over the Glue job arguments, except for a few exceptions. However, I am also trying to provide a library that works for any AWS Glue job, so that I want the configuration file name to be dynamic. I also am required to log audit events, but the user of the configuration file needs to provide some details. So, the basic are clear, but I'd like to have some dynamic arguments settings_customize_sources are available within.

The BaseSettings object will have some parameters that come from the command-line and control the GlueWorkflowConfigSource, so that I've considered adding a path: Path to the base settings.

class CommonJobConfig(BaseSettings):
    job_name: str = Field(alias='JOB_NAME')
    job_run_id: str = Field(alias='JOB_RUN_ID')
    workflow_name: Optional[str] = Field(default=None, alias='WORKFLOW_NAME')
    workflow_run_id: Optional[str] = Field(default=None, alias='WORKFLOW_RUN_ID')

    # some other proprietary settings...

    @classmethod
    def settings_customise_sources(
            cls,
            settings_cls: Type[BaseSettings],
            init_settings: PydanticBaseSettingsSource,
            env_settings: PydanticBaseSettingsSource,
            dotenv_settings: PydanticBaseSettingsSource,
            file_secret_settings: PydanticBaseSettingsSource,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        # NOTE: not quite sure how to make the file path configurable
        return (
            init_settings,
            env_settings,
            CliSettingsSource(settings_cls, cli_parse_args=True, cli_ignore_unknown_args=True),
            ConfigFileSettingsSource(settings_cls, 'appconfig.yaml'),     # path should be dynamic
            GlueWorkflowConfigSource(settings_cls),                                # glue workflow parameter resolver needs something dynamic
        )

[hramezani]

Thanks @danizen for reporting this issue.

Unfortunately, there is not a clear way to do this. there is an open issue for this and you can find some workaround there.

We can mark this issue as a duplicate of #259 and continue with the original issue. what do you think?

[danizen]

Duplicate of #259.