ENV override not properly explained

[Verhaeg]

According to the documentation, sub level environments can be overwritten by more specific definitions:

https://docs.pydantic.dev/latest/usage/pydantic_settings/#parsing-environment-variable-values

export V0=0
export SUB_MODEL='{"v1": "json-1", "v2": "json-2"}'
export SUB_MODEL__V2=nested-2
export SUB_MODEL__V3=3
export SUB_MODEL__DEEP__V4=v4

In this case, Settings.SubModel.V2 = 'nested-2' and that's ok. But what is NOT mentioned in the documentation, is if you have an Environment named V2 and specially if you reuse that value in multiple submodules. The value is passed to each submodule.

This is very frustrating, specially when dealing with the ENV PORT as it is very common for multiple services to have such definition (i.e.: Databases, Services, etc)

What I had to do so far:

from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict

class RedisConfig(BaseSettings):
    """Configuration for Redis"""
    host: str = Field(default='localhost')
    # Setting alias to prevent override from env PORT
    port: int = Field(default=6379, gt=0, alias='redis.port')
    db: int = Field(default=0, ge=0, le=15)
    prefix: Optional[str] = Field(default=None)
    ttl: Optional[int] = Field(default=None, gt=0)

class ServiceWithCache(BaseSettings):
    url: str
    redis: RedisConfig

class Config(BaseSettings):
    model_config = SettingsConfigDict(
        env_nested_delimiter='__',
        env_file=('dist.env', '.env'),
        env_file_encoding='utf-8',
        extra='ignore',
        use_enum_values=True,
        populate_by_name=True
    )

    cache: RedisConfig
    service: ServiceWithCache

config = Config()

The example above "works" but is not very nice, as I had to add populate_by_name and set different and "unusable" alias for the port field.

Otherwise, if I had the PORT=8000 env for example, all my Redis instances where I thought I was using the default port value (6379) would try to use port 8000.

This should AT least be in the documentation! But I think this should be fixed and prevented from being the default case.

Selected Assignee: @samuelcolvin

[hramezani]

Thanks @Verhaeg for reporting this 🙏

I've tested the example in the doc and created a V2=test env variable before running the example. but the result was the same.

So, I couldn't reproduce the problem.
Please let me know what is your python, Pydantic, pydantic-core, and pydantic-settings version.
Update all of the above packages to the latest version if you can.
If you still have the problem, Please check your environment variables

Please let me know what is the result.

[Verhaeg]

Ok.. Here is my previous code (simplified) and example of executions:

# pylint: disable=too-few-public-methods,invalid-name
from enum import Enum
from typing import Annotated, Literal, Optional

from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict


class BaseSettingsIgnore(BaseSettings):
    """Basic model with already ignoring extra parameters"""
    model_config = SettingsConfigDict(extra='ignore', populate_by_name=True)


class CacheType(str, Enum):
    """Possible values for Cache class"""
    redis = 'redis'
    memory = 'memory'


class EventType(str, Enum):
    """Possible values for Events class"""
    memory = 'memory'
    pubsub = 'pubsub'
    redis = 'redis'


class RedisConfig(BaseSettingsIgnore):
    """Configuration for Redis"""
    host: str = Field(default='localhost')
    # Setting alias to prevent override from env PORT
    port: int = Field(default=6379, gt=0)
    db: int = Field(default=0, ge=0, le=15)
    prefix: Optional[str] = Field(default=None)
    ttl: Optional[int] = Field(default=None, gt=0)


class MemoryCacheConfig(BaseSettingsIgnore):
    """Memory type cache config - Dummy usage"""
    type: Literal[CacheType.memory]
    ttl: int = Field(default=60 * 60 * 24, ge=0)


class RedisCacheConfig(RedisConfig):
    """Configuration for Redis"""
    type: Literal[CacheType.redis]


CacheAnnotation = Annotated[RedisCacheConfig | MemoryCacheConfig, Field(discriminator='type')]


class FeatureFlagConfig(BaseSettingsIgnore):
    """Configurations for SplitIO"""
    key: str = Field(default='localhost')
    file: Optional[str] = Field(default=None)
    redis: Optional[RedisConfig] = Field(default=None)


class PubSubEventHandler(BaseSettingsIgnore):
    """Configuration for PubSub"""
    type: Literal[EventType.pubsub]
    project: str
    topic: str


class RedisEventHandler(RedisConfig):
    """Configuration for PubSub"""
    type: Literal[EventType.redis]


class MemoryEventHandler(BaseSettingsIgnore):
    """Memory dummy event handler"""
    type: Literal[EventType.memory]


EventAnnotation = Annotated[Optional[MemoryEventHandler | PubSubEventHandler | RedisEventHandler],
                            Field(default=None, discriminator='type')]


class Config(BaseSettings):
    """Base settings for project"""
    model_config = SettingsConfigDict(
        env_nested_delimiter='__',
        env_file=('dist.env', '.env'),
        env_file_encoding='utf-8',
        extra='ignore',
        use_enum_values=True,
        populate_by_name=True
    )

    cache: CacheAnnotation
    events: EventAnnotation
    feature_flag: FeatureFlagConfig

    max_per_page: int = Field(default=10, gt=0)


config = Config()  # pyright: ignore

if __name__ == '__main__':
    print(config.model_dump())

.env file

FEATURE_FLAG__KEY=localhost
FEATURE_FLAG__FILE=test.ff.yaml

CACHE__TYPE=redis
CACHE__HOST=localhost
CACHE__TTL=60

Tests:

❯ PORT=8000 python config.py
{'cache': {'host': 'localhost', 'port': 8000, 'db': 0, 'prefix': None, 'ttl': 60, 'type': <CacheType.redis: 'redis'>}, 'events': None, 'feature_flag': {'key': 'localhost', 'file': 'test.ff.yaml', 'redis': None}, 'max_per_page': 10}

See that Redis port for cache is 8000 instead of default 6379

Sorry for the long example, perhaps it is due to some especial ENV cases like PORT?
Oh.. and by the way, I'm using the latest versions for pydantic and pydantic-settings

[hramezani]

The problem is because sub models are inheriting from BaseSettings. They should inherit from BaseModel.
I've explained the reason in a similar issue.

Also, you can see in the example that all sub models are inheriting from BaseModel.

So, you can fix the problem by changing class BaseSettingsIgnore(BaseSettings): to class BaseSettingsIgnore(BaseModel):

[Verhaeg]

Thanks for the informaion @hramezani, double-checked that documentation uses BaseModel (did not check against that) but also didn't find mention to this "condition".. I thought that it would only behave like that if initialized in the outer scope, not as sub-settings.

I think it would be nice to have this mentioned in the documentation ;)

In any case, it is explained ;). I'll change my code to follow this standard.

Thanks for the help

[hramezani]

I think it would be nice to have this mentioned in the documentation ;)

Yes, agree.

would you like to open a PR?

[hramezani]

Closed in f7e810d