Nested dataclass settings class gives `AttributeError: __pydantic_fields__`

[bschoenmaeckers]

I have a dataclass that I don't control as a nested object (wrapped by a BaseSettings class). This used to work great but after v2.3.0 (#214) it throws a AttributeError. See the following testcase that passes on <=v2.2.1 but fails on >=v2.3.0.

def test_nested_dataclass_setting(env):
    @dataclasses.dataclass
    class DataClass:
        value: str

    class SubSettings(BaseSettings, DataClass):
        model_config = SettingsConfigDict(alias_generator=str.lower)

    class Cfg(BaseSettings):
        model_config = SettingsConfigDict(env_nested_delimiter='__')

        sub: SubSettings

    env.set('SUB__VALUE', 'something')
    cfg = Cfg()

The exception

pydantic_settings\main.py:141: in __init__
  **__pydantic_self__._settings_build_values(
pydantic_settings\main.py:260: in _settings_build_values
  CliSettingsSource(
pydantic_settings\sources.py:902: in __init__
  self._connect_root_parser(
pydantic_settings\sources.py:1236: in _connect_root_parser
  self._add_parser_args(
pydantic_settings\sources.py:1328: in _add_parser_args
  self._add_parser_args(
pydantic_settings\sources.py:1255: in _add_parser_args
  for field_name, resolved_name, field_info in self._sort_arg_fields(model):
pydantic_settings\sources.py:1151: in _sort_arg_fields
  fields = model.__pydantic_fields__ if is_pydantic_dataclass(model) else model.model_fields
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <class 'test.test_nested_dataclass_setting.<locals>.SubSettings'>, item = '__pydantic_fields__'

  def __getattr__(self, item: str) -> Any:
      """This is necessary to keep attribute access working for class attribute access."""
      private_attributes = self.__dict__.get('__private_attributes__')
      if private_attributes and item in private_attributes:
          return private_attributes[item]
      if item == '__pydantic_core_schema__':
          # This means the class didn't get a schema generated for it, likely because there was an undefined reference
          maybe_mock_validator = getattr(self, '__pydantic_validator__', None)
          if isinstance(maybe_mock_validator, MockValSer):
              rebuilt_validator = maybe_mock_validator.rebuild()
              if rebuilt_validator is not None:
                  # In this case, a validator was built, and so `__pydantic_core_schema__` should now be set
                  return getattr(self, '__pydantic_core_schema__')
>       raise AttributeError(item)
E       AttributeError: __pydantic_fields__. Did you mean: '__pydantic_fields_set__'?

maybe related to #303

[hramezani]

@kschwab Could you please take a look?

[kschwab]

@bschoenmaeckers I'm unable to reproduce the issue. Can you paste the versions for pydantic and pydantic-settings? The above should be resolved in pydantic-settings v2.3.2.

Also, the SubSettings class should inherit BaseModel and not BaseSettings. See relevant note in Parsing environment variable values.

[bschoenmaeckers]

Tested it on the latest commit of main:

>python -c "import pydantic; print(pydantic.version.version_info())" 
             pydantic version: 2.7.0
        pydantic-core version: 2.18.1
          pydantic-core build: profile=release pgo=true
                 install path: D:\repo\pydantic-settings\.venv\Lib\site-packages\pydantic
               python version: 3.12.4 (tags/v3.12.4:8e8a4ba, Jun  6 2024, 19:30:16) [MSC v.1940 64 bit (AMD64)]
                     platform: Windows-11-10.0.22631-SP0
             related packages: typing_extensions-4.11.0
                       commit: unknown

[bschoenmaeckers]

Also, the SubSettings class should inherit BaseModel and not BaseSettings. See relevant note in Parsing environment variable values.

My SubSettings has to ingerit from BaseSettings because I want to set custom config values to that sub model only, like secrets_dir for example.

[bschoenmaeckers]

If you run my example with pytest on the current main then it should crash.

Run pytest test.py

###test.py

import dataclasses
import os

import pytest

from pydantic_settings import BaseSettings, SettingsConfigDict


class SetEnv:
    def __init__(self):
        self.envars = set()

    def set(self, name, value):
        self.envars.add(name)
        os.environ[name] = value

    def pop(self, name):
        self.envars.remove(name)
        os.environ.pop(name)

    def clear(self):
        for n in self.envars:
            os.environ.pop(n)


@pytest.fixture
def env():
    setenv = SetEnv()

    yield setenv

    setenv.clear()


def test_nested_dataclass_setting(env):
    @dataclasses.dataclass
    class DataClass:
        value: str

    class SubSettings(BaseSettings, DataClass):
        model_config = SettingsConfigDict(alias_generator=str.lower)

    class Cfg(BaseSettings):
        model_config = SettingsConfigDict(env_nested_delimiter='__')

        sub: SubSettings

    env.set('SUB__VALUE', 'something')
    cfg = Cfg()

[kschwab]

Thanks @bschoenmaeckers. It was the dataclasses. I was using pydantic dataclasses instead of generic dataclasses.

I was able to reproduce prior to pydantic-settings 2.3.2. For this case, it should be resolved with pydantic-settings 2.3.2+. Can you confirm?

However, it does raise a potential issue with either CLI settings or pydantic is_pydantic_dataclass. @hramezani is the below expected:

import dataclasses
from pydantic import BaseModel
from pydantic.dataclasses import is_pydantic_dataclass

@dataclasses.dataclass
class DataClass:
    value: str

class DataModel(BaseModel, DataClass):
    ...

print(is_pydantic_dataclass(DataModel))
#> True

[bschoenmaeckers]

Thanks @bschoenmaeckers. It was the dataclasses. I was using pydantic dataclasses instead of generic dataclasses.

Yes my bad, it is the vanilla dataclass.

For this case, it should be resolved with pydantic-settings 2.3.2+.

My test passes for version 2.3.2 & 2.3.3 but it fails again for 2.3.4 & 2.4.0

[kschwab]

Got it. Yes, the 2.3.4 and 2.4.0 failures are different from the CliSettingsSource exception:

pydantic_settings/main.py:145: in __init__
    **__pydantic_self__._settings_build_values(
pydantic_settings/main.py:330: in _settings_build_values
    source_state = source()
pydantic_settings/sources.py:430: in __call__
    field_value = self.prepare_field_value(field_name, field, field_value, value_is_complex)
pydantic_settings/sources.py:623: in prepare_field_value
    env_val_built = self.explode_env_vars(field_name, field, self.env_vars)
pydantic_settings/sources.py:749: in explode_env_vars
    target_field = self.next_field(target_field, last_key, self.case_sensitive)
pydantic_settings/sources.py:700: in next_field
    annotation.__pydantic_fields__
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <class 'test_bed.test_nested_dataclass_setting.<locals>.SubSettings'>, item = '__pydantic_fields__'

    def __getattr__(self, item: str) -> Any:
        """This is necessary to keep attribute access working for class attribute access."""
        private_attributes = self.__dict__.get('__private_attributes__')
        if private_attributes and item in private_attributes:
            return private_attributes[item]
        if item == '__pydantic_core_schema__':
            # This means the class didn't get a schema generated for it, likely because there was an undefined reference
            maybe_mock_validator = getattr(self, '__pydantic_validator__', None)
            if isinstance(maybe_mock_validator, MockValSer):
                rebuilt_validator = maybe_mock_validator.rebuild()
                if rebuilt_validator is not None:
                    # In this case, a validator was built, and so `__pydantic_core_schema__` should now be set
                    return getattr(self, '__pydantic_core_schema__')
>       raise AttributeError(item)
E       AttributeError: __pydantic_fields__. Did you mean: '__pydantic_fields_set__'?

../.local/lib/python3.12/site-packages/pydantic/_internal/_model_construction.py:242: AttributeError

@hramezani this is in the PydanticBaseEnvSettingsSource, also related to is_pydantic_dataclass query from above.

[hramezani]

However, it does raise a potential issue with either CLI settings or pydantic is_pydantic_dataclass. @hramezani is the below expected:

@kschwab Yes, dataclasses.is_dataclass(DataModel) returns True also.

@hramezani this is in the PydanticBaseEnvSettingsSource, also related to is_pydantic_dataclass query from above.

I've created #357 to fix the problem. Take a look if you have time.