Fix _consume_object_or_array on unbalanced brackets in JSON strings

[andryak]

Context

Function _consume_object_or_array parses a string that represents a JSON object or array making sure that it contains balanced parentheses.

This does not work when the input string contains a valid JSON object (or array) which itself constains strings with unbalanced parentheses inside.

For instance,

{"foo":"{"}

is valid JSON but would be parsed incorrectly because _consume_object_or_array determines that it contains 2 open brackets and only one closed bracket.

This diff updates the function so that parentheses inside strings are ignored.

Repro

Make a simple app that accepts a dict via flag:

from pydantic_settings import BaseSettings, CliApp, CliSettingsSource

class Run(BaseSettings):
    field: dict[str, str]

if __name__ == "__main__":
    cli_settings = CliSettingsSource(Run)
    CliApp.run(Run, cli_settings_source=cli_settings)

Run the app on a valid JSON with an open bracket in a string field, it fails:

python repro.py --field '{"foo":"{"}'
Traceback (most recent call last):
  File "/Users/andryak/git-repos/repro/venv/lib/python3.12/site-packages/pydantic_settings/sources/providers/cli.py", line 418, in _merge_parsed_list
    val = self._consume_object_or_array(val, merged_list)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/andryak/git-repos/repro/venv/lib/python3.12/site-packages/pydantic_settings/sources/providers/cli.py", line 459, in _consume_object_or_array
    raise SettingsError(f'Missing end delimiter "{close_delim}"')
pydantic_settings.exceptions.SettingsError: Missing end delimiter "}"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/andryak/git-repos/repro/repro.py", line 10, in <module>
    CliApp.run(Run, cli_settings_source=cli_settings)
  File "/Users/andryak/git-repos/repro/venv/lib/python3.12/site-packages/pydantic_settings/main.py", line 552, in run
    cli_settings = cli_settings_source(args=cli_parse_args)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/andryak/git-repos/repro/venv/lib/python3.12/site-packages/pydantic_settings/sources/providers/cli.py", line 309, in __call__
    return self._load_env_vars(parsed_args=self._parse_args(self.root_parser, args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/andryak/git-repos/repro/venv/lib/python3.12/site-packages/pydantic_settings/sources/providers/cli.py", line 347, in _load_env_vars
    parsed_args[field_name] = self._merge_parsed_list(val, field_name)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/andryak/git-repos/repro/venv/lib/python3.12/site-packages/pydantic_settings/sources/providers/cli.py", line 441, in _merge_parsed_list
    raise SettingsError(f'Parsing error encountered for {field_name}: {e}')
pydantic_settings.exceptions.SettingsError: Parsing error encountered for field: Missing end delimiter "}"

Test plan

After the fix in this diff, the repro above works.

Tested more combinations through this script and all work after the fix:

TESTS=(
    '{}'
    '{"key":""}'
    '{"key":"{}"}'
    '{"key":"{"}'
    '{"key":"}"}'
    '{"key":"["}'
    '{"key":"]"}'
    '{"key1":"value","key2":[""]}'
    '{"key1":"value","key2":["{"]}'
    '{"key1":"value","key2":["}"]}'
    '{"key1":"value","key2":["["]}'
    '{"key1":"value","key2":["]"]}'
    '{"key1":"value","key2":["a","b"]}'
)

for test in "${TESTS[@]}"; do
  python repro.py "$test" &> /dev/null
  echo "$? -> $test"
done

Existing and new unit tests pass: make test

How did we discover this

We pass complex JSON as arguments to a pydantic app that expects a dict via an arg file, then realized that some would fail to parse. We then bisected the JSON until we found a minimal repro and run the code through a debugger to identify the issue.

[mikelaurence]

+1

[hramezani]

Thanks @andryak for this PR.

@kschwab Can you take a look when you have time?

[kschwab]

Thanks @andryak! @hramezani looks good to me.

[hramezani]

Thanks all