Skip to content

[Bug]: IllegalStateChangeError in delete_metrics task #2638

Closed
@un-def

Description

@un-def

Steps to reproduce

Start the server and wait

Actual behaviour

No response

Expected behaviour

No response

dstack version

0.19.9

Server logs

InvalidRequestError: This session is provisioning a new connection; concurrent operations are not permitted (Background on this error at: https://sqlalche.me/e/20/isce)
  File "dstack/_internal/server/db.py", line 87, in get_session
    yield session
  File "dstack/_internal/server/background/tasks/process_metrics.py", line 53, in delete_metrics
    await asyncio.gather(
  File "sqlalchemy/ext/asyncio/session.py", line 463, in execute
    result = await greenlet_spawn(
  File "sqlalchemy/util/_concurrency_py3k.py", line 190, in greenlet_spawn
    result = context.switch(*args, **kwargs)
  File "sqlalchemy/orm/session.py", line 2365, in execute
    return self._execute_internal(
  File "sqlalchemy/orm/session.py", line 2241, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "sqlalchemy/orm/session.py", line 2110, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
    # Copyright (C) 2005-2025 the SQLAlchemy authors and contributors
  File "sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "sqlalchemy/orm/session.py", line 990, in _raise_for_prerequisite_state
    raise sa_exc.InvalidRequestError(

IllegalStateChangeError: Method 'close()' can't be called here; method '_connection_for_bind()' is already in progress and this would cause an unexpected state change to <SessionTransactionState.CLOSED: 5> (Background on this error at: https://sqlalche.me/e/20/isce)
  File "apscheduler/executors/base.py", line 181, in run_coroutine_job
    retval = await job.func(*job.args, **job.kwargs)
  File "dstack/_internal/server/background/tasks/process_metrics.py", line 52, in delete_metrics
    async with get_session_ctx() as session:
  File "contextlib.py", line 231, in __aexit__
    await self.gen.athrow(typ, value, traceback)
  File "dstack/_internal/server/db.py", line 86, in get_session
    async with _db.get_session() as session:
  File "sqlalchemy/ext/asyncio/session.py", line 1031, in close
    await greenlet_spawn(self.sync_session.close)
  File "sqlalchemy/util/_concurrency_py3k.py", line 190, in greenlet_spawn
    result = context.switch(*args, **kwargs)
  File "sqlalchemy/orm/session.py", line 2521, in close
    self._close_impl(invalidate=False)
  File "sqlalchemy/orm/session.py", line 2590, in _close_impl
    transaction.close(invalidate)
  File "<string>", line 2, in close
    # Copyright (C) 2005-2025 the SQLAlchemy authors and contributors
  File "sqlalchemy/orm/state_changes.py", line 121, in _go
    raise sa_exc.IllegalStateChangeError(

Additional information

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingmetrics

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions