pkg/dds/test/dap/integration tests are flaky

[derekxu16]

The tests

pkg/dds/test/dap/integration/debug_attach_test RuntimeError (expected Pass)

are failing on configurations

unittest-asserts-release-win-arm64

00:01 �[32m+0�[0m�[31m -1�[0m: debug mode can attach to a simple script using vmServiceUri �[1m�[31m[E]�[0m�[0m

  SocketException: Write failed (OS Error: The pipe is being closed.

  , errno = 232), port = 0

  dart:io-patch/socket_patch.dart 1246:34       _NativeSocket.write
  dart:io-patch/socket_patch.dart 2004:15       _RawSocket.write
  dart:io-patch/socket_patch.dart 2481:18       _Socket._write
  dart:io-patch/socket_patch.dart 2216:28       _SocketStreamConsumer.write
  dart:io-patch/socket_patch.dart 2168:11       _SocketStreamConsumer.addStream.<fn>
  dart:async/zone.dart 1407:47                  _rootRunUnary
  dart:async/zone.dart 1308:19                  _CustomZone.runUnary
  dart:async/zone.dart 1217:7                   _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 365:11            _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 541:14            _DelayedData.perform
  dart:async/stream_impl.dart 646:11            _PendingEvents.handleNext
  dart:async/stream_impl.dart 617:7             _PendingEvents.schedule.<fn>
  dart:async/zone.dart 1391:47                  _rootRun
  dart:async/zone.dart 1301:19                  _CustomZone.run
  dart:async/zone.dart 1209:7                   _CustomZone.runGuarded
  dart:async/zone.dart 1249:23                  _CustomZone.bindCallbackGuarded.<fn>
  dart:async/zone.dart 1399:13                  _rootRun
  dart:async/zone.dart 1301:19                  _CustomZone.run
  dart:async/zone.dart 1209:7                   _CustomZone.runGuarded
  dart:async/zone.dart 1249:23                  _CustomZone.bindCallbackGuarded.<fn>
  dart:async/schedule_microtask.dart 40:21      _microtaskLoop
  dart:async/schedule_microtask.dart 49:5       _startMicrotaskLoop
  dart:isolate-patch/isolate_patch.dart 118:13  _runPendingImmediateCallback
  dart:isolate-patch/isolate_patch.dart 185:5   _RawReceivePort._handleMessage

https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/pkg-win-release-arm64/1155/overview

This could be a Windows pipe problem, or it could be a problem with how DAP is using pipes.

cc @brianquinlan and @DanTup

[DanTup]

Is this consistently failing or was it just once? Is it only on ARM? (I can't repro with my x86 Windows PC).

[a-siva]

Is this consistently failing or was it just once? Is it only on ARM? (I can't repro with my x86 Windows PC).

It doesn't appear to be a flaky failure as the bot has flagged it as an error.
If you wanted to try out some logging it is possible to use the CI to run that code on a Windows ARM64 machine.

[derekxu16]

Is this consistently failing or was it just once? Is it only on ARM? (I can't repro with my x86 Windows PC).

I think it only failed once, the test is back to passing now on all configurations (https://dart-current-results.web.app/#/filter=pkg/dds/test/dap/integration/debug_attach_test&showAll).

[a-siva]

Closing this based on comment above.

[DanTup]

While investigating #55705 I'm seeing this issue locally quite reliably. Re-opening this to move the discussion here.

The error is caused by the test teardown trying to send a terminate() request to the debug adapter in case tests didn't close it down. However if the test did and the debug adapter was shutting down, there is a race that can result in this error.

What's odd, is that no matter what I try to do to catch this error, I cannot. I've tried try/catch for sync code, around the await, and the code is already in a runZoneGuarded to handle any errors.

Copying my comment from #55705 (comment) (cc @aam):

---

This one is due to how during test session tearDown, client queues command "terminate" to the server's stdin, which races with server process itself shutting down first closing stdin socket, before that "terminate" command has a chance of being written into that server's stdin.

That's the conclusion I arrived at yesterday, though I was unable to handle this error. I can "fix" it by changing the test to not auto-resume-on-exit, but I feel like the teardown should be more robust. I tried adding try/catch for the synchronous + async parts, and the write to the stream is even in a zone, but the error always seems to cause the test failure. Do you know why I don't seem to be able to handle this (the terminate call seems to synchronously write to the stream, so I would've expected a try/catch, or the zone to handle it).

Here I inlined the write call (removing the zone) and added try/catch but it still fails (these are the only write calls).

[DanTup]

So, I can reproduce the issue seen here in an isolated script.. It runs "dart --version" and tries to write to the stdin stream even though the process will is exiting:

import 'dart:convert';
import 'dart:io';

Future<void> main(List<String> arguments) async {
  final process =
      await Process.start(Platform.resolvedExecutable, ['--version']);

  process.stdout.transform(utf8.decoder).listen(print);
  process.stderr.transform(utf8.decoder).listen(print);

  final exitFuture =
      process.exitCode.then((code) => print('Exited with code $code'));

  for (int i = 0; i < 100; i++) {
    try {
      process.stdin.add(ascii.encode('Hello'));
    } catch (e) {
      print('caught write error');
    }
    await Future.delayed(Duration.zero);
  }

  await exitFuture;
}

On most runs, it will fail like this:

Although it seemed like the try/catch would work.

Changing to a zone does not help:

try {
  runZonedGuarded(
    () {
      process.stdin.add(ascii.encode('Hello'));
    },
    (error, stack) {
      print('caught write error via zone');
    },
  );
} catch (e) {
  print('caught write error');
}

Is this a bug, or is there some other way I should be able to handle this error?

[DanTup]

Maybe related to #48501

Seems there was an attempted fix to allow calling flush() to trigger the exception so it can be caught, but that was reverted in 97d8445

I'm not sure how we can handle this, but I'll ping on that issue to see if there are any current options (otherwise I may just change this test to not auto-resume which will avoid the race for this test - though it might show up in others).