Parallel disptach read fragment instance #15639
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #15639 +/- ##
============================================
- Coverage 39.14% 39.12% -0.03%
Complexity 198 198
============================================
Files 4835 4837 +2
Lines 314405 314565 +160
Branches 39436 39458 +22
============================================
- Hits 123067 123059 -8
- Misses 191338 191506 +168 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| @@ -84,6 +84,8 @@ public class FragmentInstanceManager { | |||
| private final ExecutorService intoOperationExecutor; | |||
| private final ExecutorService modelInferenceExecutor; | |||
|
|
|||
| private final ExecutorService dispatchExecutor; | |||
There was a problem hiding this comment.
shouldn't put it here, better put in it in Coordinator. FragmentInstanceManager takes effect actually in backend, however parallel dispatch happends in front end.
| // TODO: (xingtanzjr) currently we use a sequential dispatch policy for READ, which is | ||
| // unsafe for current FragmentInstance scheduler framework. We need to implement the | ||
| // topological dispatch according to dependency relations between FragmentInstances |
There was a problem hiding this comment.
you can delete this todo
| // it | ||
| // will | ||
| // release lots of memory |
There was a problem hiding this comment.
why not put it in one line
| TSStatusCode.INTERNAL_SERVER_ERROR, UNEXPECTED_ERRORS + t.getMessage())); | ||
| } finally { | ||
| for (SubPlan child : plan.getChildren()) { | ||
| queue.add(new Pair<>(child, success)); |
There was a problem hiding this comment.
if not successful, there is no need to continue putting children into queue?
There was a problem hiding this comment.
When a failure occurs, at least one needs to be sent to notify the outer loop to exit
| } | ||
| } catch (InterruptedException e) { | ||
| Thread.currentThread().interrupt(); | ||
| LOGGER.error("Interrupted when dispatching read async", e); |
|
Description
Parallel disptach read fragment instance