[REF] Cache nodes in workflow to speed up construction, other optimizations

[HippocampusGirl]

Summary

I'm working with fMRIPrep and large data sets (via the ENIGMA consortium). In my testing, Nipype workflows with more than a few hundred thousand nodes can be prohibitively slow, so that it can take days before processing starts. This comes specifically from the function _generate_flatgraph and generate_expanded_graph. While my previous pull requests #3184 and #3260 fixed some bottlenecks, I recently had the time to do some more profiling and try to fix some more.
I understand that Pydra is just around the corner, and that switching to it may solve my performance troubles then. However, the small changes outlined here can reduce the time before processing starts by approximately half (in my testing with a workflow of 75k nodes).
I have tried to divide up the changes into independent commits, so that you can decide which ones may be interesting for inclusion.

Here is my profile before starting out. Please not that the absolute durations are not interpretable, because the profiler reduces performance by a lot.

Here is the profile with all proposed changes. Note that _create_flat_graph has become the dark gray box without caption on the bottom right.

List of changes proposed in this PR (pull-request)

• The workflow._has_node function is called for every new connection, and currently iterates through all nodes in the workflow and nested workflows. It then compares each node to the wanted_node. We can speed this up, because networkx graphs are dictionaries and have support the in statement. This way we can reduce the scaling behavior from linear to almost constant time, but only in the best-case scenario that the wanted_node is in the workflow.
  Otherwise, we still need to iterate to find all nested workflows. To overcome this last limitations, we can cache the nested workflows in a set, and update it whenever the graph is changes.
  

  nipype/nipype/pipeline/engine/workflows.py

  Lines 906 to 913 in 4e8f54d

  	def _has_node(self, wanted_node):
  	for node in self._graph.nodes():
  	if wanted_node == node:
  	return True
  	if isinstance(node, Workflow):
  	if node._has_node(wanted_node):
  	return True
  	return False

• The _merge_graphs function uses np.unique for a list of strings. We can use the built-in set to avoid casting to Numpy array. np.unique additionally sorts the input array, which we really don't need here
  

  nipype/nipype/pipeline/engine/utils.py

  Line 756 in 4e8f54d

  if len(np.unique(ids)) != len(ids):

• Use sets wherever order is not important and we just need to check whether a string or a node is contained or not, again reducing linear scaling to a constant time look-up.
  

  nipype/nipype/pipeline/engine/workflows.py

  Lines 173 to 174 in 4e8f54d

  	if destname not in connected_ports[destnode]:
  	connected_ports[destnode] += [destname]

• Remove a topological sort that may not be necessary. Please correct me if I'm wrong :-)
  

  nipype/nipype/pipeline/engine/workflows.py

  Line 942 in 4e8f54d

  nodes = list(nx.topological_sort(self._graph))

• Avoid re-compiling regular expressions in a loop by using Python's built-in functions
  

  nipype/nipype/pipeline/engine/utils.py

  Lines 1114 to 1116 in 4e8f54d

  	if re.match(
  	src_id + r"((\.[a-z](I\.[a-z])?|J)\d+)?$", node.itername
  	):

Acknowledgment

• (Mandatory) I acknowledge that this contribution will be available under the Apache 2 license.

[codecov]

Codecov Report

Merging #3331 (0c485d3) into master (54f5029) will increase coverage by 3.26%.
The diff coverage is 92.68%.

@@            Coverage Diff             @@
##           master    #3331      +/-   ##
==========================================
+ Coverage   65.05%   68.31%   +3.26%     
==========================================
  Files         302      302              
  Lines       39953    48781    +8828     
  Branches     5287     7224    +1937     
==========================================
+ Hits        25992    33327    +7335     
- Misses      12896    14234    +1338     
- Partials     1065     1220     +155     

Flag	Coverage Δ
unittests	68.16% <92.68%> (+3.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
nipype/pipeline/engine/utils.py	76.10% <75.00%> (+1.61%)	⬆️
nipype/pipeline/engine/workflows.py	70.75% <96.96%> (+5.86%)	⬆️
nipype/utils/logger.py	81.60% <0.00%> (-3.01%)	⬇️
nipype/utils/onetime.py	81.81% <0.00%> (-2.80%)	⬇️
nipype/pipeline/plugins/base.py	57.89% <0.00%> (-1.84%)	⬇️
nipype/interfaces/niftyseg/label_fusion.py	55.42% <0.00%> (-1.72%)	⬇️
nipype/interfaces/mrtrix3/preprocess.py	83.72% <0.00%> (-1.58%)	⬇️
nipype/interfaces/diffusion_toolkit/dti.py	61.64% <0.00%> (-1.57%)	⬇️
nipype/pipeline/plugins/legacymultiproc.py	67.64% <0.00%> (-1.48%)	⬇️
nipype/interfaces/freesurfer/base.py	75.18% <0.00%> (-1.44%)	⬇️
... and 101 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 54f5029...0c485d3. Read the comment docs.

[effigies]

Awesome. This all looks correct. A few thoughts, but nothing blocking. Looking at the style check, it looks like black released a new version that affects a lot of files, so it should not block this PR, either.

[effigies]

Two last thoughts, but otherwise this LGTM.

[HippocampusGirl]

Thank you for your quick feedback! I always fell like I'm learning a lot from your code reviews

[effigies]

Thank you!