Provide additional functions for import (#639)

* Provide additional functions for import

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add cloudpickle_register to API

* add socket functions

* Separate user interface and developer interface

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add documentation

* add links in readme

* add explanation

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: jan-janssen

README.md

@@ -158,5 +158,6 @@ as hierarchical job scheduler within the allocations.
   * [Modules](https://executorlib.readthedocs.io/en/latest/4-developer.html#modules)
   * [Interface Class Hierarchy](https://executorlib.readthedocs.io/en/latest/4-developer.html#interface-class-hierarchy)
   * [Communication](https://executorlib.readthedocs.io/en/latest/4-developer.html#communication)
+  * [External Libraries](https://executorlib.readthedocs.io/en/latest/4-developer.html#external-libraries)
   * [External Executables](https://executorlib.readthedocs.io/en/latest/4-developer.html#external-executables)
 * [Interface](https://executorlib.readthedocs.io/en/latest/api.html)

executorlib/standalone/__init__.py

@@ -1,8 +1,10 @@
 """
 Submodules in the executorlib.standalone module do not depend on other modules of the executorlib package. This strict
-separation simplifies the development, testing and debugging.
+separation simplifies the development, testing and debugging. The functionality in executorlib.standalone is designed
+to be used independently in other libraries.
 """
 
+from executorlib.standalone.command import get_command_path
 from executorlib.standalone.interactive.communication import (
     SocketInterface,
     interface_bootup,
@@ -11,14 +13,20 @@
     interface_send,
     interface_shutdown,
 )
-from executorlib.standalone.interactive.spawner import MpiExecSpawner
+from executorlib.standalone.interactive.spawner import MpiExecSpawner, SubprocessSpawner
+from executorlib.standalone.queue import cancel_items_in_queue
+from executorlib.standalone.serialize import cloudpickle_register
 
-__all__ = [
-    "SocketInterface",
+__all__: list[str] = [
+    "cancel_items_in_queue",
+    "cloudpickle_register",
+    "get_command_path",
     "interface_bootup",
     "interface_connect",
+    "interface_receive",
     "interface_send",
     "interface_shutdown",
-    "interface_receive",
     "MpiExecSpawner",
+    "SocketInterface",
+    "SubprocessSpawner",
 ]

notebooks/4-developer.ipynb

@@ -109,7 +109,37 @@
   {
    "id": "7bc073aa-6036-48e7-9696-37af050d438a",
    "cell_type": "markdown",
-   "source": "## Communication\nThe key functionality of the `executorlib` package is the up-scaling of python functions with thread based parallelism, MPI based parallelism or by assigning GPUs to individual python functions. In the background this is realized using a combination of the [zero message queue](https://zeromq.org) and [cloudpickle](https://github.com/cloudpipe/cloudpickle) \nto communicate binary python objects. The `executorlib.standalone.interactive.communication.SocketInterface` is an abstraction of this \ninterface, which is used in the other classes inside `executorlib` and might also be helpful for other projects. It comes with a series of utility functions:\n\n* `executorlib.standalone.interactive.communication.interface_bootup()`: To initialize the interface\n* `executorlib.standalone.interactive.communication.interface_connect()`: To connect the interface to another instance\n* `executorlib.standalone.interactive.communication.interface_send()`: To send messages via this interface \n* `executorlib.standalone.interactive.communication.interface_receive()`: To receive messages via this interface \n* `executorlib.standalone.interactive.communication.interface_shutdown()`: To shutdown the interface\n\nWhile `executorlib` was initially designed for up-scaling python functions for HPC, the same functionality can be \nleveraged to up-scale any executable independent of the programming language it is developed in.",
+   "source": [
+    "## Communication\n",
+    "The key functionality of the executorlib package is the up-scaling of python functions with thread based parallelism, MPI based parallelism or by assigning GPUs to individual python functions. In the background this is realized using a combination of the [zero message queue](https://zeromq.org) and [cloudpickle](https://github.com/cloudpipe/cloudpickle)\n",
+    "to communicate binary python objects. The `executorlib.standalone.interactive.communication.SocketInterface` is an abstraction of this \n",
+    "interface, which is used in the other classes inside `executorlib` and might also be helpful for other projects. It comes with a series of utility functions:\n",
+    "\n",
+    "* `executorlib.standalone.interactive.communication.interface_bootup()`: To initialize the interface\n",
+    "* `executorlib.standalone.interactive.communication.interface_connect()`: To connect the interface to another instance\n",
+    "* `executorlib.standalone.interactive.communication.interface_send()`: To send messages via this interface \n",
+    "* `executorlib.standalone.interactive.communication.interface_receive()`: To receive messages via this interface \n",
+    "* `executorlib.standalone.interactive.communication.interface_shutdown()`: To shutdown the interface\n",
+    "\n",
+    "While executorlib was initially designed for up-scaling python functions for HPC, the same functionality can be\n",
+    "leveraged to up-scale any executable independent of the programming language it is developed in.\n",
+    "\n",
+    "## External Libraries\n",
+    "For external libraries executorlib provides a standardized interface for a subset of its internal functionality, which is designed to remain stable with minor version updates. Developers can import the following functionality from `executorlib.standalone`:\n",
+    "* `cancel_items_in_queue()` - Cancel items which are still waiting in the Python standard library queue - `queue.queue`.\n",
+    "* `cloudpickle_register()` - Cloudpickle can either pickle by value or pickle by reference. The functions which are communicated have to be pickled by value rather than by reference, so the module which calls the map function is pickled by value.\n",
+    "* `get_command_path()` - Get path of the backend executable script `executorlib.backend`.\n",
+    "* `interface_bootup()` - Start interface for ZMQ communication.\n",
+    "* `interface_connect()` - Connect to an existing `SocketInterface` instance by providing the hostname and the port as strings.\n",
+    "* `interface_receive()` - Receive instructions from a `SocketInterface` instance.\n",
+    "* `interface_send()` - Send results to a `SocketInterface` instance.\n",
+    "* `interface_shutdown()` - Close the connection to a `SocketInterface` instance.\n",
+    "* `MpiExecSpawner` - Subprocess interface to start `mpi4py` parallel process.\n",
+    "* `SocketInterface` - The `SocketInterface` is an abstraction layer on top of the zero message queue.\n",
+    "* `SubprocessSpawner` - Subprocess interface to start serial Python process.\n",
+    "\n",
+    "It is not recommended to import components from other parts of executorlib in other libraries, only the interfaces in `executorlib` and `executorlib.standalone` are designed to be stable. All other classes and functions are considered for internal use only."
+   ],
    "metadata": {}
   },
   {