feat: protocol version 2 and n-dimensional array ingestion support

Author: kafka1991

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 2.0.4
+current_version = 3.0.0rc1
 commit = True
 tag = False
 

CHANGELOG.rst

@@ -1,8 +1,74 @@
 .. _changelog:
 
+
 Changelog
+
 =========
 
+3.0.0rc1 (2025-06-02)
+---------------------
+
+This is the pre-release of a major release introducing array ingestion and some
+minor breaking changes.
+
+Features
+~~~~~~~~
+* Array Data Type Support. Adds native support for NumPy arrays
+  (currently only for ``np.float64`` element type and up to 32 dimensions).
+
+.. code-block:: python
+
+        import numpy as np
+
+        # Create 2D numpy array
+        array_2d = np.array([
+            [1.1, 2.2, 3.3],
+            [4.4, 5.5, 6.6]], dtype=np.float64)
+
+        sender.row(
+            'table',
+            columns={'array_2d': array_2d},
+            at=timestamp)
+
+* Implements binary protocol for columns of ``float`` (double-precision) and
+  ``numpy.ndarray[np.float64]``, with performance improvements for these
+  two datatypes.
+
+Breaking Changes
+~~~~~~~~~~~~~~~~
+* Buffer Constructor Changes. The ``Buffer`` constructor now requires the ``protocol_version`` parameter.
+  You can create buffer through the sender for automatic ``protocol_version`` management:
+
+.. code-block:: python
+
+    buf = sender.new_buffer()  # protocol_version determined automatically
+    buf.row(
+      'table',
+      columns={'arr': np.array([1.5, 3.0], dtype=np.float64)},
+      at=timestamp)
+
+* To access the raw payload, call ``bytes(sender)`` or ``bytes(buffer)`` (
+  rather than calling the ``str`` function on the same objects as in version
+  2.x.x of the questdb library) method.
+
+* **NumPy Dependency**
+
+  Array functionality mandates NumPy installation.
+
+* **Sender/Buffer String Conversion Removal**
+
+  The legacy string conversion via `str(sender)` is removed.
+  Access raw binary payloads through the `bytes(sender)` method:
+
+  .. code-block:: python
+
+      # for debugging
+      payload = bytes(sender)
+
+* Python 3.8 support is dropped.
+
+  The minimum supported Python version is now 3.9.
+
 2.0.4 (2025-04-02)
 ------------------
 

README.rst

@@ -5,7 +5,7 @@ QuestDB Client Library for Python
 This is the official Python client library for `QuestDB <https://questdb.io>`_.
 
 This client library implements QuestDB's variant of the
-`InfluxDB Line Protocol <https://questdb.io/docs/reference/api/ilp/overview/>`_
+`Ingestion Line Protocol <https://questdb.io/docs/reference/api/ilp/overview/>`_
 (ILP) over HTTP and TCP.
 
 ILP provides the fastest way to insert data into QuestDB.
@@ -15,16 +15,29 @@ This implementation supports `authentication
 and full-connection encryption with
 `TLS <https://py-questdb-client.readthedocs.io/en/latest/conf.html#tls>`_.
 
-Quickstart
-==========
+Install
+=======
 
-The latest version of the library is 2.0.3 (`changelog <https://py-questdb-client.readthedocs.io/en/latest/changelog.html>`_).
+The latest *stable* version of the library is **2.0.4** (`changelog <https://py-questdb-client.readthedocs.io/en/latest/changelog.html>`_).
 
 ::
 
     python3 -m pip install -U questdb[dataframe]
 
-Please start by `setting up QuestDB <https://questdb.io/docs/quick-start/>`_ . Once set up, you can use this library to insert data.
+
+The latest *pre-release* version of the library is **3.0.0r1** (`changelog <https://py-questdb-client.readthedocs.io/en/latest/changelog.html>`_).
+This release supports NumPy float64 arrays which are transmitted over a new
+protocol version supported by QuestDB 8.4.0 or later.
+
+:: 
+
+    python3 -m pip install --pre -U questdb[dataframe]
+
+Quickstart
+==========
+
+Start by `setting up QuestDB <https://questdb.io/docs/quick-start/>`_ .
+Once set up, you can use this library to insert data.
 
 The most common way to insert data is from a Pandas dataframe.
 
@@ -38,6 +51,13 @@ The most common way to insert data is from a Pandas dataframe.
         'side': pd.Categorical(['sell', 'sell']),
         'price': [2615.54, 39269.98],
         'amount': [0.00044, 0.001],
+
+        # NumPy float64 arrays are supported from v3.0.0rc1 onwards.
+        'ord_book_bids': [
+            np.array([2615.54, 2618.63]),
+            np.array([39269.98, 39270.00])
+        ],
+
         'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])})
 
     conf = f'http::addr=localhost:9000;'
@@ -57,7 +77,13 @@ You can also send individual rows. This only requires a more minimal installatio
         sender.row(
             'trades',
             symbols={'symbol': 'ETH-USD', 'side': 'sell'},
-            columns={'price': 2615.54, 'amount': 0.00044},
+            columns={
+                'price': 2615.54,
+                'amount': 0.00044,
+
+                # NumPy float64 arrays are supported from v3.0.0rc1 onwards.
+                'ord_book_bids': np.array([2615.54, 2618.63]),
+            },
             at=TimestampNanos.now())
         sender.flush()
 

RELEASING.rst

@@ -93,10 +93,9 @@ Inside the VM, open a terminal (or use the terminal Window in VSCode) and run th
 
     rustup update stable
 
-    /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 \
-        -m pip install -U pip
-    /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 \
-        -m pip install -U setuptools wheel twine Cython cibuildwheel pandas numpy pyarrow
+    python3 -m pip install -U pip
+    python3 -m pip install -U \
+        setuptools wheel twine Cython cibuildwheel pandas numpy pyarrow
 
 Smoke-testing the build
 -----------------------

c-questdb-client

@@ -52,7 +52,6 @@ stages:
       - job: linux_arm64
         pool:
           name: "arm64-clients"
-          vmImage:
           demands:
             - Agent.Name -equals arm64-clients-$(Build.BuildId)
         dependsOn:
@@ -92,23 +91,6 @@ stages:
           - task: PublishBuildArtifacts@1
             inputs: {pathtoPublish: 'wheelhouse'}
 
-      - job: linux_x64_cpython_manylinux_i686
-        pool: {vmImage: 'ubuntu-latest'}
-        timeoutInMinutes: 90
-        steps:
-          - task: UsePythonVersion@0
-          - bash: |
-              set -o errexit
-              python3 -m pip install --upgrade pip
-              python3 -m pip install cibuildwheel
-            displayName: Install dependencies
-          - bash: cibuildwheel --output-dir wheelhouse .
-            displayName: Build wheels
-            env:
-              CIBW_BUILD: cp*-manylinux_i686
-          - task: PublishBuildArtifacts@1
-            inputs: {pathtoPublish: 'wheelhouse'}
-
       - job: linux_x64_cpython_musllinux
         pool: {vmImage: 'ubuntu-latest'}
         timeoutInMinutes: 90
@@ -168,9 +150,22 @@ stages:
               python3 -m pip install --upgrade pip
               python3 -m pip install cibuildwheel
             displayName: Install dependencies
-          - bash: cibuildwheel --output-dir wheelhouse .
+          - powershell: |
+              $vsPath = Resolve-Path "C:\Program Files (x86)\Microsoft Visual Studio\2019\*\VC\Auxiliary\Build\vcvars32.bat"
+              cmd /c "call `"$vsPath`" && set > env_vars.txt"
+
+              Get-Content env_vars.txt | ForEach-Object {
+                if ($_ -match "^([^=]+?)=(.*)$" -and $matches[1] -notmatch '^(SYSTEM|AGENT|BUILD|RELEASE|VSTS|TASK|USE_|FAIL_|MSDEPLOY|AZP_75787|AZP_AGENT|AZP_ENABLE|AZURE_HTTP|COPYFILESOVERSSHV0|ENABLE_ISSUE_SOURCE_VALIDATION|MODIFY_NUMBER_OF_RETRIES_IN_ROBOCOPY|MSBUILDHELPERS_ENABLE_TELEMETRY|RETIRE_AZURERM_POWERSHELL_MODULE|ROSETTA2_WARNING|AZP_PS_ENABLE)') {
+                  [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2], "Process")
+                  Write-Host "##vso[task.setvariable variable=$($matches[1])]$($matches[2])"
+                }
+              }
+
+              where.exe cl.exe
+              cibuildwheel --output-dir wheelhouse .
             displayName: Build wheels
             env:
+              CIBW_ENVIRONMENT: "MSSdk=1 DISTUTILS_USE_SDK=1 SETUP_DO_GIT_SUBMODULE_INIT=1"
               CIBW_BUILD: "*win32*"
           - task: PublishBuildArtifacts@1
             inputs: {pathtoPublish: 'wheelhouse'}
@@ -185,9 +180,24 @@ stages:
               python3 -m pip install --upgrade pip
               python3 -m pip install cibuildwheel
             displayName: Install dependencies
-          - bash: cibuildwheel --output-dir wheelhouse .
+          - powershell: |
+              $vsPath = Resolve-Path "C:\Program Files (x86)\Microsoft Visual Studio\2019\*\VC\Auxiliary\Build\vcvars64.bat"
+              cmd /c "call `"$vsPath`" && set > env_vars.txt"
+
+              Get-Content env_vars.txt | ForEach-Object {
+                if ($_ -match "^([^=]+?)=(.*)$" -and $matches[1] -notmatch '^(SYSTEM|AGENT|BUILD|RELEASE|VSTS|TASK|USE_|FAIL_|MSDEPLOY|AZP_75787|AZP_AGENT|AZP_ENABLE|AZURE_HTTP|COPYFILESOVERSSHV0|ENABLE_ISSUE_SOURCE_VALIDATION|MODIFY_NUMBER_OF_RETRIES_IN_ROBOCOPY|MSBUILDHELPERS_ENABLE_TELEMETRY|RETIRE_AZURERM_POWERSHELL_MODULE|ROSETTA2_WARNING|AZP_PS_ENABLE)') {
+                  [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2], "Process")
+                  Write-Host "##vso[task.setvariable variable=$($matches[1])]$($matches[2])"
+                }
+              }
+
+              where.exe cl.exe
+              cibuildwheel --output-dir wheelhouse .
             displayName: Build wheels
             env:
               CIBW_BUILD: "*win_amd64*"
+              CIBW_ENVIRONMENT: "MSSdk=1 DISTUTILS_USE_SDK=1 SETUP_DO_GIT_SUBMODULE_INIT=1"
+              CIBW_BUILD_VERBOSITY: "3"
+              DISTUTILS_DEBUG: "1"
           - task: PublishBuildArtifacts@1
             inputs: {pathtoPublish: 'wheelhouse'}

ci/cibuildwheel.yaml

@@ -66,20 +66,19 @@ def install_old_pandas_and_numpy(args):
     try_pip_install('numpy<2')
 
 def install_new_pandas_and_numpy():
-    try_pip_install('pandas')
-    try_pip_install('numpy')
+    try_pip_install('pandas>2')
+    try_pip_install('numpy>2')
 
 def main(args):
     ensure_timezone()
     pip_install('pip')
     pip_install('setuptools')
-    try_pip_install('fastparquet>=2023.10.1')
-
     if args.pandas_version is not None and args.pandas_version != '':
         install_old_pandas_and_numpy(args)
     else:
         install_new_pandas_and_numpy()
 
+    try_pip_install('fastparquet>=2023.10.1')
     try_pip_install('pyarrow')
 
     on_linux_is_glibc = (

ci/pip_install_deps.py

@@ -9,7 +9,61 @@ questdb.ingress
 
     from questdb.ingress import *
 
-.. automodule:: questdb.ingress
+.. autoclass:: questdb.ingress.Sender
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: questdb.ingress.Buffer
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: questdb.ingress.SenderTransaction
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: questdb.ingress.IngressError
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: questdb.ingress.IngressErrorCode
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: questdb.ingress.Protocol
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: questdb.ingress.TimestampMicros
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: questdb.ingress.TimestampNanos
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: questdb.ingress.TlsCa
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: questdb.ingress.ServerTimestampType
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autodata:: questdb.ingress.ServerTimestamp
+    :annotation:
+    :no-value:
+
+.. autoclass:: questdb.ingress.TaggedEnum
    :members:
    :undoc-members:
    :show-inheritance:

docs/api.rst

@@ -3,6 +3,8 @@
 
 import os
 
+from questdb.ingress import *
+
 extensions = [
     'sphinx.ext.autodoc',
     'sphinx.ext.autosummary',
@@ -12,15 +14,16 @@
     'sphinx.ext.ifconfig',
     'sphinx.ext.napoleon',
     'sphinx.ext.todo',
-    'sphinx.ext.viewcode'
+    'sphinx.ext.viewcode',
+    "sphinx.ext.intersphinx",
 ]
 source_suffix = '.rst'
 master_doc = 'index'
 project = 'questdb'
 year = '2024'
 author = 'QuestDB'
 copyright = '{0}, {1}'.format(year, author)
-version = release = '2.0.4'
+version = release = '3.0.0rc1'
 
 github_repo_url = 'https://github.com/questdb/py-questdb-client'
 
@@ -67,6 +70,11 @@
     'undoc-members': True
 }
 
+intersphinx_mapping = {
+    "python": ("https://docs.python.org/3", None),
+    "numpy": ("https://numpy.org/doc/stable/", None),
+    "pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
+}
 
 # def do_not_skip_dunder_members(_app, _what, name, _obj, would_skip, _options):
 #     if name in ('__init__', '__call__', '__str__', '__enter__', '__exit__'):