Add DuckDB integration with example usage and tests #1193
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new integration with DuckDB, an in-process SQL OLAP database, to enhance support for analytical workloads and similarity search in Retrieval-Augmented Generation (RAG) workflows. The changes include the addition of a new module for DuckDB integration, an example script, tests, and updates to documentation and dependencies.
Related issue: #924
DuckDB Integration:
New DuckDB Integration Module:
DuckDBIntegration
class inllmware/duckdb_integration.py
, providing methods to initialize the database, create tables, insert data, query data, and close the connection.Example Script:
examples/Models/duckdb_rag_example.py
, demonstrating how to use theDuckDBIntegration
class for creating tables, inserting data, and querying a DuckDB database.Unit Tests:
tests/models/test_duckdb_integration.py
to validate the functionality of theDuckDBIntegration
class, including creating tables, inserting data, querying data, and closing the database connection.Documentation and Dependencies:
Documentation Update:
docs/components/model_catalog.md
to include a new section on DuckDB integration, detailing its features, use cases, and references to the example script and tests.Dependency Update:
duckdb>=0.7.1
tollmware/requirements.txt
to ensure the necessary library is available for the integration.