Apache Spark Connector for Lance

The Apache Spark Connector for Lance allows Apache Spark to efficiently read datasets stored in Lance format. By using the Apache Spark Connector for Lance, you can leverage Apache Spark's powerful data processing, SQL querying, and machine learning training capabilities on the AI data lake powered by Lance.

Features

The connector is built using the Spark DatasourceV2 (DSv2) API. Please check this presentation to learn more about DSv2 features. Specifically, you can use the Apache Spark Connector for Lance to:

• Read & Write Lance Datasets: Seamlessly read and write datasets stored in the Lance format using Spark.
• Distributed, Parallel Scans: Leverage Spark's distributed computing capabilities to perform parallel scans on Lance datasets.
• Column and Filter Pushdown: Optimize query performance by pushing down column selections and filters to the data source.

Quick Start

The project contains a docker image in the docker folder you can build and run a simple example notebook. To do so, clone the repo and run:

make docker-build
make docker-up

And then open the notebook at http://localhost:8888.

Contributing

See contributing for the detailed contribution guidelines and local development setup.