DuckDB Access Over HTTPS

Lets do a Deeper dive with an example from hugging face ??

The Hugging Face Hub is dedicated to providing open access to datasets for everyone and giving users the tools to explore and understand them. That is right open access to the dataset.

Note the /parquet in the above URL. We can discuss this in a bit.

Load remote data over HTTPS

import duckdb

url = "https://huggingface.co/datasets/blog_authorship_corpus/resolve/refs%2Fconvert%2Fparquet/blog_authorship_corpus/blog_authorship_corpus-train-00000-of-00002.parquet"

con = duckdb.connect()
con.execute("INSTALL httpfs;")
con.execute("LOAD httpfs;")
        

Above we setup a url from Hugging Face's Public URL instantiated duckdb . Setup the https connctivity

About the parquet url:

Parquet?is columnar making it easier to load, store and analyze.?This is important, especially when working with large datasets. We're seeing this more and more in the LLM age.?Datasets Server converts a dataset and publishes it on the Hub in Parquet format.?You can retrieve the URL of the Parquet file using the?/parquet?Endpoint.

e.g URL:

https://datasets-server.huggingface.co/parquet        

?You can locate some of the datasets used to training large language models (LLMs) like?Falcon,?Dolly,?MPT?on hugging face. Go on play and around.

Now lets run SQL Queries over it :

con.sql(f"SELECT horoscope
??? count(*),
??? AVG(LENGTH(text)) AS avg_blog_length
??? FROM '{url}'
??? GROUP BY horoscope
??? ORDER BY avg_blog_length
??? DESC LIMIT(5)"
)

        

Cool eh? We can now run a SQL over a URL! Never occurred to me this we will come to see this. Impressive !!

Credits shared with Steven Liu of Hugging Face fame!

DuckDB from AWS

Duck db with PyArrow

Polars

Apache Spark vs Polars