NEWS

duckdbfs 0.1.2.99

open_dataset(), spatial_join(), and write_dataset() now quote the SQL identifier for the view/table name, so table names that start with a digit (e.g. file stems like 000016.parquet) no longer produce a parser error (#21).
to_sf() now reads native GEOMETRY columns directly instead of forcing a ST_AsWKB round-trip, and auto-populates crs from the column's CRS annotation (e.g. GEOMETRY('EPSG:4326')) when the caller hasn't supplied one (#30, #18).
duckdb_config() docs gain examples for common resource settings (threads, memory_limit, temp_directory, max_temp_directory_size) (#44) and for HTTP retry/back-off (http_retries, http_retry_wait_ms) when a server returns HTTP 429 (#54).
New function raw_sql() provides an escape hatch for executing arbitrary SQL and getting back a lazy dplyr::tbl, useful for DuckDB-specific syntax such as UNION ALL BY NAME that dbplyr does not emit.
spatial_join() now casts its geometry columns to plain GEOMETRY, avoiding a binder error from newer DuckDB spatial extensions when the two inputs have different CRS type tags (e.g. EPSG:4326 vs OGC:CRS84).
All methods that write to a file / path now return that path (invisibly). Previously the return was just inherited from dbExecute() call, except for write_dataset() which always followed this convention. An additional optional argument has been added which can format the returned path as an HTTP address.

new function duckdb_config() streamlines common configurations, like duckdb_config(threads = 1, memory_limit = '10GB')
related helpers duckdb_get_config() shows any or all configuration settings, duckdb_reset() restores defaults.
new function duckdb_extensions() lists all available, installed, or loaded extensions and descriptions.
cached_connection() is aliased as duckdb_connect(), reflecting its use as more than an under-the-hood utility.
load_h3() and load_spatial() are called by default. Opt out by closing any active cached connection first (with close_connection()) and re-instantiating the with connect(with_h3 = FALSE) etc.
open_dataset() gains the argument parser_options to pass arbitrary options to parsers such as duckdb's read_csv(), see https://duckdb.org/docs/stable/data/csv/overview.html#parameters.
write_dataset() gains the argument options to support custom options controlling the COPY behavior writing to file, such as thread parallelism, file naming conventions, and more. see https://duckdb.org/docs/stable/sql/statements/copy.html#copy--to-options
S3-based access will no longer automatically try recursion if path ends in a recognized extension, .parquet, .csv, or .csv.gz

Adds to_h3j() method for streaming data to H3J format
Adds duckdb_secrets() as more modern credential manager
Adds write_geo() method, currently writes geojson #37
cached_connection() / connect() now supports config argument and sets a temporary directory to the R tempdir by default, allowing disk-backed storage when duckdb detects memory limits.

work-around for error The file was built for DuckDB version 'v1.1.3', but we can only load extensions built for DuckDB version '19864453f7'. by using nightly repo for extensions by default.

The default cached_connection() helper will configure a temporary storage location by default. It also now supports all options supported by duckdb::duckdb() for connection creation.
New as_dataset() utility copies a local in-memory data.frame into the connection.

bugfix: reading from local disk recursively no longer requires manual **. Also, trying to read from an existing local file won't try and append recursive search even when given the default recursive=TRUE option.
bugfix: open_dataset() uses random table name by default, avoid naming collisions.

bugfix write_dataset() no longer adds ** into paths when writing some partitions.
Protect from unsupported table names generated from file names that start with a digit, fixes #21.

open_dataset() gains the ability to read spatial vector data formats (objects read by sf) using format="sf"
default geometry column in to_sf() is now termed geom, to match the default used in duckdb's st_read() function.
open_dataset() now tries to guess the data format instead of defaulting to parquet when no format is explicitly provided.
a new function, spatial_join(), allows a variety of spatial joins.
a new function, st_read_meta(), exposes the spatial metadata of remote spatial objects.
new helper function, as_view(), creates a temporary view of a query.

duckdbfs now has spatial data query support! Users can leverage spatial data operations like st_distance() and st_area() and request return values as sf objects. Supports network-based access too. See README.md
Added write_dataset() which can write to (potentially partitioned) parquet to local directories or remote (S3) buckets.
The S3 interface supports arrow-compatible URI notation:
- Alternate endpoints can now be passed like so s3://userid:secret_token@bucket-name?endpoint_override=data.ecoforecast.org
- Users can omit the use of * (match any file) or ** (recursive search) and just supply a path. Recursive search is then assumed automatically. Note: unlike arrow, this still supports the use of globs (*) elsewhere in the path, e.g. s3://bucket/*/path
duckdb_s3_config gains argument anonymous allowing users to ignore existing AWS keys that may be set in environmental variables or AWS configuration files. This can also be passed as the username position in URI notation, e.g. s3://anonymous@bucket_name.
open_dataset drops use of endpoint as an argument. Instead, alternative S3 endpoints can be set either by using the URI query notation or calling duckdb_s3_config() first. Additionally, any arguments to duckdb_s3_config(), including s3_endpoint, can now be passed to open_dataset through the .... Note these settings will override any set by the URI notation.