stac_geoparquet.arrow¶
Arrow-based format conversions.
stac_geoparquet.arrow ¶
DEFAULT_JSON_CHUNK_SIZE
module-attribute
¶
DEFAULT_JSON_CHUNK_SIZE = 65536
The default chunk size to use for reading JSON into memory.
DEFAULT_PARQUET_SCHEMA_VERSION
module-attribute
¶
DEFAULT_PARQUET_SCHEMA_VERSION: SUPPORTED_PARQUET_SCHEMA_VERSIONS = '1.1.0'
The default GeoParquet schema version written to file.
SUPPORTED_PARQUET_SCHEMA_VERSIONS
module-attribute
¶
SUPPORTED_PARQUET_SCHEMA_VERSIONS = Literal['1.0.0', '1.1.0']
A Literal type with the supported GeoParquet schema versions.
InferredSchema ¶
A schema representing the original STAC JSON with absolutely minimal modifications.
The only modifications from the data are converting any geometry fields from GeoJSON to WKB, and typing the bbox as a fixed-size list of 4 or 6 values.
inner
instance-attribute
¶
The underlying Arrow schema.
from_geoparquet
classmethod
¶
from_geoparquet(schema: Schema) -> InferredSchema
Make the STAC JSON schema of items from the schema of a GeoParquet file or table, for example to write more items to it.
to_geoparquet ¶
The schema of these items as written to GeoParquet, with their properties as top-level columns.
update_from_items ¶
Update this inferred schema from STAC Items.
update_from_json ¶
update_from_json(
path: str | Path | Iterable[str | Path],
*,
chunk_size: int | None = None,
limit: int | None = None,
) -> None
Update this inferred schema from one or more newline-delimited JSON STAC files.
Parameters:
-
path(str | Path | Iterable[str | Path]) –One or more paths to files with STAC items.
-
chunk_size(int | None, default:None) –The chunk size to load into memory at a time. Defaults to the
STAC_GEOPARQUET_CHUNK_SIZEenvironment variable, or 65536.
Other Parameters:
-
limit(int | None) –The maximum number of JSON Items to use for schema inference
parse_stac_items_to_arrow ¶
parse_stac_items_to_arrow(
items: Iterable[Item | dict[str, Any]],
chunk_size: int | None = None,
schema: ACCEPTED_SCHEMA_OPTIONS = "FullFile",
tmpdir: str | Path | None = None,
drop_invalid_properties: bool = True,
) -> RecordBatchReader
Parse a collection of STAC Items to an iterable of
pyarrow.RecordBatch.
The objects under properties are moved up to the top-level of the
Table, similar to
geopandas.GeoDataFrame.from_features.
Parameters:
-
items(Iterable[Item | dict[str, Any]]) –the STAC Items to convert. Geometries can also be WKB or EWKB bytes.
-
chunk_size(int | None, default:None) –The chunk size to use for Arrow record batches. It has no effect when
schemais "FullFile", which parses the input into a single record batch. A chunk can end early if it wouldn't fit in the memory available, unless theSTAC_GEOPARQUET_MEMORY_CAPenvironment variable is false. Defaults to theSTAC_GEOPARQUET_CHUNK_SIZEenvironment variable, or 65536. -
schema(ACCEPTED_SCHEMA_OPTIONS, default:'FullFile') –The schema of the input data. If provided, can improve memory use; otherwise all items need to be parsed into a single array for schema inference. This can also be set to a string value of "FullFile" which will scan the entire input in memory to get the schema, "FirstBatch" which will use the first batch of items to infer the schema, or "ChunksToDisk" which will write each chunk of items to disk as an Arrow IPC file and then read them back in to unify the schema. Defaults to "FullFile". A
pa.Schema's bbox type sets whether bboxes are 2d or 3d: a fixed-size list of 6 values is 3d, and any other list is 2d. -
drop_invalid_properties(bool, default:True) –If True (default), an Item property whose name collides with a top-level field. If False, a ValueError is raised instead.
Returns:
-
RecordBatchReader–pyarrow RecordBatchReader with a stream of STAC Arrow RecordBatches.
parse_stac_items_to_parquet ¶
parse_stac_items_to_parquet(
items: Iterable[Item | dict[str, Any]],
*,
chunk_size: int | None = None,
schema: ACCEPTED_SCHEMA_OPTIONS = "FirstBatch",
output_path: str | Path,
tmpdir: str | Path | None = None,
schema_version: SUPPORTED_PARQUET_SCHEMA_VERSIONS = DEFAULT_PARQUET_SCHEMA_VERSION,
filesystem: FileSystem | None = None,
drop_invalid_properties: bool = True,
**kwargs: Any,
) -> str
Parse an iterable of Stac Items into Parquet.
parse_stac_ndjson_to_arrow ¶
parse_stac_ndjson_to_arrow(
path: str | Path | Iterable[str | Path],
*,
chunk_size: int | None = None,
schema: Schema | None = None,
limit: int | None = None,
drop_invalid_properties: bool = True,
) -> RecordBatchReader
Convert one or more newline-delimited JSON STAC files to a generator of Arrow RecordBatches.
Each RecordBatch in the returned iterator is guaranteed to have an identical schema, and can be used to write to one or more Parquet files.
Parameters:
-
path(str | Path | Iterable[str | Path]) –One or more paths to files with STAC items.
-
chunk_size(int | None, default:None) –The chunk size. Defaults to the
STAC_GEOPARQUET_CHUNK_SIZEenvironment variable, or 65536. -
schema(Schema | None, default:None) –The schema to represent the input STAC data. Defaults to None, in which case the schema will first be inferred via a full pass over the input data. In this case, there will be two full passes over the input data: one to infer a common schema across all data and another to read the data. The schema's bbox type sets whether bboxes are 2d or 3d: a fixed-size list of 6 values is 3d, and any other list is 2d.
Other Parameters:
-
limit(int | None) –The maximum number of JSON Items to use for schema inference
-
drop_invalid_properties(bool) –If True (default), an Item property whose name collides with a top-level field. If False, a ValueError is raised instead.
Returns:
-
RecordBatchReader–pyarrow RecordBatchReader with a stream of STAC Arrow RecordBatches.
parse_stac_ndjson_to_delta_lake ¶
parse_stac_ndjson_to_delta_lake(
input_path: str | Path | Iterable[str | Path],
table_or_uri: str | Path | DeltaTable,
*,
chunk_size: int | None = None,
schema: Schema | None = None,
limit: int | None = None,
schema_version: SUPPORTED_PARQUET_SCHEMA_VERSIONS = DEFAULT_PARQUET_SCHEMA_VERSION,
**kwargs: Any,
) -> None
Convert one or more newline-delimited JSON STAC files to Delta Lake
Parameters:
-
input_path(str | Path | Iterable[str | Path]) –One or more paths to files with STAC items.
-
table_or_uri(str | Path | DeltaTable) –A path to the output Delta Lake table
Parameters:
-
chunk_size(int | None, default:None) –The chunk size to use for reading JSON into memory. Defaults to the
STAC_GEOPARQUET_CHUNK_SIZEenvironment variable, or 65536. -
schema(Schema | None, default:None) –The schema to represent the input STAC data. Defaults to None, in which case the schema will first be inferred via a full pass over the input data. In this case, there will be two full passes over the input data: one to infer a common schema across all data and another to read the data and iteratively convert to GeoParquet.
-
limit(int | None, default:None) –The maximum number of JSON records to convert.
-
schema_version(SUPPORTED_PARQUET_SCHEMA_VERSIONS, default:DEFAULT_PARQUET_SCHEMA_VERSION) –GeoParquet specification version; if not provided will default to latest supported version.
parse_stac_ndjson_to_parquet ¶
parse_stac_ndjson_to_parquet(
input_path: str | Path | Iterable[str | Path],
output_path: str | Path,
*,
chunk_size: int | None = None,
schema: Schema | InferredSchema | None = None,
limit: int | None = None,
schema_version: SUPPORTED_PARQUET_SCHEMA_VERSIONS = DEFAULT_PARQUET_SCHEMA_VERSION,
collections: Mapping[str, Mapping[str, Any]] | None = None,
collection_metadata: Mapping[str, Any] | None = None,
filesystem: FileSystem | None = None,
drop_invalid_properties: bool = True,
**kwargs: Any,
) -> None
Convert one or more newline-delimited JSON STAC files to GeoParquet
Parameters:
-
input_path(str | Path | Iterable[str | Path]) –One or more paths to files with STAC items.
-
output_path(str | Path) –A path to the output Parquet file.
Other Parameters:
-
chunk_size(int | None) –The chunk size. Defaults to the
STAC_GEOPARQUET_CHUNK_SIZEenvironment variable, or 65536. -
schema(Schema | InferredSchema | None) –The schema to represent the input STAC data. Defaults to None, in which case the schema will first be inferred via a full pass over the input data. In this case, there will be two full passes over the input data: one to infer a common schema across all data and another to read the data and iteratively convert to GeoParquet.
-
limit(int | None) –The maximum number of JSON records to convert.
-
schema_version(SUPPORTED_PARQUET_SCHEMA_VERSIONS) –GeoParquet specification version; if not provided will default to latest supported version.
-
collections(Mapping[str, Mapping[str, Any]] | None) –A dictionary mapping collection IDs to dictionaries representing a Collection in a SpatioTemporal Asset Catalog. This will be stored under the key
stac-geoparquetin the parquet file metadata, under the keycollections. -
collection_metadata(Mapping[str, Any] | None) –A dictionary representing a Collection in a SpatioTemporal Asset Catalog. This will be stored under the key
stac-geoparquetin the parquet file metadata, under the keycollection.Deprecated in favor of
collections. -
filesystem(FileSystem | None) –PyArrow FileSystem to use for writing. If not provided, will be inferred from output_path for local files.
-
drop_invalid_properties(bool) –If True (default), an Item property whose name collides with a top-level field. If False, a ValueError is raised instead.
All other keyword args are passed on to
pyarrow.parquet.ParquetWriter.
stac_table_to_items ¶
stac_table_to_items(
table: Table | RecordBatchReader | ArrowStreamExportable,
) -> Iterable[dict]
Convert STAC Arrow to a generator of STAC Item dicts.
Parameters:
-
table(Table | RecordBatchReader | ArrowStreamExportable) –STAC in Arrow form. This can be a pyarrow Table, a pyarrow RecordBatchReader, or any other Arrow stream object exposed through the Arrow PyCapsule Interface. A RecordBatchReader or stream object will not be materialized in memory.
Yields:
stac_table_to_ndjson ¶
stac_table_to_ndjson(
table: Table | RecordBatchReader | ArrowStreamExportable,
dest: str | Path | PathLike[bytes],
) -> None
Write STAC Arrow to a newline-delimited JSON file.
Note
This function appends to the JSON file at dest; it does not overwrite any
existing data.
Parameters:
-
table(Table | RecordBatchReader | ArrowStreamExportable) –STAC in Arrow form. This can be a pyarrow Table, a pyarrow RecordBatchReader, or any other Arrow stream object exposed through the Arrow PyCapsule Interface. A RecordBatchReader or stream object will not be materialized in memory.
The 'type' field is not required in stac-geoparquet. If not present, a 'type' field will be added where each record is 'Feature'.
-
dest(str | Path | PathLike[bytes]) –The destination where newline-delimited JSON should be written.
to_parquet ¶
to_parquet(
table: Table | RecordBatchReader | ArrowStreamExportable,
output_path: str | Path,
*,
schema_version: SUPPORTED_PARQUET_SCHEMA_VERSIONS = DEFAULT_PARQUET_SCHEMA_VERSION,
collections: Mapping[str, Mapping[str, Any]] | None = None,
collection_metadata: Mapping[str, Any] | None = None,
filesystem: FileSystem | None = None,
**kwargs: Any,
) -> None
Write an Arrow table with STAC data to GeoParquet
This writes metadata compliant with either GeoParquet 1.0 or 1.1.
Parameters:
-
table(Table | RecordBatchReader | ArrowStreamExportable) –STAC in Arrow form. This can be a pyarrow Table, a pyarrow RecordBatchReader, or any other Arrow stream object exposed through the Arrow PyCapsule Interface. A RecordBatchReader or stream object will not be materialized in memory.
-
output_path(str | Path) –The destination for saving.
Other Parameters:
-
schema_version(SUPPORTED_PARQUET_SCHEMA_VERSIONS) –GeoParquet specification version; if not provided will default to latest supported version.
-
collections(Mapping[str, Mapping[str, Any]] | None) –A dictionary mapping collection IDs to dictionaries representing a Collection in a SpatioTemporal Asset Catalog. This will be stored under the key
stac-geoparquetin the parquet file metadata, under the keycollections. -
collection_metadata(Mapping[str, Any] | None) –A dictionary representing a Collection in a SpatioTemporal Asset Catalog. This will be stored under the key
stac-geoparquetin the parquet file metadata, under the keycollection.Deprecated in favor of
collections. -
filesystem(FileSystem | None) –PyArrow FileSystem to use for writing. If not provided, will be inferred from output_path for local files.
All other keyword args are passed on to
pyarrow.parquet.ParquetWriter.