AWS S3

rustac.store.S3Store

Interface to an Amazon S3 bucket.

All constructors will check for environment variables. All environment variables starting with AWS_ will be evaluated. Names must match keys from [S3ConfigInput][obstore.store.S3ConfigInput]. Only upper-case environment variables are accepted.

Some examples of variables extracted from environment:

AWS_ACCESS_KEY_ID -> access_key_id
AWS_SECRET_ACCESS_KEY -> secret_access_key
AWS_DEFAULT_REGION -> region
AWS_ENDPOINT -> endpoint
AWS_SESSION_TOKEN -> token
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI -> https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html
AWS_REQUEST_PAYER -> set to "true" to permit requester-pays connections.

Examples:

Using requester-pays buckets:

Pass request_payer=True as a keyword argument or have AWS_REQUESTER_PAYS=True set in the environment.

Anonymous requests:

Pass skip_signature=True as a keyword argument or have AWS_SKIP_SIGNATURE=True set in the environment.

client_options `property`

client_options: ClientConfig | None

Get the store's client configuration.

config `property`

config: S3Config

Get the underlying S3 config parameters.

prefix `property`

prefix: str | None

Get the prefix applied to all operations in this store, if any.

retry_config `property`

retry_config: RetryConfig | None

Get the store's retry configuration.

init

__init__(
    bucket: str | None = None,
    *,
    prefix: str | None = None,
    config: S3Config | S3ConfigInput | None = None,
    client_options: ClientConfig | None = None,
    retry_config: RetryConfig | None = None,
    credential_provider: S3CredentialProvider | None = None,
    **kwargs: Unpack[S3ConfigInput],
) -> None

Create a new S3Store.

Parameters:

bucket (str | None, default: None ) –

The AWS bucket to use.

Other Parameters:

prefix (str | None) –

A prefix within the bucket to use for all operations.
config (S3Config | S3ConfigInput | None) –

AWS configuration. Values in this config will override values inferred from the environment. Defaults to None.
client_options (ClientConfig | None) –

HTTP Client options. Defaults to None.
retry_config (RetryConfig | None) –

Retry configuration. Defaults to None.
credential_provider (S3CredentialProvider | None) –

A callback to provide custom S3 credentials.
kwargs (Unpack[S3ConfigInput]) –

AWS configuration values. Supports the same values as config, but as named keyword args.

Returns:

None –

S3Store

from_url `classmethod`

from_url(
    url: str,
    *,
    config: S3Config | S3ConfigInput | None = None,
    client_options: ClientConfig | None = None,
    retry_config: RetryConfig | None = None,
    credential_provider: S3CredentialProvider | None = None,
    **kwargs: Unpack[S3ConfigInput],
) -> S3Store

Parse available connection info from a well-known storage URL.

The supported url schemes are:

s3://<bucket>/<path>
s3a://<bucket>/<path>
https://s3.<region>.amazonaws.com/<bucket>
https://<bucket>.s3.<region>.amazonaws.com
https://ACCOUNT_ID.r2.cloudflarestorage.com/bucket

Parameters:

url (str) –

well-known storage URL.

Other Parameters:

config (S3Config | S3ConfigInput | None) –

AWS Configuration. Values in this config will override values inferred from the url. Defaults to None.
client_options (ClientConfig | None) –

HTTP Client options. Defaults to None.
retry_config (RetryConfig | None) –

Retry configuration. Defaults to None.
credential_provider (S3CredentialProvider | None) –

A callback to provide custom S3 credentials.
kwargs (Unpack[S3ConfigInput]) –

AWS configuration values. Supports the same values as config, but as named keyword args.

Returns:

S3Store –

S3Store

rustac.store.S3Config

Bases: TypedDict

Configuration parameters returned from S3Store.config.

Note that this is a strict subset of the keys allowed for input into the store, see [S3ConfigInput][obstore.store.S3ConfigInput].

aws_access_key_id `instance-attribute`

aws_access_key_id: str

AWS Access Key

aws_bucket `instance-attribute`

aws_bucket: str

Bucket name

aws_checksum_algorithm `instance-attribute`

aws_checksum_algorithm: str

Sets the checksum algorithm which has to be used for object integrity check during upload.

aws_conditional_put `instance-attribute`

aws_conditional_put: str

See [S3ConfigInput.aws_conditional_put][obstore.store.S3ConfigInput.aws_conditional_put].

aws_container_credentials_relative_uri `instance-attribute`

aws_container_credentials_relative_uri: str

See [S3ConfigInput.aws_container_credentials_relative_uri][obstore.store.S3ConfigInput.aws_container_credentials_relative_uri].

aws_copy_if_not_exists `instance-attribute`

aws_copy_if_not_exists: str

See [S3ConfigInput.aws_copy_if_not_exists][obstore.store.S3ConfigInput.aws_copy_if_not_exists].

aws_default_region `instance-attribute`

aws_default_region: str

Default region

aws_disable_tagging `instance-attribute`

aws_disable_tagging: bool

Disable tagging objects. This can be desirable if not supported by the backing store.

aws_endpoint `instance-attribute`

aws_endpoint: str

Sets custom endpoint for communicating with AWS S3.

aws_imdsv1_fallback `instance-attribute`

aws_imdsv1_fallback: str

Fall back to ImdsV1

aws_metadata_endpoint `instance-attribute`

aws_metadata_endpoint: str

Set the instance metadata endpoint

aws_region `instance-attribute`

aws_region: str

Region

aws_request_payer `instance-attribute`

aws_request_payer: bool

If True, enable operations on requester-pays buckets.

aws_s3_express `instance-attribute`

aws_s3_express: bool

Enable Support for S3 Express One Zone

aws_secret_access_key `instance-attribute`

aws_secret_access_key: str

Secret Access Key

aws_server_side_encryption `instance-attribute`

aws_server_side_encryption: str

See [S3ConfigInput.aws_server_side_encryption][obstore.store.S3ConfigInput.aws_server_side_encryption].

aws_session_token `instance-attribute`

aws_session_token: str

Token to use for requests (passed to underlying provider)

aws_skip_signature `instance-attribute`

aws_skip_signature: bool

If True, S3Store will not fetch credentials and will not sign requests.

aws_sse_bucket_key_enabled `instance-attribute`

aws_sse_bucket_key_enabled: bool

If set to True, will use the bucket's default KMS key for server-side encryption. If set to False, will disable the use of the bucket's default KMS key for server-side encryption.

aws_sse_customer_key_base64 `instance-attribute`

aws_sse_customer_key_base64: str

The base64 encoded, 256-bit customer encryption key to use for server-side encryption. If set, the server side encryption config value must be "sse-c".

aws_sse_kms_key_id `instance-attribute`

aws_sse_kms_key_id: str

The KMS key ID to use for server-side encryption.

If set, the server side encryption config value must be "aws:kms" or "aws:kms:dsse".

aws_token `instance-attribute`

aws_token: str

Token to use for requests (passed to underlying provider)

aws_unsigned_payload `instance-attribute`

aws_unsigned_payload: bool

Avoid computing payload checksum when calculating signature.

aws_virtual_hosted_style_request `instance-attribute`

aws_virtual_hosted_style_request: bool

If virtual hosted style request has to be used.

rustac.store.S3Credential

Bases: TypedDict

An S3 credential.

access_key_id `instance-attribute`

access_key_id: str

AWS access key ID.

expires_at `instance-attribute`

expires_at: datetime | None

Expiry datetime of credential. The datetime should have time zone set.

If None, the credential will never expire.

secret_access_key `instance-attribute`

secret_access_key: str

AWS secret access key

token `instance-attribute`

token: NotRequired[str | None]

AWS token.

rustac.store.S3CredentialProvider

Bases: Protocol

A type hint for a synchronous or asynchronous callback to provide custom S3 credentials.

This should be passed into the credential_provider parameter of S3Store.

Examples:

Return static credentials that don't expire:

def get_credentials() -> S3Credential:
    return {
        "access_key_id": "...",
        "secret_access_key": "...",
        "token": None,
        "expires_at": None,
    }

Return static credentials that are valid for 5 minutes:

from datetime import datetime, timedelta, UTC

async def get_credentials() -> S3Credential:
    return {
        "access_key_id": "...",
        "secret_access_key": "...",
        "token": None,
        "expires_at": datetime.now(UTC) + timedelta(minutes=5),
    }

A class-based credential provider with state:

from __future__ import annotations

from typing import TYPE_CHECKING

import boto3
import botocore.credentials

if TYPE_CHECKING:
    from obstore.store import S3Credential


class Boto3CredentialProvider:
    credentials: botocore.credentials.Credentials

    def __init__(self, session: boto3.session.Session) -> None:
        credentials = session.get_credentials()
        if credentials is None:
            raise ValueError("Received None from session.get_credentials")

        self.credentials = credentials

    def __call__(self) -> S3Credential:
        frozen_credentials = self.credentials.get_frozen_credentials()
        return {
            "access_key_id": frozen_credentials.access_key,
            "secret_access_key": frozen_credentials.secret_key,
            "token": frozen_credentials.token,
            "expires_at": None,
        }

call `staticmethod`

__call__() -> S3Credential | Coroutine[Any, Any, S3Credential]

Return an S3Credential.

AWS S3

rustac.store.S3Store

client_options property

config property

prefix property

retry_config property

__init__

from_url classmethod

rustac.store.S3Config

aws_access_key_id instance-attribute

aws_bucket instance-attribute

aws_checksum_algorithm instance-attribute

aws_conditional_put instance-attribute

aws_container_credentials_relative_uri instance-attribute

aws_copy_if_not_exists instance-attribute

aws_default_region instance-attribute

aws_disable_tagging instance-attribute

aws_endpoint instance-attribute

aws_imdsv1_fallback instance-attribute

aws_metadata_endpoint instance-attribute

aws_region instance-attribute

aws_request_payer instance-attribute

aws_s3_express instance-attribute

aws_secret_access_key instance-attribute

aws_server_side_encryption instance-attribute

aws_session_token instance-attribute

aws_skip_signature instance-attribute

aws_sse_bucket_key_enabled instance-attribute

aws_sse_customer_key_base64 instance-attribute

aws_sse_kms_key_id instance-attribute

aws_token instance-attribute

aws_unsigned_payload instance-attribute

aws_virtual_hosted_style_request instance-attribute