Table of Contents

Class S3ConnectorConfiguration

Namespace
Datafication.Connectors.S3Connector
Assembly
Datafication.S3Connector.dll

Represents a configuration class for the S3 connector.

public class S3ConnectorConfiguration : IDataConnectorConfiguration
Inheritance
object
S3ConnectorConfiguration
Implements

Remarks

The configuration provides the necessary properties required for reading data from an S3 source. Supports both authenticated and anonymous (public bucket) access.

Constructors

S3ConnectorConfiguration() 

public S3ConnectorConfiguration()

Properties

AccessKeyId

Gets or sets the AWS access key ID for authentication.

public string? AccessKeyId { get; set; }

Property Value

string

Remarks

Optional. If null, the connector will attempt anonymous access for public buckets. Required for accessing private S3 buckets.

AllowMultipleSegments

Gets or sets a value indicating whether to allow reading multiple file segments from S3.

public bool AllowMultipleSegments { get; set; }

Property Value

bool

Remarks

When false (default), ObjectKey must reference a single file.

When true, ObjectKey can be a prefix pattern to read multiple files (e.g., "data/partitioned/").

Multi-segment mode only works with GetStorageDataAsync() to prevent memory issues with large datasets.

All segments must be of the same file type (CSV, Parquet, JSON, or Excel).

BucketName

Gets or sets the S3 bucket name.

public string BucketName { get; set; }

Property Value

string

Remarks

The name of the S3 bucket containing the data to be accessed. Must be a valid S3 bucket name.

CsvHeaderRow

Gets or sets a value indicating whether the CSV file has a header row.

public bool? CsvHeaderRow { get; set; }

Property Value

bool?

Remarks

Optional. If null, defaults to true (assumes header row exists). Set to false if the CSV file does not contain a header row. Only applies when reading CSV files.

CsvSeparator

Gets or sets the CSV separator character.

public char? CsvSeparator { get; set; }

Property Value

char?

Remarks

Optional. If null, the CSV connector will use its default separator (comma). Common values: ',' (comma), ';' (semicolon), '\t' (tab), '|' (pipe). Only applies when reading CSV files.

ErrorHandler

Gets or sets the error handler for managing exceptions.

public Action<Exception>? ErrorHandler { get; set; }

Property Value

Action<Exception>

Remarks

This provides a means to globally handle exceptions that may occur during S3 data processing without having to handle them individually on each call. If the handler is not set, exceptions will propagate as usual.

ExcelHasHeader

Gets or sets a value indicating whether the Excel file has a header row.

public bool? ExcelHasHeader { get; set; }

Property Value

bool?

Remarks

Optional. If null, defaults to true (assumes header row exists). Set to false if the Excel worksheet does not contain a header row. Only applies when reading Excel files.

ExcelSheetIndex

Gets or sets the Excel worksheet index to read.

public int? ExcelSheetIndex { get; set; }

Property Value

int?

Remarks

Optional. If null, defaults to 0 (first worksheet). Specify the zero-based index of the worksheet to read (0 = first sheet, 1 = second sheet, etc.). Only applies when reading Excel files.

ForcePathStyle

Gets or sets a value indicating whether to use path-style URLs.

public bool ForcePathStyle { get; set; }

Property Value

bool

Remarks

When true, uses path-style URLs (http://s3.amazonaws.com/bucket/key). When false, uses virtual-hosted-style URLs (http://bucket.s3.amazonaws.com/key). Some S3-compatible services require path-style URLs.

Id

Gets or sets the unique identifier for this configuration.

public string Id { get; set; }

Property Value

string

Remarks

This identifier can be useful for distinguishing between multiple connector configurations, if needed.

ObjectKey

Gets or sets the S3 object key or prefix.

public string ObjectKey { get; set; }

Property Value

string

Remarks

For single file mode: Specify the exact object key (e.g., "data/myfile.csv").

For multi-segment mode: Specify a prefix pattern (e.g., "data/partitioned/" or "logs/2024-*").

Multi-segment mode requires AllowMultipleSegments to be true and only works with GetStorageDataAsync().

Region

Gets or sets the AWS region for the S3 bucket.

public string Region { get; set; }

Property Value

string

Remarks

This represents the AWS region where the S3 bucket is located (e.g., "us-east-1", "eu-west-1"). Required for all S3 operations.

SecretAccessKey

Gets or sets the AWS secret access key for authentication.

public string? SecretAccessKey { get; set; }

Property Value

string

Remarks

Optional. If null, the connector will attempt anonymous access for public buckets. Required for accessing private S3 buckets. Must be provided together with AccessKeyId.

ServiceUrl

Gets or sets a custom service URL for S3-compatible services.

public string? ServiceUrl { get; set; }

Property Value

string

Remarks

Optional. Used for S3-compatible services like MinIO, DigitalOcean Spaces, etc. If null, uses the default AWS S3 endpoint for the specified region. Example: "https://s3.wasabisys.com" for Wasabi.

SessionToken

Gets or sets the AWS session token for temporary credentials.

public string? SessionToken { get; set; }

Property Value

string

Remarks

Optional. Used when working with temporary AWS credentials (STS tokens). Only needed when using temporary credentials from AWS STS.