Interface IStorageConnector

Namespace

Datafication.Storage.Velocity.Interfaces

Assembly

Datafication.Storage.Velocity.dll

Enhanced data connector interface that supports storage-aware operations and streaming for large datasets.

public interface IStorageConnector : IDataConnector

Inherited Members

IDataConnector.GetDataAsync()

IDataConnector.GetStorageDataAsync(IStorageDataBlock, int)

IDataConnector.GetConnectorId()

Extension Methods

DataConnectorExtensions.AsStorageConnector(IDataConnector)

DataConnectorExtensions.GetDataBatchesAsync(IDataConnector, int)

DataConnectorExtensions.GetSourceInfoAsync(IDataConnector)

DataConnectorExtensions.MaterializeAsync(IStorageConnector, int)

Methods

GetDataBatchesAsync(int)

Retrieves data in batches for memory-efficient processing of large datasets. This is the preferred method for storage operations.

IAsyncEnumerable<DataBlock> GetDataBatchesAsync(int batchSize = 10000)

Parameters

batchSize int

The number of rows to include in each batch.

Returns

IAsyncEnumerable<DataBlock>

An async enumerable of DataBlock batches.

GetFilteredBatchesAsync(DataFilter, int)

Retrieves filtered data in batches, allowing for query pushdown optimizations.

IAsyncEnumerable<DataBlock> GetFilteredBatchesAsync(DataFilter filter, int batchSize = 10000)

Parameters

filter DataFilter

The filter to apply to the data.

batchSize int

The number of rows to include in each batch.

Returns

IAsyncEnumerable<DataBlock>

An async enumerable of filtered DataBlock batches.

GetSourceInfoAsync()

Gets metadata about the data source without loading the actual data. This enables efficient tier selection and resource planning.

Task<DataSourceInfo> GetSourceInfoAsync()

Returns

Task<DataSourceInfo>

Information about the data source including row count estimates and schema.