Class JsonDataConnector

Namespace

Datafication.Connectors.JsonConnector

Assembly

Datafication.JsonConnector.dll

Provides functionality to connect to and retrieve data from a JSON data source.

public class JsonDataConnector : IDataConnector

Inheritance

object

JsonDataConnector

Implements

IDataConnector

Extension Methods

DataConnectorExtensions.AsStorageConnector(IDataConnector)

DataConnectorExtensions.GetDataBatchesAsync(IDataConnector, int)

DataConnectorExtensions.GetSourceInfoAsync(IDataConnector)

Remarks

This connector utilizes the provided configuration to establish a connection to the JSON data source, either local or remote, and then extracts data records from the retrieved JSON.

Constructors

JsonDataConnector(JsonConnectorConfiguration)

Initializes a new instance of the JsonDataConnector class.

public JsonDataConnector(JsonConnectorConfiguration configuration)

Parameters

configuration JsonConnectorConfiguration

The configuration to be used by the connector.

Remarks

This constructor ensures that the JSON data connector is properly configured before any operations are performed.

Exceptions

ArgumentNullException

Thrown if the provided configuration is null.

ArgumentException

Thrown when the configuration validation fails.

Properties

Configuration

Gets the configuration used by this JSON data connector.

public JsonConnectorConfiguration Configuration { get; }

Property Value

JsonConnectorConfiguration

Methods

GetConnectorId()

Retrieves the unique identifier associated with this connector's configuration.

public string GetConnectorId()

Returns

string

The unique identifier of the configuration.

GetDataAsync()

Asynchronously retrieves data from the JSON source specified in the configuration.

public Task<DataBlock> GetDataAsync()

Returns

Task<DataBlock>

A task that represents the asynchronous operation. The task result contains a DataBlock with the JSON data.

Remarks

This method reads the JSON data from the source specified in the configuration and converts it into a DataBlock. The JSON data should be in a format that can be converted to a tabular structure (array of objects).

Exceptions

Exception

Thrown when there is an error reading the JSON source or when the configuration is invalid.

GetStorageDataAsync(IStorageDataBlock, int)

Asynchronously retrieves data from the JSON source and appends it in batches to the provided storage data block.

public Task<IStorageDataBlock> GetStorageDataAsync(IStorageDataBlock target, int batchSize = 10000)

Parameters

target IStorageDataBlock

The storage data block to append batches to.

batchSize int

The maximum number of rows to accumulate before appending a batch. Defaults to 10000.

Returns

Task<IStorageDataBlock>

A task that represents the asynchronous operation. The task result contains the provided IStorageDataBlock with all data appended.

Remarks

This method mirrors the exact data processing logic from GetDataAsync() but streams data in batches to the provided storage block. The JSON data should be in a format that can be converted to a tabular structure (array of objects).

Exceptions

ArgumentNullException

Thrown when target is null.

Exception

Thrown when there is an error reading the JSON source or when the configuration is invalid.