Flat File (Native)

The Flat File (Native) connector enables querying and transformation of flat file data using familiar SQL syntax. This allows seamless integration of flat file data into SQL-based workflows for data manipulation and analysis.

Prerequisites

Before connecting, ensure the following prerequisites are met:

Verify that the storage or folder location is accessible from the cluster.
Gather valid user credentials and ensure the user has appropriate permissions to read the files.
Ensure at least one file is present in the folder.

Connecting to Amazon S3

AWS Custom Credential Authentication

Use one or more properties from the table below to create a valid connection. Properties marked with an asterisk (*) are required.

Name	Description	Example Values
Connection Name *	A unique name that identifies the connection.	prod_sales_data
File Type *	The type or format of the file.	Text
Container *	The name of the S3 bucket that contains the flat files.	prod-bucket
Folder Path *	The directory path within the bucket that points to the folder containing the flat files.	/data/prod-files/
Region *	AWS region where the S3 bucket is located.	us-east-1
Type *	The connection type, either System Connection or User Connection. Refer Connections for more details.	System
Access Key *	The key used to authenticate the connection to the storage account.	AKIAIOSFODNN7EXAMPLE
Secret Key *	The secret key used to authenticate the connection to the storage account.	wJalrXUtnFEPLEKEY
Session Token	Temporary token used in case of session-based authentication.	FQoJ//wEaUExampleToken==

warning

The Access and Secret keys must be updated whenever the session token expires.

AWS EC2 Role Authentication

Name	Description	Example Values
Connection Name *	A unique name that identifies the connection.	prod_sales_data
File Type *	The type or format of the file.	Text
Container *	The name of the S3 bucket that contains the flat files.	prod-bucket
Folder Path *	The directory path within the bucket that points to the folder containing the flat files.	/data/prod-files/
Region *	AWS region where the S3 bucket is located.	us-east-1
Type *	EC2 role authentication supports only System Connections. Refer Connections for more details.	System

AWS IAM Authentication

Name	Description	Example Values
Connection Name *	A unique name that identifies the connection.	prod_sales_data
File Type *	The type or format of the file.	Text
Container *	The name of the S3 bucket that contains the flat files.	prod-bucket
Folder Path *	The directory path within the bucket that points to the folder containing the flat files.	/data/prod-files/
Region *	AWS region where the S3 bucket is located.	us-east-1
Type *	The connection type, either System Connection or User Connection. Refer Connections for more details.	System
Access Key *	The key used to authenticate the connection to the storage account.	AKIAIOSFODNN7MPLE
Secret Key *	The secret key used to authenticate the connection to the storage account.	wJalrXUtnFEMILEKEY

Connecting to Azure Blob Storage

Azure Access Key Authentication

Name	Description	Example Values
Connection Name *	A unique name that identifies the connection.	prod_sales_data
File Type *	The type or format of the file.	Text
Container *	Name of the storage container in Azure Blob Storage where the flat files are stored.	prod-data-container
Folder Path *	The directory path within the container that points to the folder containing the flat files.	/data/prod-files/
Type *	The connection type, either System Connection or User Connection. Refer Connections for more details.	System
Account Name *	The storage account name used for authentication.	prodstorageaccount
Access Key *	The key used to authenticate the connection to the storage account.	tdfhsdftjhy456edg78923asdafrthrf98236

Azure Service Principal Authentication

Name	Description	Example Values
Connection Name *	A unique name that identifies the connection.	prod_sales_data
File Type *	The type or format of the file.	Text
Container *	The name of the storage container in Azure Blob Storage where the flat files are stored.	prod-data-container
Folder Path *	The directory path within the container that points to the folder containing the flat files.	/data/prod-files/
Type *	Service principal authentication supports only System Connections. Refer Connections for more details.	System
Account Name *	The Azure Storage account name used for authentication.	prodstorageaccount
Tenant ID *	The Microsoft Entra ID (formerly Azure AD) tenant ID used for authentication.	12345678-abcd-1234-efgh-9876543210ab
Client ID *	The Application (client) ID registered in Microsoft Entra ID.	abcd1234-5678-90ef-ghij-1234567890kl
Client Secret *	The secret associated with the registered client app in Microsoft Entra ID.	exedrhsrtas57fd

Azure Shared Signature Authentication

Name	Description	Example Values
Connection Name *	A unique name that identifies the connection.	prod_sales_data
File Type *	The type or format of the file.	Text
Container *	Name of the storage container in Azure Blob Storage where the flat files are stored.	prod-data-container
Folder Path *	The directory path within the container that points to the folder containing the flat files.	/data/prod-files/
Type *	The connection type, either System Connection or User Connection. Refer Connections for more details.	System
Account Name *	The storage account name that hosts the Blob Storage container.	prodstorageaccount
Security Token *	The shared access signature (SAS) token used for authentication.	sv=2024-01-:59Z&st=2025-07-01T00:00:00Z&spr=https&sig=signature

Connecting to Local Storage

Users can read files from local storage, which may refer to a server directory for uploaded files or a network-mounted directory accessible to the cluster.

Name	Description	Example Values
Connection Name *	A unique name that identifies the connection.	prod_sales_data
Folder Path *	The directory path that points to the folder containing the flat files.	/data/prod-files/

warning

The connector does not support authentication for accessing files from local storage.

Custom Properties

The following optional connection properties can be configured based on user requirements.

Property	Default Value	Possible Values	Description
BatchSize	0	Numeric value	Specifies the maximum number of rows included in each batch operation. Set to 0 to submit the entire batch as a single request.
Readonly	false	true, false	Enforces read-only access to flat files from the provider.
MaxRow	-1	Numeric value	Limits the number of rows returned when no aggregation or GROUP BY is used.

Supported Datatypes

The following data types are supported:

INTEGER
FLOAT
DECIMAL
TEXT
BOOLEAN / BIT
DATE
TIME
TIMESTAMP

Unsupported Datatypes

The following data types are not supported:

LIST
DICT
SET
ARRAY
DATETIME WITH TIMEZONE
BINARY / IMAGE DATA

The following resources provide additional guidance on working with Flat Files in iceDQ.

Description	Link
Read Delimited File	View Article

Prerequisites​

Connecting to Amazon S3​

AWS Custom Credential Authentication​

AWS EC2 Role Authentication​

AWS IAM Authentication​

Connecting to Azure Blob Storage​

Azure Access Key Authentication​

Azure Service Principal Authentication​

Azure Shared Signature Authentication​

Connecting to Local Storage​

Custom Properties​

Supported Datatypes​

Unsupported Datatypes​

Related Articles​