Skip to main content

Connectors

Establishing a connection to a supported data source connector enables seamless data access for purposes such as testing and monitoring. Each connector is designed to support the JDBC (Java Database Connectivity) protocol, which provides a standardized method for connecting to and interacting with various data sources.

Type of Connectors​

Below are the high-level categories of connectors, each serving a specific purpose based on the type of data source and use case:

  • Database - Connectors in this category interface with traditional relational databases using structured query language (SQL). They are commonly used for transactional systems and structured data storage.
    Examples: MySQL, PostgreSQL, Oracle, Microsoft SQL Server, IBM Db2

  • File - These connectors access data stored in flat files or structured formats, allowing ingestion from local or distributed file systems.
    Examples: CSV, Excel, Parquet,JSON, XML

  • Application - Application connectors enable data extraction from enterprise or SaaS applications through APIs or custom interfaces.
    Examples: Salesforce, SAP

  • Analysis Service - These connectors are used to access data from Online Analytical Processing (OLAP) systems, where data is pre-aggregated for multidimensional analysis.
    Examples: Microsoft SQL Server Analysis Services (SSAS)

  • BI Report - These connectors extract data directly from business intelligence tools and reports, allowing analysis of published dashboards and insights.
    Examples: Power BI

  • Query Engine - Connectors in this category integrate with engines designed to query large datasets across multiple sources, often used in big data environments.
    Examples: Athena, Databricks, Denodo

  • Cloud Data Warehouse - These connectors interface with cloud-native platforms that provide scalable and high-performance data warehousing solutions.
    Examples: Snowflake, Google BigQuery

  • NoSQL Database - These connectors provide access to non-relational databases designed for unstructured or semi-structured data, offering scalability and flexibility.
    Examples: MongoDB, Cosmo DB, Azure Table

Engines​

The system supports two primary types of engines for executing operations, each optimized for different performance and scalability needs:

  • Standard Engine Type: Single Actor (Single-Threaded)

    • This engine operates with a single actor, executing tasks sequentially on a single thread.
    • It is suitable for low to moderate workloads where concurrency is not a primary concern.

  • High Throughput (HT) Engine Type: Multi Actor (Multi-Threaded)

    • The High Throughput engine is designed for performance-intensive scenarios.
    • It operates with multiple actors running in parallel threads.
    • User can configure the number of actors by specifying the degree of parallelism.
    • Each actor typically occupies a separate CPU core to maximize throughput and efficiency.
    • These multi-actor operations benefit significantly from parallel processing, making the HT engine ideal for large-scale data handling and complex computation scenarios.
      Use Cases:
         a. Expression Evaluation.
         b. Deferred Sort.

Authentication Types​

Different connectors support various authentication mechanisms based on the data source, cloud provider, or enterprise security requirements. Below is a categorized list of authentication types with brief explanations:

  • Standard Authentication

    • Username and Password: Basic credential-based authentication using a user ID and password combination.
    • SQL Server Authentication: Uses a SQL Server-defined username and password, separate from Windows credentials.
    • Local: Authenticates against a local account on the system or application.
    • Anonymous: No credentials required; used for publicly accessible resources or during development.

  • Cloud Provider Authentication

    • AWS Authentication

      • AWS Custom Credential: Uses manually provided AWS Access Key ID and Secret Access Key.
      • AWS EC2 Role: Automatically uses the IAM role attached to the running EC2 instance for authentication.
      • AWS IAM: Grants access based on Identity and Access Management (IAM) policies and roles.

    • Azure Authentication

      • Azure Access Key: Authenticates using a shared access key typically provided for blob storage or other services.
      • Azure Standard Signature: Uses a Shared Access Signature (SAS) token to authenticate with Azure services for time-limited access.
      • Azure Account Key: Provides full access using the primary or secondary storage account key.
      • Azure Service Principal: Authenticates via an enterprise application registered in Azure AD using Client ID and Secret or Certificate.

  • Enterprise Authentication

    • Kerberos: Ticket-based secure authentication protocol used in many enterprise environments.
    • Kerberos Ticket Cache: Uses a previously obtained Kerberos ticket from the local ticket cache for authentication.
    • LDAP: Authenticates against a Lightweight Directory Access Protocol directory.
    • Active Directory Password: Uses a domain username and password to authenticate against Active Directory.
    • Windows Authentication: Integrated authentication that uses the credentials of the currently logged-in Windows user (SSO).

  • Token and Certificate-Based Authentication

    • OAuth Connected App: Utilizes OAuth 2.0 to authenticate through an external authorization server (e.g., Salesforce, Google, Microsoft).
    • Service Account: Uses a predefined account typically with limited and scoped permissions for automation or backend access.
    • Key Pair: Authenticates using a private/public key pair, often used in SSH or API-based systems.

  • Advanced Security Protocols

    • SCRAM-SHA-256: Salted Challenge Response Authentication Mechanism using SHA-256 for secure password exchange.
    • SCRAM-SHA-1: Similar to SCRAM-SHA-256 but uses SHA-1; provides mutual authentication and password security.

Type of Connection​

Organizations may enforce policies that mandate the use of either generic service accounts or individual user accounts to establish connectivity with data sources. iceDQ supports two types of connections to align with these policies: System Connection and User Connection.

System Connection​

System connections are ideal in scenarios where a generic user or service account is used to interact with a data source. This allows multiple iceDQ users to utilize the same credentials in their rules, streamlining access management. Users automatically inherit the database access granted to the system connection account. For example, if the account has access to databases X, Y, and Z, all users using this connection will share that same access.

Important

System connections require a Public Secret for authentication. While each connection supports only one secret, multiple keys derived from that secret can be used to protect various sensitive data fields.

User Connection​

User connections offer maximum control and compliance and are the preferred option in environments where data access must be tightly regulated. This approach requires each user to provide their own credentials to access the data source. Upon connection, iceDQ retrieves the user's Private Secret, which adds an additional layer of security. This method ensures that access is personalized and secure, reducing the risk of unauthorized usage.

Important

User connections require a Private Secret for authentication. As with system connections, only one secret is supported per connection, but multiple keys derived from that secret can be used to protect different sensitive data fields.

How To: Create a Connection​

Follow the steps below to create a connection with various types of connectors in iceDQ:

  1. From the main application menu, go to the Connectors section.
  2. Within the Connectors section, select the Secrets tab and create a secret. For detailed instructions on how to create a secret, refer to the Secrets Page link.
  3. Still within the Connectors section, switch to the Connections tab.
  4. Click the New Connection button to initiate the creation process.
  5. From the thumbnail list, choose the connector for which you want to create a connection.
    Note: You can simplify connector selection using the following methods:
    • Search: Use the search bar to find and select the desired connector.
    • Select Connection Type: Optionally, use categorized filters to narrow down the list and then select the connector.
  6. Once on the connector configuration page, proceed as follows:
      In the Overview Tab:
      a. Add the connection configuration details. (Refer to the specific connector's page for detailed descriptions of each field.)
      b. Select the Type of Connection: User or System
      c. Select the Secret Name created in Step 2.

    In the Properties Tab:
    Add any additional connection properties:
    a. Click Add
    b. Enter the following details:
       Name – Property name
       Value – Value to assign to the property
    c. Click the ✔ (tick mark) to save the property
    d. Optionally, click the ✖ (X) to discard the property

  1. Click the Save button to finalize and store the connection configuration.
  2. Click the Test button to verify connectivity from iceDQ to the data source.

Additional options​

  • For each connection:
Field NameDescription
SaveSaves the configuration details of the data source. This action activates all mandatory configurations are provided during initial creation or editing.
TestTests the connectivity from iceDQ to the data source using the configuration provided. This action activates once the connection is saved after the initial creation or editing.
DiscardThis option is available throughout the process until the save option is hit. During the initial creation, if discard is pressed, the entire connection configuration will be discarded. Once the connection is saved and then any changes are made, discard will roll back only the changes made during the last edit.
Duplicate duplicate button    Duplicates the workflow.
Delete delete buttonDeletes the connection. This option is activated once the connection is saved.
  • In Properties Tab:
Field   NameDescription
SearchEnables users to search for specific property values.
Delete delete buttonAllows deletion of one or more properties after selecting them using the ✔️ button.
  • For individual properties:
Field   NameDescription
Delete delete buttonDelete the individual property.
Edit ✏️Opens the prperty for editing, allowing modification of its name and value.

How To: Create a System Connection​

This video shows how to create a System connection. Remember, Owner role is required.

How To: Create a User Connection​

This video shows how to create a User connection. Remember, Owner role is required.

Provide Credentials​

This video shows how to plugin credentials using a Private secret in a user connection.

How To: Delete a Connection​

  1. From the main application menu, navigate to the Connectors section.
  2. Within the Connectors section, select the Connections/User Connections tab to view and manage connections.
  3. Select one or more connections to be Deleted. Use check button.
  4. Click on the [Delete] option to delete the workflow.
  5. Select Delete option.
  6. Cancel: Select this option to exit to delete the connection process.
Last updated on