Architecture & Deployment Overview
This article provides an overview of the architecture, deployment, and security protocols to facilitate the setup and management of iceDQ in customer-managed environments. For detailed configuration and installation instructions, refer to the provided links.
Overview
iceDQ is a microservices-based application designed to be deployed within customer-managed environments, including AWS, Azure, Google Cloud, Oracle Cloud, or on-premises data centers.
Application: iceDQ includes over 20 application and engine services deployed in a Kubernetes cluster. Customers may bring their own Kubernetes cluster, such as EKS (Elastic Kubernetes Service), AKS (Azure Kubernetes Service), GKE (Google Kubernetes Engine), OKE (Oracle Kubernetes Engine), or OpenShift. Alternatively, customers may use Linux-based VMs, in which case our installer will create an embedded Kubernetes cluster.
Metadata: All application metadata and summary results are stored in a PostgreSQL repository. While an embedded PostgreSQL database is available, we recommend using an external PostgreSQL database for production environments.
Exception Reports: iceDQ identifies data issues within customer ecosystems and stores them as Parquet files on an attached persistent volume.
Connectivity: iceDQ connects to source and target data sources via JDBC or REST APIs. Therefore, the Virtual Private Cloud (VPC) in which iceDQ is deployed must have access to these data sources.
Interface: Users can access iceDQ through a web browser or REST APIs. Customers can configure the application endpoint through a load balancer.
Additional Details:
- Backend services are developed in Java.
- Frontend services are built using Angular.
- Service communication is facilitated via embedded queues or REST APIs.
- Vault service securely stores all sensitive information.
Security
- Authentication: User login is authenticated through an IAM service integrated with the customer’s IDP or LDAP.
- Credential Storage: Credentials for connecting to data sources are securely stored in the key vault.
- Decryption: The rules engine decrypts connections in-memory at runtime when accessing data sources.
- Data Persistence: Exception reports are stored on an attached persistent volume.
AWS EKS Deployment Options
Customers can deploy iceDQ in either a dedicated cluster or within a namespace in a shared cluster. While both patterns follow a similar approach, two installation methods are available:
KOTS Install
Using KOTS (Kubernetes Off-The-Shelf), the installation can be configured via a user-friendly installer UI. This method can automatically create persistent volumes (PVs) and other required resources, provided appropriate permissions are available.
- System Requirements: KOTS System Requirements
- Online Installation Guide: KOTS Online Install
HELM Install
The Helm installation option is suitable for customers with Kubernetes and Helm expertise. This method requires that persistent volumes and other necessary resources are preconfigured before installation.
- System Requirements: Helm System Requirements
- Online Installation Guide: Helm Online Install
Following is the list of items you will need to deploy the application in EKS.
| Item | Requirement |
|---|---|
| Kubernetes Service | Amazon Elastic Kubernetes Service |
| Worker Nodes | 2 x m7i.4xlarge |
| Persistent Volume | 1TB of Amazon Elastic File System for storing exception reports and logs |
| Database | 10GB dedicated PostgreSQL database for application metadata storage |
| Database User | Service account user with ALL privileges on the database |
| Registry | Amazon Elastic Container Registry for storing all application images |
| Load Balancer | One load balancer if using HELM; two load balancers if using KOTS |
| Domain | Internal domain mapped to the application, e.g., internal.icedq.app |
| Certificates | TLS certificates and keys to enable SSL for the application |
| Email Server | SMTP configuration for sending application email notifications |
| Backup Storage | S3 Bucket for application backup storage |
The above outlines the minimum requirements for production deployments. For more detailed information, please refer to the system requirements section under each installation option.