Skip to main content

AWS CloudFormation (EKS)

The deployment consists of a few docker containers, a database for storing metadata, and persistent file storage for storing files.

This document contains instructions to deploy Datagrok using CloudFormation on AWS EKS with AWS RDS (PostgreSQL) and AWS S3. The template provisions the EKS cluster (or uses yours), RDS instance, S3 bucket, Application Load Balancer, IAM roles with IRSA, and installs the Datagrok Helm chart on the cluster automatically.

We considered a lot of typical security nuances during the CloudFormation template development. As a result, you will create a Datagrok infrastructure in AWS that applies to all standard security policies.

More information about Datagrok design and components:

Migrating from ECS?

The EKS templates share the database, bucket, secrets, and load balancer with the ECS templates. Replacing an ECS stack's template with the matching EKS template keeps your data and your URL. See Migrate from ECS.

Prerequisites​

  1. Check that you have the required permissions on the AWS account to perform CloudFormation deployment to EKS.
  2. The private subnets need outbound internet access (NAT gateway). The cluster pulls Datagrok images from Docker Hub, and the installer downloads Helm and the chart.

New or existing cluster​

By default, the stack creates an EKS cluster with a managed node group (NodeInstanceType, NodeGroupDesiredSize) in the private subnets. Set GpuRequired to true to add a GPU node group for GPU plugins.

To install Datagrok on a cluster you already run, set ExistingEksClusterName to its name and use one of the vpc-eks-* templates with the cluster's VPC. The stack then skips the cluster and node groups, and:

  • adds the cluster's IAM OIDC provider if it has none;
  • opens RDS and the load balancer to the cluster security group;
  • installs the AWS Load Balancer Controller only when the cluster does not run one;
  • installs Datagrok into the HelmNamespace namespace (datagrok by default).

The existing cluster must meet these requirements:

  • Its authentication mode is API or API_AND_CONFIG_MAP. The stack grants its installer access with an EKS access entry.
  • Its nodes use the cluster security group (the default for managed node groups) and have enough free capacity for Datagrok (about 4 vCPU and 8 GiB).
  • Its API endpoint is reachable from the private subnets.

The stack installs Datagrok with a CodeBuild project. If an install or an update fails, open the build log from the DatagrokInstallerLogs stack output.

Deploy Datagrok components​

We prepared a specific template for every need of our customers. Answer the simple questions below to use the right one for you.

Would you like to use an existing VPC in your AWS account?​

The template will create a new VPC and all required network resources for you.

Do you use Route53 as your DNS provider? (new VPC)​

Requirements​
  1. Create a Route53 public hosted zone.
How to deploy​

  1. Use the link to open the CloudFormation template and fill in all required parameters.

    1. Specify stack name.

  2. Wait until AWS completes the deployment. The stack status will be CREATE_COMPLETE.

  3. Enter the platform at datagrok.<subdomain> using the admin user. Retrieve the password from DatagrokAdminPassword in stack Outputs.

  4. Complete the initial setup in the platform and you are ready to use Datagrok.

Service versions​

Every service has a dedicated CloudFormation parameter (DatagrokVersion, GrokPipeVersion, GrokSpawnerVersion, GrokConnectVersion, JKGVersion, GrokRegistryProxyVersion, RabbitmqVersion) for picking the container image tag. Defaults track the most recent Datagrok release and are tested together — see Images and versions for the current set and pinning conventions across deployment paths. HelmChartVersion defaults to DatagrokVersion; the template appends -helm automatically (e.g. 1.27.3 → chart tag 1.27.3-helm).

Update Datagrok components​

You can update your Datagrok deployment without re-creating infrastructure. Before updating, we recommend backing up the database and persistent storage. Refer to your internal backup procedures.

How to update​

Use the same deployment script and an updated version of the deployment profile:

  1. Click Update > Replace current template, and provide the new template URL corresponding to your deployment configuration.
  2. Specify new versions for image tags (see the release history). HelmChartVersion defaults to the same value as DatagrokVersion.
  3. Click Next, skip optional settings, and proceed to Review.
  4. If the stack enters a failed state (e.g., UPDATE_ROLLBACK_IN_PROGRESS), check events for details.
note
  • CloudFormation will not replace the database or file storage during update.
  • Your platform URL, admin credentials, and uploaded files will remain unchanged.
  • Database schema migrations run automatically on the first start of the new app version.
  • If you previously customized your deployment with environment variables, they will persist unless explicitly modified in parameters.

Migrate from ECS​

If you run Datagrok on the ECS CloudFormation template, you can move it to EKS by updating the stack. The ECS and EKS templates define the database, the bucket, the generated secrets, and the load balancer identically, so CloudFormation keeps them. Your users, files, settings, admin password, and URL stay the same, and external DNS records keep pointing at the same load balancer.

  1. Back up the RDS database and the S3 bucket.
  2. In the existing stack, click Update > Replace current template and point at the EKS template with the same VPC and DNS options: vpc-fargate-r53-basic.yml becomes vpc-eks-r53-basic.yml, and so on.
  3. Keep all parameter values, including the service versions. Change versions in a separate update.
  4. Review the change set before you execute it. DatagrokDB, DatagrokS3, and DatagrokLoadBalancer must not appear in it with Replacement set to True. ECS services, task definitions, and the internal load balancer are removed, and the EKS cluster and the installer are added.
  5. Execute the change set. The update takes about 30 minutes. ECS keeps serving until the Datagrok pods are ready. The pods then take over the load balancer target group, and CloudFormation removes the ECS services. Expect up to a minute of downtime during the switch.

If the change set replaces DatagrokDB or DatagrokS3, do not execute it. Contact support@datagrok.ai.

See also​