AWS CloudFormation

The deployment consists of a few docker containers, database for storing metadata, and persistent file storage for storing files

This document contains instructions to deploy Datagrok using CloudFormation on AWS ECS cluster with AWS RDS and AWS S3.

We considered a lot of typical security nuances during the CloudFormation template development. As a result, you will create a Datagrok infrastructure in AWS that applies to all standard security policies.

More information about Datagrok design and components:

• Architecture
• Infrastructure

Prerequisites

1. Check that you have required permissions on AWS account to perform CloudFormation deployment to ECS.

Deploy Datagrok components

We prepared specific template for every need of our customers, answer simple questions below to use the right one for you.

Would you like to use an existing VPC in your AWS account?

Yes

Datagrok stand will be put in an existing VPC you choose upon creation.

Do you use Route53 as DNS provider?

Yes

Requirements

1. Create Route53 public hosted zone

How to deploy

1. Use the link to open CloudFormation template and fill all required parameters.

   1. Specify stack name. To meet AWS naming requirements, name must be shorter than 10 symbols and correspond S3 Bucket naming rules. We use 'datagrok' by default, but you may prefer to also specify env in the stack name.

2. Wait until AWS completes the deployment. The stack status will be 'CREATE_COMPLETE.' The script created datagrok stand in existing VPC using existing Route53 hosted zone. Your Datagrok instance is now ready to use.

   If you see one of the following statuses then something went wrong: CREATE_FAILED, ROLLBACK_IN_PROGRESS, ROLLBACK_COMPLETE, ROLLBACK_FAILED. Check the stack events for more information about error.

3. Enter the platform datagrok.<subdomain> using the admin user. To get the password:

   1. Go to stack Outputs. Find DatagrokAdminPassword and click on the link to open AWS Secret Manager.
   2. Click 'Retrieve secret value' and copy password. It is a generated password for the first admin login.
   3. To increase security, change the password for the admin user on the first login. Datagrok will ignore the admin password from secrets on subsequent restarts.

4. Complete the initial setup in platform and you are ready to use Datagrok.

No

Our CloudFormation scripts support external DNS providers, however, it will require a few manual steps to configure the endpoint.

Requirements

1. Come up with two endpoints: DATAGROK_DNS, CVM_DNS. Datagrok requires two endpoints: DATAGROK_DNS and CVM_DNS. Users will use DATAGROK_DNS to access Datagrok Web UI, and requests CVM_DNS will be sent automatically by Datagrok Client.

2. Create RSA SSL certificate for DATAGROK_DNS and CVM_DNS.

   • If you use AWS ACM service for SSL certificates
     1. Generate ACM certificate in AWS which will be valid for both endpoints: DATAGROK_DNS, CVM_DNS.
     2. Copy AWS ARN for the created certificate. It should look like this: arn:aws:acm:<region>:<account_id>:certificate/<certificate_id>.
   • If you do not use AWS ACM service for SSL certificates, you can create a certificate for DATAGROK_DNS , CVM_DNS endpoints any way you are already using. Wildcard certificate also suffices.
     1. Upload certificate to AWS ACM
     2. Copy AWS ARN for the created certificate(s). It should look like this: arn:aws:acm:<region>:<account_id>:certificate/<certificate_id>.

How to deploy

1. Use the link to open CloudFormation template and fill all required parameters.

   1. Specify stack name. To meet AWS naming requirements, name must be shorter than 10 symbols and correspond S3 Bucket naming rules. We use 'datagrok' by default, but you may prefer to also specify env in the stack name.
   2. DatagrokArnSSLCertificate: Specify AWS ACM ARN for DATAGROK_DNS and CVM_DNS from the 2nd prerequisites step.

2. Wait until AWS completes the deployment. The stack status will be 'CREATE_COMPLETE.' The script created datagrok stand with all required infrastructure from scratch using external DNS service and existing AWS ACM certificate. Your Datagrok instance is now ready to use.

   If you see one of the following statuses then something went wrong: CREATE_FAILED, ROLLBACK_IN_PROGRESS, ROLLBACK_COMPLETE, ROLLBACK_FAILED. Check the stack events for more information about error.

3. As you chose the fulfillment option with external DNS, you need to create CNAME DNS records for CVM and Datagrok Load Balancers. To get the Load Balancer endpoints for DNS record:

   1. Go to stack Outputs. Copy values for DatagrokLoadBalancerDNSName and CvmLoadBalancerDNSName.
   2. Use copied DNS names to create CNAME DNS records, for example
      • Host: DATAGROK_DNS, Target: DatagrokLoadBalancerDNSName
      • Host: CVM_DNS, Target: CvmLoadBalancerDNSName

4. Enter the platform DATAGROK_DNS using admin user. To get the password:

   1. Go to stack Outputs. Find DatagrokAdminPassword and click on the link to open AWS Secret Manager.
   2. Click 'Retrieve secret value' and copy password. It is a generated password for the first admin login.
   3. To increase security, change the password for the admin user on the first login. Datagrok will ignore the admin password from secrets on subsequent restarts.

5. Complete the initial setup in platform and you are ready to use Datagrok.

No

Datagrok stand will create VPC and all required network resources itself.

Do you use Route53 as DNS provider?

Yes

Requirements

1. Create Route53 public hosted zone

How to deploy

1. Use the link to open CloudFormation template and fill all required parameters.

   1. Specify stack name. To meet AWS naming requirements, name must be shorter than 10 symbols and correspond S3 Bucket naming rules. We use 'datagrok' by default, but you may prefer to also specify env in the stack name.

2. Wait until AWS completes the deployment. The stack status will be 'CREATE_COMPLETE.' The script created datagrok stand with all required infrastructure from scratch using existing Route53 hosted zone. Your Datagrok instance is now ready to use.

   If you see one of the following statuses then something went wrong: CREATE_FAILED, ROLLBACK_IN_PROGRESS, ROLLBACK_COMPLETE, ROLLBACK_FAILED. Check the stack events for more information about error.

3. Enter the platform datagrok.<subdomain> using the admin user. To get the password:

   1. Go to stack Outputs. Find DatagrokAdminPassword and click on the link to open AWS Secret Manager.
   2. Click 'Retrieve secret value' and copy password. It is a generated password for the first admin login.
   3. To increase security, change the password for the admin user on the first login. Datagrok will ignore the admin password from secrets on subsequent restarts.

4. Complete the initial setup in platform and you are ready to use Datagrok.

No

Our CloudFormation scripts support external DNS providers, however, it will require a few manual steps to configure the endpoint.

Requirements

1. Come up with two endpoints: DATAGROK_DNS, CVM_DNS. Datagrok requires two endpoints: DATAGROK_DNS and CVM_DNS. Users will use DATAGROK_DNS to access Datagrok Web UI, and requests CVM_DNS will be sent automatically by Datagrok Client.

2. Create RSA SSL certificate for DATAGROK_DNS and CVM_DNS.

   • If you use AWS ACM service for SSL certificates
     1. Generate ACM certificate in AWS which will be valid for both endpoints: DATAGROK_DNS, CVM_DNS.
     2. Copy AWS ARN for the created certificate. It should look like this: arn:aws:acm:<region>:<account_id>:certificate/<certificate_id>.
   • If you do not use AWS ACM service for SSL certificates, you can create a certificate for DATAGROK_DNS , CVM_DNS endpoints any way you are already using. Wildcard certificate also suffices.
     1. Upload certificate to AWS ACM
     2. Copy AWS ARN for the created certificate(s). It should look like this: arn:aws:acm:<region>:<account_id>:certificate/<certificate_id>.

How to deploy

1. Use the link to open CloudFormation template and fill all required parameters.

   1. Specify stack name. To meet AWS naming requirements, name must be shorter than 10 symbols and correspond S3 Bucket naming rules. We use 'datagrok' by default, but you may prefer to also specify env in the stack name.
   2. DatagrokArnSSLCertificate: Specify AWS ACM ARN for DATAGROK_DNS and CVM_DNS from the 2nd prerequisites step.

2. Wait until AWS completes the deployment. The stack status will be 'CREATE_COMPLETE.' The script created datagrok stand with all required infrastructure from scratch using external DNS service and existing AWS ACM certificate. Your Datagrok instance is now ready to use.

   If you see one of the following statuses then something went wrong: CREATE_FAILED, ROLLBACK_IN_PROGRESS, ROLLBACK_COMPLETE, ROLLBACK_FAILED. Check the stack events for more information about error.

3. As you chose the fulfillment option with external DNS, you need to create CNAME DNS records for CVM and Datagrok Load Balancers. To get the Load Balancer endpoints for DNS record:

   1. Go to stack Outputs. Copy values for DatagrokLoadBalancerDNSName and CvmLoadBalancerDNSName.
   2. Use copied DNS names to create CNAME DNS records, for example
      • Host: DATAGROK_DNS, Target: DatagrokLoadBalancerDNSName
      • Host: CVM_DNS, Target: CvmLoadBalancerDNSName

4. Enter the platform DATAGROK_DNS using the admin user. To get the password:

   1. Go to stack Outputs. Find DatagrokAdminPassword and click on the link to open AWS Secret Manager.
   2. Click 'Retrieve secret value' and copy password. It is a generated password for the first admin login.
   3. To increase security, change the password for the admin user on the first login. Datagrok will ignore the admin password from secrets on subsequent restarts.

5. Complete the initial setup in platform and you are ready to use Datagrok.