Skip to main content

Audit

In Datagrok, users can perform a wide range of tasks, from self-service data analytics to using custom applications. Datagrok's audit system provides detailed insights into how users interact with the platform. With this knowledge, you can:

  • Improve UX
    • Remove unused menu items, etc.
  • Understand your users better
    • Cluster usage by usage patterns
    • Correlate usage patterns with the organizational groups
  • Track data history
    • Track origin of datasets
    • See history of modifications
    • See all actions performed on an object
  • Analyze performance
    • See how long each operation takes
  • Analyze function usage
    • Get a table of users, timestamps, inputs and outputs
  • Track errors
    • See all errors across the platform
    • Identify actions that cause errors

Audit log storage​

Datagrok automatically records and stores all user actions performed on entities in a structured format within a Postgres database, making it easy to query and analyze specific events. The audit data is stored in the following tables:

  • events
  • event_types
  • event_parameters
  • event_parameter_values

Each event is associated with a fixed type and the user session that triggered it.

Audit record types
  • query-created
  • query-edited
  • query-deleted
  • query-start
  • query-published
  • query-transformations-edited
  • connection-created
  • connection-edited
  • connection-deleted
  • connection-published
  • connection-raw-write
  • job-created
  • job-edited
  • job-deleted
  • job-transformations-edited
  • job-start
  • script-created
  • script-edited
  • script-deleted
  • script-start
  • script-published
  • predictive-model-created
  • predictive-model-edited
  • predictive-model-deleted
  • predictive-model-start
  • predictive-model-published
  • action-start
  • notebook-created
  • notebook-edited
  • notebook-deleted
  • notebook-start
  • notebook-published
  • project-created
  • project-edited
  • project-deleted
  • project-opened
  • docker-image-created
  • docker-image-rebuilt
  • docker-image-deleted
  • docker-container-created
  • docker-container-edited
  • docker-container-started
  • docker-container-stopped
  • docker-container-action-failed
  • docker-container-deleted
  • docker-container-config-created
  • docker-container-config-deleted
  • package-created
  • package-edited
  • package-deleted
  • package-version-published
  • package-version-activated
  • sync-setup-created
  • sync-setup-edited
  • sync-setup-deleted
  • user-report-created
  • user-applied
  • entity-shared
  • entity-shared-silent
  • user-invited
  • comment-posted
  • dialog-ok
  • main-menu-item-click
  • error
  • tutorial-completed
  • viewer-rendered
  • log
  • package-tested
  • alert-opened
  • alert-escalated
  • alert-cleared
  • alert-acknowledged
  • alert-resolved
  • problem-status-changed
  • connection-checked
  • user-error
  • client-audit
  • key.material.delete
  • key.revoke.immediate
  • key.revoke.force
  • key.revoke.queued
  • key.rotation.start
  • key.rotation.complete
  • key.rotation.auto
  • server-started
  • user-logged-in
  • user-logged-out
  • user-login-failed
  • user-impersonated
  • impersonation-failed
  • admin-session-started
  • admin-session-ended
  • dev-key-generated
  • email-confirmation-requested
  • email-confirmed
  • email-confirmation-failed
  • password-reset-requested
  • password-changed
  • password-change-failed
  • settings-changed
  • log-settings-changed
  • group-members-changed
  • permission-granted
  • permission-revoked
  • credentials-saved
  • admin-sql-executed
  • privileged-call

The seventeen from server-started to log-settings-changed are the platform's own security trail — who logged in and out, whose login failed and why, impersonation, admin sessions, developer keys, e-mail confirmations, password resets and changes, settings changes, and server starts. They are listed on the System Activity tab of Usage Analysis.

The last six record who can reach what: group membership changes (made in Datagrok or synced from an identity provider), permissions granted and revoked, credentials saved (the names of the saved fields, never their values), SQL run on the platform's own database connections, and functions called in an admin or impersonated session.

Passwords, tokens, keys, and other secrets are replaced with [REDACTED] in event messages and parameters before they are printed, stored, or exported.

Accessing audit logs​

You can access audit logs in a number of ways:

  • From the Context Panel
    • Activity: This info pane shows server-side activity for the current entity over the past seven days
    • Usage: This info pane shows a high-level overview of server-side usage for the current entity in the last seven days
  • From the Console
  • From a Datagrok app, Usage Analysis
  • In your own log backend, such as Amazon CloudWatch, Google Cloud Logging, or any OpenTelemetry collector (see Export logs)

Export logs​

Log sync sends log events to external destinations as they are posted, without waiting for them to be saved to the database. To add a destination:

  1. On the Sidebar, click Settings () > Logger, and expand Log sync.
  2. Click Add new sync block.
  3. Set Cloud to the destination and fill in its settings (see the table below).
  4. Select the Levels to sync. Only levels that are posted can be synced. See Changing settings for default logs.
  5. Click Save and apply.

Add one block per destination. For example, send errors to one CloudWatch log group and audit events to another.

Log sync settings
SettingApplies toDescription
EnabledAllTurns the destination off without deleting it
LevelsAllLevels to sync: error, warning, info, debug, audit, usage
Event typeAllNarrows the destination to a single audit or usage event type. Leave empty for all
ParamsAllSyncs only events whose parameters have the given values
Batch SizeAllMaximum number of events sent in one batch
FormatAlljson (one object per event, queryable in the backend) or text
CloudAllAmazon CloudWatch, Google Cloud Logging, or OpenTelemetry (OTLP)
ConnectionAllAn AWS or GCP connection with write access. Leave empty to authenticate with the instance role
Log Group, StreamAmazon CloudWatchDestination log group and stream. Without a connection, Datagrok creates them if they don't exist. With a connection, create them first
Log NameGoogle Cloud LoggingDestination log
EndpointOpenTelemetryOTLP/HTTP logs endpoint, for example http://otel-collector:4318/v1/logs
AuthOpenTelemetrynone, bearer (with Token), aws-sigv4, or gcp-oauth
Headers, Compress, Max PayloadOpenTelemetryExtra HTTP headers, gzip compression, and the size at which a batch is split
AlertsOpenTelemetrySends alert and heartbeat records regardless of the selected levels. On by default

Besides regular events, the server emits alert records and a heartbeat that let your monitoring system page you when a problem starts. The records, when they arrive, and the problems behind them are in the operator guide. alert-firing and heartbeat aren't stored in the database. They exist only in Log sync destinations.

To learn more, see Export logs to Amazon CloudWatch.

note

On a platform-managed instance, the Log sync section is hidden. The deployment owns the destinations.

Retention​

By default, Datagrok keeps all events. To delete old events automatically, turn on the Garbage Collector in Settings > Admin. It removes audit events after 365 days, errors, warnings, info, and usage events after 183 days, and debug events after 30 days. You can change each period. To keep events longer, export them to a backend with its own retention policy.

Integrity​

Events live in the Datagrok database, so whoever can change the database can change them. To keep a copy that no one can alter or delete, administrators included, export the log to write-once storage:

  1. Create an S3 bucket with Object Lock turned on at creation, in COMPLIANCE mode, with a default retention of the period you must keep audit evidence (for example, 400 days). GOVERNANCE mode can be bypassed by a privileged account. Don't add a lifecycle rule that expires objects sooner, and never delete the bucket's KMS key.
  2. Run an OpenTelemetry Collector with an OTLP/HTTP receiver behind TLS and a bearer token, and the awss3 exporter (marshaler: otlp_json, compression: gzip) writing to that bucket. Keep the AWS SDK's default checksum behavior: Object Lock rejects uploads without a checksum.
  3. In Log sync, add an OpenTelemetry destination pointing to the collector, with Auth set to bearer and at least the audit level selected.

Turning a destination off or deleting it is itself sent to that destination before the change applies. If a destination drops records or keeps failing, Datagrok raises an export problem until it delivers again.

Logging events​

Datagrok logs both client-side and server-side activities.

Client-based actions​

Thanks to Datagrok's in-memory database, many user activities related to exploratory data analysis, such as opening local files, aggregating tables, or adding viewers, occur entirely on the client. These actions generate internal named events that serve various purposes:

  • Integration of different Datagrok components
  • Ad-hoc customizations and extensions by plugins
  • Audit logs

Datagrok logs a reasonable default set of client-based actions. For example, it logs the opening of a file and its name, but not the content. Less important actions, like "rows selected", aren't logged. To learn how to customize what gets logs, see Customize audit logging.

When an action modifies the server's state in any way, it's recorded in the audit log in addition to the log file. You can see a seven-day history of such activity on the Context Panel under Activity, and the high-level overview of the entity's usage under Usage.

By default, each explicitly executed function is logged, along with its parameter values.

Customize audit logging​

To customize which events are logged and how they are handled, you have these options:

  1. Configure logging settings: For any user or group, add or remove logged events from the default set
  2. Create custom logging events and make other customizations using JS API

Changing settings for default logs​

To access logging settings, on the Sidebar, click Settings () > Logger. The Settings - Logger view opens. From here, you can specify which audit events are logged for a user or group, and set timeouts for logging events.

To specify which audit events should be logged for a user or group:

  1. Under Log settings add a user or group and the desired logging level (Standard, Verbose, Custom).
  2. Optional. Adjust settings for the selected option by clicking the Gear icon next to the dropdown.

Customizing logging via JavaScript API​

To customize audit logs using Datagrok's JavaScript API:

  1. Create a Logger object:

    // Create logger
    let logger = new DG.Logger();
    // You can pass a callback that sets predefined parameters
    let logger = new DG.Logger((m) => m.params['persistent']= 'value');

  2. To log events, use the logger.log() method:

    // Default type is "log"
    logger.log('HELLO WORLD', {test: 'value', 'foo': 'bar'});
    // But you can specify another
    logger.log('HELLO WORLD', {test: 'value', 'foo': 'bar'}, 'my_log');

  3. Errors are logged automatically, but you can log them explicitly and specify the stack trace for grouping errors:

    // Specify stack trace to group errors
    logger.error('Error!', stackTrace);

See this sample for a demonstration of how to handle events.