Audit
In Datagrok, users can perform a wide range of tasks, from self-service data analytics to using custom applications. Datagrok's audit system provides detailed insights into how users interact with the platform. With this knowledge, you can:
- Improve UX
- Remove unused menu items, etc.
- Understand your users better
- Cluster usage by usage patterns
- Correlate usage patterns with the organizational groups
- Track data history
- Track origin of datasets
- See history of modifications
- See all actions performed on an object
- Analyze performance
- See how long each operation takes
- Analyze function usage
- Get a table of users, timestamps, inputs and outputs
- Track errors
- See all errors across the platform
- Identify actions that cause errors
Audit log storage
Datagrok automatically records and stores all user actions performed on entities in a structured format within a Postgres database, making it easy to query and analyze specific events. The audit data is stored in the following tables:
events
event_types
event_parameters
event_parameter_values
Each event is associated with a fixed type and the user session that triggered it.
Audit record types
- query-created
- query-edited
- query-deleted
- query-start
- query-published
- query-transformations-edited
- connection-created
- connection-edited
- connection-deleted
- connection-published
- job-created
- job-edited
- job-deleted
- job-transformations-edited
- job-start
- script-created
- script-edited
- script-deleted
- script-start
- script-published
- predictive-model-created
- predictive-model-edited
- predictive-model-deleted
- predictive-model-start
- predictive-model-published
- action-start
- notebook-created
- notebook-edited
- notebook-deleted
- notebook-start
- notebook-published
- project-created
- project-edited
- project-deleted
- project-opened
- entity-shared
- entity-shared-silent
- table-produced
- user-invited
- comment-posted
- dialog-ok
- main-menu-item-click
- error
- tutorial-completed
- viewer-rendered
- log
- package-tested
Accessing audit logs
You can access audit logs in a number of ways:
- From the Context Panel
- Activity: This info pane shows server-side activity for the current entity over the past seven days
- Usage: This info pane shows a high-level overview of server-side usage for the current entity in the last seven days
- From the Console
- From a Datagrok app, Usage Analysis
- Using external providers like Amazon CloudWatch
Exporting logs to Amazon CloudWatch
To set up automated exports of audit logs to Amazon CloudWatch:
- On the Sidebar, click Settings () > Logger. This opens the Settings - Logger view.
- In the Settings - Logger view, click ADD NEW EXPORT BLOCK next to CloudWatch export. This shows the export settings.
- Configure the export settings:
- Level: Select the log levels to push to CloudWatch (error, warning, info, etc.)
- Parameters:
- AWS connection:
- Log Group:
- Stream: Map audit record types to specific log streams (e.g.,
log => datagrok_log
,error => datagrok_errors
). - Batch size:
- Optional. Repeat the steps for other log levels.
Logging events
Datagrok logs both client-side and server-side activities.
Client-based actions
Thanks to Datagrok's in-memory database, many user activities related to exploratory data analysis, such as opening local files, aggregating tables, or adding viewers, occur entirely on the client. These actions generate internal named events that serve various purposes:
- Integration of different Datagrok components
- Ad-hoc customizations and extensions by plugins
- Audit logs
Datagrok logs a reasonable default set of client-based actions. For example, it logs the opening of a file and its name, but not the content. Less important actions, like "rows selected", aren't logged. To learn how to customize what gets logs, see Customize audit logging.
Server-related actions
When an action modifies the server's state in any way, it's recorded in the audit log in addition to the log file. You can see a seven-day history of such activity on the Context Panel under Activity, and the high-level overview of the the entity's usage under Usage.
By default, each explicitly executed function is logged, along with its parameter values.
Customize audit logging
To customize which events are logged and how they are handled, you have these options:
- Configure logging settings: For any user or group, add or remove logged events from the default set
- Create custom logging events and make other customizations using JS API
Changing settings for default logs
To access logging settings, on the Sidebar, click Settings () > Logger. The Settings - Logger view opens. From here, you can specify which audit events are logged for a user or group, and set timeouts for logging events.
To specify which audit events should be logged for a user or group:
- Under Log settings add a user or group and the desired logging level (Standard, Verbose, Custom).
- Optional. Adjust settings for the selected option by clicking the Gear icon next to the dropdown.
Customizing logging via JavaScript API
To customize audit logs using Datagrok's JavaScript API:
-
Create a
Logger
object:// Create logger
let logger = new DG.Logger();
// You can pass a callback that sets predefined parameters
let logger = new DG.Logger((m) => m.params['persistent']= 'value'); -
To log events, use the
logger.log()
method:// Default type is "log"
logger.log('HELLO WORLD', {test: 'value', 'foo': 'bar'});
// But you can specify another
logger.log('HELLO WORLD', {test: 'value', 'foo': 'bar'}, 'my_log'); -
Errors are logged automatically, but you can log them explicitly and specify the stack trace for grouping errors:
// Specify stack trace to group errors
logger.error('Error!', stackTrace);
See this sample for a demonstration of how to handle events.