Access
Datagrok unifies data access across your organization, simplifying retrieving data from multiple sources. You can easily visualize, explore, and learn from your data, and use the insights gained to take action. Additionally, the platform provides access controls, security features, caching, and automatic monitoring of connection health.
Data sources
In addition to local files that you can drag and drop from your computer, Datagrok integrates with various data providers. You can connect to any machine readable source: a file storage (like third-party cloud services or an organization-hosted Datagrok server), databases, or webservices.
Datagrok also supports scripting in various languages, such as R, Julia, and Python, which means you can create custom data sources. For example, you can load a dataframe from an external website or package, open a specific table using its ID, or write a package to extract data from multiple sources and combine them into one. For more information on getting data using functions and scripts, see Access data section in the developers' documentation.
Datagrok also hosts public datasets that can be used for analysis, testing, and prototyping. These datasets cover various domains, including cheminformatics, clinical trials, and more.
Data connection
A data connection is an entity that contains the necessary information to connect to a specific data source, such as its address and credentials. A data connection allows you to work with files and database tables directly in Datagrok. You can access data either manually through the UI or programmatically through an application. For manual data access, Datagrok provides a convenient UI that lets you connect directly to any of the 30+ supported connectors, retrieve data using queries, and securely share data with collaborators.
A connector is a plugin that enables the integration of external data providers into the platform. It can work with a database, an Excel file, a CSV file, a web service, or any other source capable of providing the data. Most of our data connectors are open-sourced and extendable (under MIT license).
To see all available data source connections, on the Sidebar, select Manage > Connections. From there, you can search connection by name or by tag.
A data connection is an entity, which means it can be shared, assigned permissions, annotated, and more.
For instructions on how to add a supported data source, set credentials, share, and manage it from the UI, see documentation for each data source type.
For instructions on how to add a supported data source, set credentials, share, and manage it programmatically, see developer's documentation.
For specific details on the configuration required, see each individual connector's documentation page in the Connectors directory.
Data query
A data query is a function associated with a data connection that typically returns a dataframe. Queries can be executed either manually, or as part of data jobs. Datagrok has a convenient interface for creating, running, and sharing query results, including aggregation editor, auto-generated parameter dialogs, and an ability to create dynamic dashboards to visualize query results. All data governance features — such as data lineage, history, and security — are applicable to queries. For more information about queries, see documentation for the respective data source type.
In addition to querying databases with SQL, you can query other data storages:
| Data source | Query |
|---|---|
| Relational databases | SQL |
| File share | Filename |
| Excel file | Sheet name |
| Linked data | SPARQL |
| Box | Filename |
The result of a query execution is represented by the function call. For instructions on how to run, share queries, and manage queries programmatically, see developer's documentation.
A data query is an entity, which means it can be shared, assigned permissions, annotated, and more.
Data formats
With Datagrok, you can retrieve both structured and unstructured data. Datagrok supports multiple data formats, including popular formats like CSV, TXT, JSON, and scientific formats like MAT, molecular structure formats (like PDB, MOL, or SDF), geographic annotation, and others. Datagrok also offers a flexible system for extending the platform with organization-specific data formats (see Extensible framework).
Browsing and preview
Datagrok offers a rich set of features to help users efficiently browse, manage, and preview data. For more information, see:
Sharing and access control
Datagrok treats data connections, file shares, database tables and columns, and queries as entities, which means there is a common set of operations that can be applied to them. These entities can be shared with others, assigned access privileges, commented on, versioned, audited, and so on. Some of the most popular privileges are: view, edit, delete, and share. These privileges can be given to individual users, or
to groups. For more information on the access privilege model, see Access Control.
Data connections can be shared as part of a project, package, or as a standalone entity. Sharing a query automatically shares the associated database connection, since query access depends on connection permissions. However, if you share a database connection with someone, your queries won't be shared automatically. You need to share them separately.
To learn how to control access for each data source, see the documentation for the corresponding data source.
Extensible framework
Datagrok is designed as an extensible environment, where extensions can customize or enhance virtually any part of the platform. For example, you can create custom connectors, customize menus, add context actions, customize data preview, and more.
To learn more about extending and customizing Datagrok, see the Develop section of our documentation. For examples related to data access, see File Manager.
Resources
