Connect your Databricks Workspace

Prerequisites

• You need Account Admin privileges and Metastore Admin privileges for your Databricks workspace
  • https://docs.databricks.com/en/administration-guide/index.html

• A Unity Catalog-enabled workspace
  • https://docs.databricks.com/aws/en/data-governance/unity-catalog/get-started#how-do-i-know-if-my-workspace-was-enabled-for-unity-catalog

Connecting the workspace

Step 1: Get account and workspace identifiers

• Head to the Databricks console
  • https://accounts.cloud.databricks.com/
• Note down your Databricks Account ID from the menu on the top right profile icon
  • https://docs.databricks.com/aws/en/admin/account-settings/#locate-your-account-id
• Click Workspaces, select a Unity Catalog-enabled workspace
• Note down your Workspace URL
• Open the workspace, and then note down your Workspace ID. Use these docs to identify your Workspace ID
  • AWS: https://docs.databricks.com/aws/en/workspace/workspace-details
  • GCP: https://docs.databricks.com/gcp/en/workspace/workspace-details
  • Azure: https://learn.microsoft.com/en-us/azure/databricks/workspace/workspace-details

Step 2: Create a service principal

• In the workspace, select Settings from the top right Profile icon
• Under Workspace admin, select Identity and access
• Next to Service Principals, click Manage
• Click Add service principal
  • An Add new service principal modal pops up, click Add new
  • Give the service principal a memorable name (e.g. select-service-principal)
• Open the newly created service principal, select Secrets tab, then click Generate secret
• Enter lifetime of 730 days, then click Generate
  • you will need to create a new secret and update it in SELECT when it expires
• Note down your Secret and Client ID

Step 3: Create a Serverless SQL Warehouse

• Navigate to SQL→SQL Warehouse from the navigation menu
• Click Create SQL warehouse with the following config
  • Name: pick a memorable name (e.g. select-sql-warehouse)
  • Cluster size: 2X-Small
  • Type: Serverless

• Click Create
• After the warehouse is created, the Manage permissions modal window is shown
  • You can also access this modal by clicking Permissions
• Search for and select the service principal you created (using the name or client id).
• Select Can Use permission, click Add and close the modal
• Note down the Warehouse ID

Step 4: Grant Data Reader Permissions

• Navigate to Catalog from the navigation menu
• Select system from the Catalog menu
• Click Permissions → Grant on the right
• Enter the following:
  • For Principals, select the principal you created
  • For Privilege presets, select Data Reader

Step 5: Add workspace to SELECT

• Navigate to settings: https://select.dev/app/settings
• Go to the Databricks tab and click the Add Workspace button
• Enter a memorable workspace name, and fill out all the details you have noted in the previous steps:
  • Account ID
  • Workspace URL
  • Workspace ID
  • Warehouse ID
  • Client ID
  • Client Secret
• Click Add and you are good to go!

Enabling Automated Savings

The Automated Savings feature for All-Purpose Compute, Jobs Compute and SQL Warehouses requires granting the CAN_MANAGE permission to the SELECT Service Principal on each of the relevant resources. For Jobs Compute, the permissions should be granted on the Job itself rather than the compute resources which it creates.

If, as part of evaluating an All-Purpose Compute cluster for potential savings, granting the CAN_ATTACH_TO permission allows access to Spark metrics and better estimates of savings potential, before full management access is granted to enable the feature. The SELECT Automated Savings agent will begin monitoring all clusters automatically where this permission is granted.

To make these permission grants scalable on larger accounts, we recommend the use of infrastructure as code tools such as Terraform or OpenTOFU to manage the grants for the SELECT Service Principal.