# Connect your Databricks Workspace

URL: https://select.dev/docs/reference/setup-guide/connect-your-databricks-workspace
Index of all SELECT docs: https://select.dev/docs/llms.txt

> Create a new Databricks Service Principal and connect your Databricks workspace to SELECT

## Prerequisites

- You need **Account Admin** privileges and **Metastore Admin** privileges for your Databricks workspace
   - [https://docs.databricks.com/en/administration-guide/index.html](https://docs.databricks.com/en/administration-guide/index.html)

![](https://cdn.sanity.io/images/wn0ylp2c/production/6bab78cffcf65efc9a9079e021ef64c18642876c-1133x687.png)

- A Unity Catalog-enabled workspace
   - [https://docs.databricks.com/aws/en/data-governance/unity-catalog/get-started#how-do-i-know-if-my-workspace-was-enabled-for-unity-catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/get-started#how-do-i-know-if-my-workspace-was-enabled-for-unity-catalog)

## Connecting the workspace

### Manual Setup

#### Step 1: Get account and workspace identifiers

- Head to the Databricks console
   - [https://accounts.cloud.databricks.com/](https://accounts.cloud.databricks.com/)
- Note down your Databricks **Account ID** from the menu on the top right profile icon
   - [https://docs.databricks.com/aws/en/admin/account-settings/#locate-your-account-id](https://docs.databricks.com/aws/en/admin/account-settings/#locate-your-account-id)
- Click **Workspaces**, select a Unity Catalog-enabled workspace
- Note down your **Workspace URL**

#### Step 2: Create a service principal

- In the workspace, select **Settings** from the top right Profile icon
- Under **Workspace admin**, select **Identity and access**
- Next to **Service Principals**, click **Manage**
- Click **Add service principal** 
   - An **Add new service principal** modal pops up, click **Add new**
   - Give the service principal a memorable name (e.g. `select-service-principal`)
- Open the newly created service principal, select **Secrets** tab, then click **Generate secret**
- Enter lifetime of 730 days, then click **Generate**
   - you will need to create a new secret and update it in SELECT when it expires
- Note down your **Secret** and **Client ID**

#### Step 3: Add the service principal to the `databricks_pii_access` group

By default, Databricks masks SQL statement text in system tables (`system.query.history.statement_text` returns `<REDACTED>`) for any principal outside the account-level `databricks_pii_access` group. Adding SELECT’s service principal to this group lets SELECT read your query text, which powers query patterns, query insights, and AI summaries.

> [!NOTE]
> **This is done in the Databricks** **_account console_, not workspace settings — you must be an account admin.** The group is account-level; creating it under Workspace admin settings will not work, and the group name is case-sensitive.

- Go to the Databricks account console ([`https://accounts.cloud.databricks.com`](https://accounts.cloud.databricks.com/), or `accounts.azuredatabricks.net` / `accounts.gcp.databricks.com` for Azure/GCP) and sign in as an **account admin**
- In the sidebar, select **User management**
- On the **Groups** tab, click **Add group**
- Enter the name exactly as `databricks_pii_access` (case-sensitive), then click **Confirm**
   - If a `databricks_pii_access` group already exists (e.g. created for another connection), skip creating it and open the existing group instead
- Open the group, click **Add members**, search for the service principal you created in Step 2 (e.g. `select-service-principal`), select it, and click **Add**
- Membership takes about **10 minutes** to propagate before query text becomes readable

#### Step 4: Create a Serverless SQL Warehouse

- Navigate to **SQL**→**SQL Warehouse** from the navigation menu
- Click **Create SQL warehouse** with the following config 
   - Name: pick a memorable name (e.g. `select-sql-warehouse`)
   - Cluster size: **2X-Small**
   - Type: **Serverless**
   - Auto-stop: set to **5 minutes** to minimize idle costs (can be reduced to 1 minute via the [Databricks Warehouses API](https://docs.databricks.com/api/workspace/warehouses/edit) after creation — the UI minimum is 5 minutes)

![](https://cdn.sanity.io/images/wn0ylp2c/production/e5e15528379e18e55639b078b3791aa9bc238e2e-1786x994.png)

- Click **Create**
- After the warehouse is created, the **Manage permissions** modal window is shown
   - You can also access this modal by clicking **Permissions**
- Search for and select the service principal you created (using the name or client id).
- Select **Can Use** permission, click **Add** and close the modal
- Note down the **Warehouse ID**

![](https://cdn.sanity.io/images/wn0ylp2c/production/e43954db2bcb3d5ac6cfe7712ad78f0636d355a9-782x592.png)

#### Step 5: Grant Read Access to `system` catalog tables

Grant the service principal the exact privileges SELECT needs:

- `USE CATALOG` on the `system` catalog
- `USE SCHEMA` on each schema we read from
- `SELECT` on each table we read from

Open a SQL editor in your Databricks workspace (attached to the warehouse created in Step 3), find/replace `<service-principal-client-id>` with the \*\*Client ID\*\* noted in Step 2, then run:

```sql
-- Catalog
grant use catalog on catalog system to `<service-principal-client-id>`;

-- Schemas
grant use schema on schema system.query    to `<service-principal-client-id>`;
grant use schema on schema system.billing  to `<service-principal-client-id>`;
grant use schema on schema system.compute  to `<service-principal-client-id>`;
grant use schema on schema system.access   to `<service-principal-client-id>`;
grant use schema on schema system.lakeflow to `<service-principal-client-id>`;

-- Tables
grant select on table system.query.history                  to `<service-principal-client-id>`;

grant select on table system.billing.usage                  to `<service-principal-client-id>`;
grant select on table system.billing.account_prices         to `<service-principal-client-id>`;  -- skip on Azure
grant select on table system.billing.list_prices            to `<service-principal-client-id>`;

grant select on table system.compute.warehouses             to `<service-principal-client-id>`;
grant select on table system.compute.warehouse_events       to `<service-principal-client-id>`;
grant select on table system.compute.clusters               to `<service-principal-client-id>`;
grant select on table system.compute.node_timeline          to `<service-principal-client-id>`;

grant select on table system.access.table_lineage           to `<service-principal-client-id>`;
grant select on table system.access.column_lineage          to `<service-principal-client-id>`;
grant select on table system.access.workspaces_latest       to `<service-principal-client-id>`;

grant select on table system.lakeflow.job_run_timeline      to `<service-principal-client-id>`;
grant select on table system.lakeflow.job_task_run_timeline to `<service-principal-client-id>`;
grant select on table system.lakeflow.jobs                  to `<service-principal-client-id>`;
grant select on table system.lakeflow.job_tasks             to `<service-principal-client-id>`;
grant select on table system.lakeflow.pipelines             to `<service-principal-client-id>`;

```

#### Step 6: Allow SELECT IP Addresses

If your Databricks workspace has configured a network policy to restrict access based on IP address, you'll need to update the network policy to allow SELECT to connect. Refer to the following documentation for more information:

- [Workspace IP Access list](https://docs.databricks.com/aws/en/security/network/front-end/ip-access-list-workspace)
- [Context-based Ingress Control](https://docs.databricks.com/aws/en/security/network/front-end/context-based-ingress)

```markdown
34.23.79.180,
34.139.189.198,
35.229.105.183,
35.243.245.125,
34.138.184.161,
34.73.135.174,
34.148.143.80,
34.23.67.112,
```

The following python code can be run in a notebook if you would like to configure this at a workspace level:

```python
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.settings import ListType

w = WorkspaceClient()

# Create a new IP allow list
created_list = w.ip_access_lists.create(
    label="SELECT-allowed-ips",
    list_type=ListType.ALLOW,
    ip_addresses=[
        "34.23.79.180",
        "34.139.189.198",
        "35.229.105.183",
        "35.243.245.125",
        "34.138.184.161",
        "34.73.135.174",
        "34.148.143.80",
        "34.23.67.112",
    ],
)
```

#### Step 7: Enabling Automated Savings

> [!NOTE]
> Automated Savings for Jobs Compute is currently in development. In the meantime, you can set up the required permissions now so you're ready when it launches.

The Automated Savings feature for All-Purpose Compute, Jobs Compute and SQL Warehouses requires granting the `CAN_MANAGE` permission to the SELECT Service Principal on each of the relevant resources. For Jobs Compute, the permissions should be granted on the Job itself rather than the compute resources which it creates. 

For Warehouses, the Service Principal also needs the `Databricks SQL access` entitlement.

Repeat this for all Clusters, Warehouses and Jobs in your workspace. For connecting to Clusters, Warehouses and Jobs in other workspaces, ensure that the service principal is added to those workspaces as well (see instructions [here](https://docs.databricks.com/aws/en/admin/users-groups/manage-service-principals?language=Account%C2%A0console#assign-a-service-principal-to-a-workspace))

For an automated way to assign permissions, see: [https://select.dev/docs/reference/faq/automated-savings/how-do-i-assign-can_manage-to-all-my-compute-resources-in-databricks](https://select.dev/docs/reference/faq/automated-savings/how-do-i-assign-can_manage-to-all-my-compute-resources-in-databricks).

**All-Purpose Compute**

![Editing permissions on All-purpose Compute](https://cdn.sanity.io/images/wn0ylp2c/production/089a79d54cd956a68a96a4550e05a976acb52ddb-800x362.gif)

**Jobs**

![Editing permissions on Jobs](https://cdn.sanity.io/images/wn0ylp2c/production/6c930827a6e59f961fe8a5be617d5a702f77fe9f-800x560.gif)

**SQL Warehouses**

![Editing permissions on SQL Warehouses](https://cdn.sanity.io/images/wn0ylp2c/production/1fda725c258a8fd20fa6c3ee52032b2b7979c940-800x429.gif)

If, as part of evaluating an All-Purpose Compute cluster for potential savings, granting the `CAN_ATTACH_TO` permission allows access to Spark metrics and better estimates of savings potential, before full management access is granted to enable the feature. The SELECT Automated Savings agent will begin monitoring all clusters automatically where this permission is granted.

### Automated setup

Running the following script will do all of the manual steps above (apart from creating the secret). This script provisions a Databricks service principal for select.dev by creating/finding the SP, assigning it to the target workspace, granting system catalog access, and setting up a SQL warehouse. If `USE_AUTOMATED_SAVINGS` is enabled, it also iterates all workspaces in the same region and grants the SP `CAN_MANAGE` permissions on all jobs, clusters, and warehouses to enable automated savings features. If `IP_ALLOW_LIST_ENABLED` is `True`, it will create an allowed list of IPs for all the workspaces as well. Copy and save to a file locally (e.g. `setup-select.py`), edit the first 10 lines with your details, and then just run `python setup-select.py`:

```python
# Dependencies: pip install databricks-sdk

# ── Config — edit these before running ───────────────────────────────────────

WORKSPACE_URL = "<https://dbc-xxxxx.cloud.databricks.com>"
ACCOUNT_ID = "<databricks-account-id>"
SERVICE_PRINCIPAL_NAME = "<service-principal-name>"
WAREHOUSE_NAME = "<sql-warehouse-name>"
IP_ALLOW_LIST_ENABLED = False
IP_ALLOW_LIST_LABEL = "select-ip-allow-list"
USE_AUTOMATED_SAVINGS = True
GRANT_PII_ACCESS = (
    True  # add SP to databricks_pii_access so SELECT reads unmasked query text
)
PII_ACCESS_GROUP_NAME = "databricks_pii_access"  # account-level, case-sensitive

# ─────────────────────────────────────────────────────────────────────────────

from urllib.parse import urlparse

from databricks.sdk import AccountClient
from databricks.sdk.errors.platform import ResourceConflict
from databricks.sdk.service import catalog as cat
from databricks.sdk.service import sql
from databricks.sdk.service.compute import ClusterSource
from databricks.sdk.service.iam import (
    AccessControlRequest,
    Patch,
    PatchOp,
    PatchSchema,
    PermissionLevel,
    WorkspacePermission,
)
from databricks.sdk.service.settings import ListType

IP_ADDRESSES = [
    "34.23.79.180",
    "34.139.189.198",
    "35.229.105.183",
    "35.243.245.125",
    "34.138.184.161",
    "34.73.135.174",
    "34.148.143.80",
    "34.23.67.112",
    "70.31.179.9",
    "216.209.96.31",
]


def main():
    parsed = urlparse(WORKSPACE_URL)
    workspace_host = f"{parsed.scheme}://{parsed.netloc}"
    account_host = detect_account_host(workspace_host)
    account_client = AccountClient(
        host=account_host, account_id=ACCOUNT_ID, auth_type="external-browser"
    )

    ws_info = find_current_workspace(account_client, workspace_host)
    ws_client = account_client.get_workspace_client(ws_info)
    region = ws_info.location or ws_info.aws_region
    print(f"Workspace : {ws_info.workspace_name}")
    print(f"Region    : {region}")

    sp = get_or_create_service_principal(account_client, SERVICE_PRINCIPAL_NAME)
    if GRANT_PII_ACCESS:
        ensure_sp_in_pii_access_group(account_client, sp)
    ensure_sp_assigned_to_workspace(account_client, ws_client, ws_info, sp)
    ensure_system_catalog_access(ws_client, sp, account_host)
    warehouse = get_or_create_warehouse(ws_client, WAREHOUSE_NAME)
    grant_permission(ws_client, "warehouses", warehouse.id, sp, PermissionLevel.CAN_USE)

    workspaces_in_region = get_workspaces_in_region(account_client, region)
    print(f"\n{len(workspaces_in_region)} workspace(s) in region '{region}'")

    for ws in workspaces_in_region:
        print(f"\n  Workspace: {ws.workspace_name}")
        ws_client = account_client.get_workspace_client(ws)

        ensure_sp_assigned_to_workspace(account_client, ws_client, ws, sp)

        if IP_ALLOW_LIST_ENABLED:
            ensure_ip_access_list(ws_client, IP_ALLOW_LIST_LABEL, IP_ADDRESSES)
        else:
            print("    Skipping IP allow list (IP_ALLOW_LIST_ENABLED=False)")

        if USE_AUTOMATED_SAVINGS:
            print(f"  Granting CAN_MANAGE to SP on jobs in {ws.workspace_name}")
            for job in ws_client.jobs.list():
                grant_permission(
                    ws_client, "jobs", job.job_id, sp, PermissionLevel.CAN_MANAGE
                )

            print(f"  Granting CAN_MANAGE to SP on clusters in {ws.workspace_name}")
            for cluster in ws_client.clusters.list():
                if cluster.cluster_source not in (ClusterSource.API, ClusterSource.UI):
                    continue
                grant_permission(
                    ws_client,
                    "clusters",
                    cluster.cluster_id,
                    sp,
                    PermissionLevel.CAN_MANAGE,
                )

            print(f"  Granting CAN_MANAGE to SP on warehouses in {ws.workspace_name}")
            for wh in ws_client.warehouses.list():
                grant_permission(
                    ws_client, "warehouses", wh.id, sp, PermissionLevel.CAN_MANAGE
                )
        else:
            print(
                "    Skipping automated savings permissions (USE_AUTOMATED_SAVINGS=False)"
            )

    secrets_url = (
        f"{account_host}/user-management/serviceprincipals/{sp.id}"
        f"/credentials-and-secrets?account_id={ACCOUNT_ID}"
    )
    print(
        "\n── Done ─────────────────────────────────────────────────────────────────────"
    )
    print(f"  Account ID   : {ACCOUNT_ID}")
    print(f"  Workspace URL: {workspace_host}")
    print(f"  Warehouse ID : {warehouse.id}")
    print(f"  SP Client ID : {sp.application_id}")
    print(f"\n  Create a client secret at:\n  {secrets_url}")
    print(
        "─────────────────────────────────────────────────────────────────────────────"
    )


# ── Helpers ───────────────────────────────────────────────────────────────────


def detect_account_host(workspace_host):
    if "azuredatabricks.net" in workspace_host:
        return "https://accounts.azuredatabricks.net"
    elif "gcp.databricks.com" in workspace_host:
        return "https://accounts.gcp.databricks.com"
    else:
        return "https://accounts.cloud.databricks.com"


def find_current_workspace(account_client, workspace_host):
    for ws in account_client.workspaces.list():
        if ws.deployment_name and ws.deployment_name in workspace_host:
            return ws
    raise ValueError(
        f"No workspace found matching '{workspace_host}'. "
        "Ensure WORKSPACE_URL contains a valid Databricks workspace URL."
    )


def get_or_create_service_principal(account_client, name):
    for s in account_client.service_principals.list():
        if s.display_name == name:
            print(f"\nFound SP '{s.display_name}' (app_id={s.application_id})")
            return s
    print(f"\nCreating service principal '{name}' …")
    sp = account_client.service_principals.create(display_name=name)
    print(f"  Created (app_id={sp.application_id})")
    return sp


def ensure_sp_in_pii_access_group(account_client, sp):
    """Add the SP to the account-level databricks_pii_access group.

    Databricks masks statement_text to <REDACTED> in system.query.history for
    principals outside this group; membership lets SELECT read the real text.

    Args:
        account_client: An authenticated Databricks AccountClient.
        sp: The service principal to add to the group.

    Returns:
        None. Creates the group if absent and adds the SP if not already a
        member; both steps are skipped when already satisfied.
    """
    group = _get_or_create_account_group(account_client, PII_ACCESS_GROUP_NAME)
    if any(str(m.value) == str(sp.id) for m in (group.members or [])):
        print(f"\nSP already in '{PII_ACCESS_GROUP_NAME}', skipping.")
        return
    print(f"\nAdding SP to '{PII_ACCESS_GROUP_NAME}' …")
    account_client.groups.patch(
        id=group.id,
        operations=[
            Patch(op=PatchOp.ADD, path="members", value=[{"value": str(sp.id)}])
        ],
        schemas=[PatchSchema.URN_IETF_PARAMS_SCIM_API_MESSAGES_2_0_PATCH_OP],
    )


def _get_or_create_account_group(account_client, name):
    """Return the account group named ``name``, creating it if absent.

    Args:
        account_client: An authenticated Databricks AccountClient.
        name: The case-sensitive account-level group display name.

    Returns:
        The group, re-fetched so its ``members`` are populated.
    """
    found = _find_account_group(account_client, name)
    if found is not None:
        return found
    print(f"\nCreating account group '{name}' …")
    try:
        return account_client.groups.create(display_name=name)
    except ResourceConflict:
        # Already exists (raced, or the server-side filter missed it) — find it.
        found = _find_account_group(account_client, name)
        if found is None:
            raise
        return found


def _find_account_group(account_client, name):
    # List and match in Python; the server-side displayName filter can miss it.
    for g in account_client.groups.list():
        if g.display_name == name:
            return account_client.groups.get(g.id)  # re-fetch to include members
    return None


# Tables SELECT replicates from the system catalog. Mirrors
SYSTEM_TABLES_BY_SCHEMA = {
    "query": ["history"],
    "billing": ["usage", "account_prices", "list_prices"],
    "serving": ["served_entities", "endpoint_usage"],
    "compute": ["warehouses", "warehouse_events", "clusters", "node_timeline"],
    "access": ["table_lineage", "column_lineage", "workspaces_latest"],
    "lakeflow": [
        "job_run_timeline",
        "job_task_run_timeline",
        "jobs",
        "job_tasks",
        "pipelines",
    ],
}
# system.billing.account_prices is only available on AWS/GCP.
AZURE_UNAVAILABLE_TABLES = {("billing", "account_prices")}


def ensure_system_catalog_access(ws_client, sp, account_host):
    principal = str(sp.application_id)
    is_azure = "azuredatabricks.net" in account_host

    print("\nGranting USE_CATALOG on 'system' …")
    _grant_unity_catalog(
        ws_client, "catalog", "system", principal, [cat.Privilege.USE_CATALOG]
    )

    for schema, tables in SYSTEM_TABLES_BY_SCHEMA.items():
        full_schema = f"system.{schema}"
        print(f"Granting USE_SCHEMA on '{full_schema}' …")
        _grant_unity_catalog(
            ws_client, "schema", full_schema, principal, [cat.Privilege.USE_SCHEMA]
        )

        for table in tables:
            if is_azure and (schema, table) in AZURE_UNAVAILABLE_TABLES:
                print(f"  Skipping system.{schema}.{table} (not available on Azure)")
                continue
            full_table = f"{full_schema}.{table}"
            print(f"  Granting SELECT on '{full_table}' …")
            _grant_unity_catalog(
                ws_client, "table", full_table, principal, [cat.Privilege.SELECT]
            )


def _grant_unity_catalog(ws_client, securable_type, full_name, principal, privileges):
    required = set(privileges)
    for pa in (
        ws_client.grants.get(securable_type, full_name).privilege_assignments or []
    ):
        if pa.principal == principal and required.issubset(set(pa.privileges or [])):
            print("    already granted, skipping.")
            return
    ws_client.grants.update(
        securable_type=securable_type,
        full_name=full_name,
        changes=[cat.PermissionsChange(principal=principal, add=list(privileges))],
    )


def get_or_create_warehouse(ws_client, name):
    for w in ws_client.warehouses.list():
        if w.name == name:
            print(f"\nFound existing warehouse '{name}' (ID: {w.id})")
            return w
    print(f"\nCreating serverless SQL warehouse '{name}' …")
    warehouse = ws_client.warehouses.create(
        name=name,
        cluster_size="2X-Small",
        enable_serverless_compute=True,
        auto_stop_mins=1,
        max_num_clusters=1,
        warehouse_type=sql.CreateWarehouseRequestWarehouseType.PRO,
    ).result()
    print(f"  Created (ID: {warehouse.id})")
    return warehouse


def get_workspaces_in_region(account_client, region):
    workspaces = []
    for ws in account_client.workspaces.list():
        if (ws.location or ws.aws_region) == region:
            workspaces.append(ws)
    return workspaces


def ensure_sp_assigned_to_workspace(account_client, ws_client, ws, sp):
    for s in ws_client.service_principals.list():
        if s.application_id == sp.application_id:
            return
    print("    Assigning SP to workspace …")
    account_client.workspace_assignment.update(
        workspace_id=ws.workspace_id,
        principal_id=int(sp.id),
        permissions=[WorkspacePermission.USER],
    )


def ensure_ip_access_list(ws_client, label, ip_addresses):
    desired = set(ip_addresses)
    for existing in ws_client.ip_access_lists.list():
        if existing.label != label:
            continue
        current = set(existing.ip_addresses or [])
        if desired.issubset(current) and existing.enabled:
            print(f"    IP allow list '{label}' already configured, skipping.")
            return
        merged = sorted(current | desired)
        print(f"    Updating IP allow list '{label}' …")
        ws_client.ip_access_lists.update(
            ip_access_list_id=existing.list_id,
            label=label,
            list_type=ListType.ALLOW,
            ip_addresses=merged,
            enabled=True,
        )
        return
    print(f"    Creating IP allow list '{label}' …")
    ws_client.ip_access_lists.create(
        label=label,
        list_type=ListType.ALLOW,
        ip_addresses=ip_addresses,
        enabled=True,
    )


def grant_permission(ws_client, resource_type, resource_id, sp, level):
    sp_app_id = str(sp.application_id)
    perms = ws_client.permissions.get(resource_type, resource_id)
    already_granted = any(
        entry.service_principal_name == sp_app_id
        and any(p.permission_level == level for p in (entry.all_permissions or []))
        for entry in (perms.access_control_list or [])
    )
    if already_granted:
        print(
            f"      {level.value} on {resource_type}/{resource_id} already granted, skipping."
        )
        return
    print(f"      Granting {level.value} on {resource_type}/{resource_id} …")
    ws_client.permissions.update(
        resource_type,
        resource_id,
        access_control_list=[
            AccessControlRequest(
                service_principal_name=sp_app_id, permission_level=level
            )
        ],
    )


if __name__ == "__main__":
    main()
```

### Add workspace to SELECT

- Navigate to settings: [https://select.dev/app/settings](https://select.dev/app/settings)
- Go to the **Databricks** tab and click the **Add Workspace** button
- Enter a memorable workspace name, and fill out all the details you have noted in the previous steps:  
   - Account ID
   - Workspace URL
   - Warehouse ID
   - Client ID
   - Client Secret
- Click **Add** and you are good to go!