We’re excited to announce that SELECT users can now leverage additional metadata associated with query workloads to better understand and attribute their Snowflake consumption. If you’re using our dbt or custom workloads integrations, this will apply to you!
Considering the example dbt_project.yml
below, which implements a combination of dbt meta fields and tags.
# dbt_project.yml name: 'select_dbt' version: '1.0.0' config-version: 2 models: +meta: team: 'data' # all models get tagged with team = 'data' by default engineering: +meta: team: 'engineering' marketing: +meta: team: 'marketing' forecasts: +meta: team: 'finance' +tags: - "sensitive" - "critical"
The file structure for this dbt project would look something like what is shown below. All resources under the engineering directory would be get a team
meta field added with the value engineering
. Similarly, the one model in the forecasts
directory would get a team
field of finance
added. It would also have following tags: ["sensitive", "critical"]
.
. ├── dbt_project.yml ├── packages.yml └── models/ ├── engineering/ | ├── syncs.sql | └── ai.sql ├── marketing/ | ├── attribition.sql | └── page_clicks.sql └── forecasts/ └── forecast.sql
To expose dbt meta fields in SELECT, you can go into the settings and specify which fields you want enabled. No such specification is required for dbt tags.
Once added, you can filter and slice by this metadata throught the UI!
Looking to do cost chargebacks or showbacks using dbt metadata? You can now flexibly allocate costs using dbt meta fields and tags you've specified using SELECT's Usage Groups feature!
Custom workloads allow users to tag their queries with comments or Snowflake query tags. Here’s an example query comment SELECT has for one of its custom workloads:
select ... from queries group by all order by sum_query_cost desc nulls last limit 1000 /* { "select_schema_version": "1.0.0", "app": "SELECT_WEB_APP", "workload_id": "DailyWorkloadPoPQuery", "environment": "prod", "tenant_id": "scwxhob-ad38017", "referrer_base_url": "https://select.dev/app/scwxhob-ad38017/workloads", "is_admin_user": true, "user_id": "google-oauth2|115381118870474481840" }*/
Custom workload fields like app
, workload_id
and environment
are already available for filtering & slicing in the UI out of the box. To enable the same behaviour for other fields you specify in your custom workload query comment/tag, head to Settings → Custom and specify the keys you want enabled.
Once added, you can filter and slice by this metadata throught the UI!
Looking to do cost chargebacks or showbacks using the metadata you've added to your query? You can now flexibly allocate costs based on custom workload metadata using SELECT's Usage Groups feature!
We currently do not support filtering/slicing on custom meta keys which contain complex data types like arrays or objects. If this is a requirement for you, please let us know and we will add this functionality.
select * from credit_cards /* { "select_schema_version": "1.0.0", "app": "generated_tables", "workload_id": "my_etl_job", "tags": ["critical", "sensitive"], "meta": {"task_key": "A1SF249X", "attempt_number": 1} }
Note, you can still add this data to your query comment/tag and perform you own analysis on it. You just want be able to interact with that data in the SELECT UI as of today.
Get up and running with SELECT in 15 minutes.
Gain visibility into Snowflake usage, optimize performance and automate savings with the click of a button.