View full changelog

Slice, filter and attribute costs using query metadata

Date
  • Niall Woodward
    Co-founder & CTO of SELECT

We’re excited to announce that SELECT users can now leverage additional metadata associated with query workloads to better understand and attribute their Snowflake consumption. If you’re using our dbt or custom workloads integrations, this will apply to you!

Leveraging dbt meta & tags

Considering the example dbt_project.yml below, which implements a combination of dbt meta fields and tags.

# dbt_project.yml
name: 'select_dbt'
version: '1.0.0'
config-version: 2
models:
+meta:
team: 'data' # all models get tagged with team = 'data' by default
engineering:
+meta:
team: 'engineering'
marketing:
+meta:
team: 'marketing'
forecasts:
+meta:
team: 'finance'
+tags:
- "sensitive"
- "critical"

The file structure for this dbt project would look something like what is shown below. All resources under the engineering directory would be get a team meta field added with the value engineering. Similarly, the one model in the forecasts directory would get a team field of finance added. It would also have following tags: ["sensitive", "critical"].

.
├── dbt_project.yml
├── packages.yml
└── models/
├── engineering/
| ├── syncs.sql
| └── ai.sql
├── marketing/
| ├── attribition.sql
| └── page_clicks.sql
└── forecasts/
└── forecast.sql

To expose dbt meta fields in SELECT, you can go into the settings and specify which fields you want enabled. No such specification is required for dbt tags.

SELECT dbt meta keys settings

Once added, you can filter and slice by this metadata throught the UI!

Leverage dbt metadata in your Usage Groups

Looking to do cost chargebacks or showbacks using dbt metadata? You can now flexibly allocate costs using dbt meta fields and tags you've specified using SELECT's Usage Groups feature!

Leveraging custom metadata fields

Custom workloads allow users to tag their queries with comments or Snowflake query tags. Here’s an example query comment SELECT has for one of its custom workloads:

select ...
from queries
group by all
order by sum_query_cost desc nulls last
limit 1000
/* {
"select_schema_version": "1.0.0",
"app": "SELECT_WEB_APP",
"workload_id": "DailyWorkloadPoPQuery",
"environment": "prod",
"tenant_id": "scwxhob-ad38017",
"referrer_base_url": "https://select.dev/app/scwxhob-ad38017/workloads",
"is_admin_user": true,
"user_id": "google-oauth2|115381118870474481840"
}*/

Custom workload fields like app, workload_id and environment are already available for filtering & slicing in the UI out of the box. To enable the same behaviour for other fields you specify in your custom workload query comment/tag, head to Settings → Custom and specify the keys you want enabled.

SELECT custom workload meta keys settings

Once added, you can filter and slice by this metadata throught the UI!

Leverage custom meta keys in your Usage Groups

Looking to do cost chargebacks or showbacks using the metadata you've added to your query? You can now flexibly allocate costs based on custom workload metadata using SELECT's Usage Groups feature!

We don't currently support keys with complex data types

We currently do not support filtering/slicing on custom meta keys which contain complex data types like arrays or objects. If this is a requirement for you, please let us know and we will add this functionality.

select * from credit_cards
/* {
"select_schema_version": "1.0.0",
"app": "generated_tables",
"workload_id": "my_etl_job",
"tags": ["critical", "sensitive"],
"meta": {"task_key": "A1SF249X", "attempt_number": 1} }
*/

Note, you can still add this data to your query comment/tag and perform you own analysis on it. You just want be able to interact with that data in the SELECT UI as of today.

Get up and running with SELECT in 15 minutes.

Automatically optimize, understand and monitor Snowflake usage

Gain visibility into Snowflake usage, optimize performance and automate savings with the click of a button.

SELECT web application screenshot

Want to get notified when we release new features? 
Subscribe to get notified.