Leverage custom top-level dbt keys
- Date
Niall WoodwardCo-founder & CTO of SELECT
Similar to how we enable users to leverage the meta keys / tags they've configured in their dbt project, users can now filter and slice by any custom top-level query comment/tag keys. Once you've specified which keys you want to access in the settings, you'll be able to access them as filter/slicing options throughout SELECT.
What are custom top-level keys?
Consider the following example dbt model, queries:
{{ config(
materialized='incremental',
meta={'team': 'data'}
) }}
select *
from {{ ref('stg_query_history') }}
Once this model is run and the dbt project has our dbt-snowflake-query-tags package installed, the SQL will look like this:
select *
from analytics.stg_query_history
/* {"app": "dbt", "dbt_snowflake_query_tags_version": "2.3.2", "dbt_version": "1.7.5", "project_name": "select_dbt", "target_name": "prod", "target_database": "SELECT_INGEST", "target_schema": "analytics", "invocation_id": "ac36fd96-48eb-4ab8-b585-f4da8d7e0de5", "node_name": "queries", "node_alias": "queries", "node_package_name": "select_dbt", "node_original_file_path": "models/queries.sql", "node_database": "SELECT_INGEST", "node_schema": "analytics", "node_id": "model.select_dbt.queries", "node_resource_type": "model", "node_meta": {"team": "data"}, "node_tags": [], "full_refresh": false, "which": "build", "invocation_command": "dbt build --vars {'source_schema': 'analytics', 'snowflake_organization_account_source_schema': 'analytics', 'destination_schema': 'analytics'} --target prod --select fqn:*", "node_refs": ["stg_query_history", ], "materialized": "incremental"} */;
It's possible for users to add additional top level keys to the query comment metadata. A custom top level key would be anything not included by default in the comment above. The internal_job_name key (with a value of hourly_run) in the comment below would be an example of this:
select *
from analytics.stg_query_history
/* {"app": "dbt", "dbt_snowflake_query_tags_version": "2.3.2", "dbt_version": "1.7.5", "project_name": "select_dbt", "target_name": "prod", "target_database": "SELECT_INGEST", "target_schema": "analytics", "invocation_id": "ac36fd96-48eb-4ab8-b585-f4da8d7e0de5", "node_name": "queries", "node_alias": "queries", "node_package_name": "select_dbt", "node_original_file_path": "models/queries.sql", "node_database": "SELECT_INGEST", "node_schema": "analytics", "node_id": "model.select_dbt.queries", "node_resource_type": "model", "node_meta": {"team": "data"}, "node_tags": [], "full_refresh": false, "which": "build", "invocation_command": "dbt build --vars {'source_schema': 'analytics', 'snowflake_organization_account_source_schema': 'analytics', 'destination_schema': 'analytics'} --target prod --select fqn:*", "node_refs": ["stg_query_history", ], "materialized": "incremental", "internal_job_name": "hourly_run"} */;