dbt-snowflake-monitoring

Quickstart

Step 1: Grant access

Grant dbt's role access to the snowflake database:

grant imported privileges on database snowflake to role your_dbt_role_name;

Step 2: Update packages.yml

Add the package to your packages.yml file:

packages:

  - package: get-select/dbt_snowflake_monitoring

    version: [">=5.0.0", "<6.0.0"] # We'll never make a breaking change without creating a new major version.

Step 3: Enable query tags

  1. To attribute costs to individual models via the dbt_metadata column in the query_history_enriched model, query comments and tags are added to all dbt-issued queries. Both query comments and tags are needed to collect the required metadata for the dbt_queries model. First, make sure to remove any existing +query_tag: dbt from your dbt_project.yml, then add the query tags with one of these two options:

Option 1: If running dbt >= 1.2, simply configure the dispatch search order in dbt_project.yml.

dispatch:

  - macro_namespace: dbt

    search_order:

      - <YOUR_PROJECT_NAME>

      - dbt_snowflake_monitoring

      - dbt

Option 2: If running dbt < 1.2, create a folder named macros in your dbt project's top level directory (if it doesn't exist). Inside, make a new file called query_tags.sql with the following content:

{% macro set_query_tag() -%}

{% do return(dbt_snowflake_monitoring.set_query_tag()) %}

{% endmacro %}



{% macro unset_query_tag(original_query_tag) -%}

{% do return(dbt_snowflake_monitoring.unset_query_tag(original_query_tag)) %}

{% endmacro %}

Step 4: Enable query comments

  1. To configure the query comments, add the following config to dbt_project.yml.
query-comment:

  comment: '{{ dbt_snowflake_monitoring.get_query_comment(node) }}'

  append: true # Snowflake removes prefixed comments.

That's it! All dbt-issued queries will now be tagged and start appearing in the dbt_queries.sql model.

Building the models

By default, the models in this package will automatically get built during any full run of your dbt project. If you would like to manually create the models, or run them on a separate schedule, you can do so using dbt's node selection syntax:

dbt run -s dbt_snowflake_monitoring

dbt Cloud URLs

If you're using dbt Cloud, columns dbt_cloud_job_url and dbt_cloud_run_url can be configured in the dbt_queries model. To do so, set the variable dbt_cloud_account_id. This id can be retrieved from in between /deploy/ and /projects/ in any dbt Cloud URL.

By default, the URL prefix of https://cloud.getdbt.com/deploy/ is used. If you're using a different region of dbt Cloud, this prefix can be overridden by specifying the dbt_cloud_url variable.

Troubleshooting

No data populated in the tables

If you aren't seeing data populated in the tables (example here), it is possible that your account is missing ORGADMIN access. Please see here for instructions on enabling ORGADMIN access in your Snowflake account.