Changelog

All the latest updates and improvements to SELECT.

Sign up to receive product updates

We’ve added more columns to all workloads tables, and given you the option to select which fields you want to see.

Outside of the added query performance stats, some new fields that will be particularly helpful are the “Annualized Cost Change” and “Percent Cost Change”. Use these to identify workloads which spiked in the period and will contribute the highest amount to your annual spend.

Date

As of today, SELECT syncs your Snowflake metadata once per day at 3AM UTC. If you’re ever curious why data is not showing up for a particular day or hour, you can now check when your account was last synced at the bottom of the side bar.

SELECT last sync time

If you hover over the info icon, we’ll show the detailed timestamps for when the sync last finished, and when we have data up until.

SELECT last sync time
Date

Many of our customers ask us if they can query the data we show in SELECT. Next year, we plan to support this by offering a Native Application you can install in your account. In the meantime, we’ve added new data sharing capabilities.

If you’d like to directly access the data shown in SELECT, you can request a data share to be setup for you. We have a limited number of early access spaces, so please reach out to our team if you are interested.

To help users understand how to query the datasets we surface in the data share, we've added some really cool functionality. Once enabled, you can hover over any chart or table in SELECT and you will see a little code icon appear in the top left. Copy the SQL code and run it directly in your Snowflake account. You’ll see the exact data used to populate the UI in SELECT!

Date

Prior to this release, SELECT gave customers the ability to receive Spend Digests on a scheduled basis. A limitation of these is that if Snowflake usage spikes, you won't find out until your next scheduled digest - which could be weeks away for some users.

To help notify you immediately if your usage has spiked, we've added a new "Daily Spend Threshold Alert". In Settings -> Alerts, you can now go in and specify a daily spend threshold. If your daily usage exceeds this amount, you will be alerted. The UI gives you immediate feedback on how often you would have been alerted historical based on the amount you set.

We recommend all users set up a daily spend threshold alert, and then enable the weekly or monthly spend digest in conjunction to catch other changes in your usage patterns.

Learn more in our Alerts documentation.

Date

Users can now search for specific workloads by name, and select more than one. Both the spend over time chart and workloads table will automatically update based on the selection. You can go a step further and “slice by” workload name to understand the contribution of each workload over time!

The demo below is for dbt, but the same functionality exists for all workloads (with the exception of Query Patterns, since they don’t have a name).

Date

We’re excited to announce that SELECT users can now leverage additional metadata associated with query workloads to better understand and attribute their Snowflake consumption. If you’re using our dbt or custom workloads integrations, this will apply to you!

Leveraging dbt meta & tags

Considering the example dbt_project.yml below, which implements a combination of dbt meta fields and tags.

# dbt_project.yml
name: 'select_dbt'
version: '1.0.0'
config-version: 2
models:
+meta:
team: 'data' # all models get tagged with team = 'data' by default
engineering:
+meta:
team: 'engineering'
marketing:
+meta:
team: 'marketing'
forecasts:
+meta:
team: 'finance'
+tags:
- "sensitive"
- "critical"

The file structure for this dbt project would look something like what is shown below. All resources under the engineering directory would be get a team meta field added with the value engineering. Similarly, the one model in the forecasts directory would get a team field of finance added. It would also have following tags: ["sensitive", "critical"].

.
├── dbt_project.yml
├── packages.yml
└── models/
├── engineering/
| ├── syncs.sql
| └── ai.sql
├── marketing/
| ├── attribition.sql
| └── page_clicks.sql
└── forecasts/
└── forecast.sql

To expose dbt meta fields in SELECT, you can go into the settings and specify which fields you want enabled. No such specification is required for dbt tags.

SELECT dbt meta keys settings

Once added, you can filter and slice by this metadata throught the UI!

Leverage dbt metadata in your Usage Groups

Looking to do cost chargebacks or showbacks using dbt metadata? You can now flexibly allocate costs using dbt meta fields and tags you've specified using SELECT's Usage Groups feature!

Leveraging custom metadata fields

Custom workloads allow users to tag their queries with comments or Snowflake query tags. Here’s an example query comment SELECT has for one of its custom workloads:

select ...
from queries
group by all
order by sum_query_cost desc nulls last
limit 1000
/* {
"select_schema_version": "1.0.0",
"app": "SELECT_WEB_APP",
"workload_id": "DailyWorkloadPoPQuery",
"environment": "prod",
"tenant_id": "scwxhob-ad38017",
"referrer_url": "https://select.dev/app/scwxhob-ad38017/workloads",
"referrer_base_url": "https://select.dev/app/scwxhob-ad38017/workloads?startDate=2023-11-16&endDate=2023-11-22&dateGrain=day&sliceBy=nice_workload_type&sortKey=sum_query_cost&sortOrder=desc&column=name&column=workload_type&column=user_names&column=warehouse_names&column=executions",
"is_admin_user": true,
"user_id": "google-oauth2|115381118870474481840"
}*/

Custom workload fields like app, workload_id and environment are already available for filtering & slicing in the UI out of the box. To enable the same behaviour for other fields you specify in your custom workload query comment/tag, head to Settings → Custom and specify the keys you want enabled.

SELECT custom workload meta keys settings

Once added, you can filter and slice by this metadata throught the UI!

Leverage custom meta keys in your Usage Groups

Looking to do cost chargebacks or showbacks using the metadata you've added to your query? You can now flexibly allocate costs based on custom workload metadata using SELECT's Usage Groups feature!

We don't currently support keys with complex data types

We currently do not support filtering/slicing on custom meta keys which contain complex data types like arrays or objects. If this is a requirement for you, please let us know and we will add this functionality.

select * from credit_cards
/* {
"select_schema_version": "1.0.0",
"app": "generated_tables",
"workload_id": "my_etl_job",
"tags": ["critical", "sensitive"],
"meta": {"task_key": "A1SF249X", "attempt_number": 1} }
*/

Note, you can still add this data to your query comment/tag and perform you own analysis on it. You just want be able to interact with that data in the SELECT UI as of today.

Date

We’re excited to announce Periscope support, our fourth business intelligence (BI) tool integration.

The Periscope integration helps users keep track of Periscope dashboard costs and performance.

Periscope integration

Dive into dashboard statistics, cost & performance trends, and understand which charts and users are driving significant usage.

Periscope integration

SELECT doesn't require any access to your Periscope account for this feature, if you're using Periscope, you'll get to use the integration's functionality right away.

Learn how to get started in the Periscope integration documentation.

Date

We’re excited to announce Hex support, our third business intelligence (BI) tool integration.

The Hex integration helps users keep track of Hex project costs and performance, and identify unused projects.

Hex integration

Use the query timeline view to understand project query durations and costs, or leverage the Cost and Performance tabs to see how project usage statistics have been trending. On the Run History tab, you’ll see a complete history of all project runs.

Hex integration

SELECT doesn't require any access to your Hex account for this feature, if you're using Hex, you'll get to use the integration's functionality right away.

Learn how to get started in the Hex integration documentation.

Date

We're excited to introduce a brand new feature, Usage Groups. Usage Groups provide a flexible way of creating cost categories within SELECT. They can be used to create showbacks or chargebacks for teams in your organization. That helps:

  1. Reduce the burden on central teams playing the role of 'query cop' 🚓
  2. Distribute costs across organizational units
  3. Encourage efficient use of Snowflake
  4. Flexibly allocate costs for shared warehouses

Learn more in the Usage Groups documentation.

Date

We’re excited to announce Mode support, our second business intelligence (BI) tool integration. As BI tooling can drive significant Snowflake usage, it’s important to understand the cost of each dashboard's queries in these tools.

The Mode integration helps users keep track of dashboard costs and performance, identify unused reports, and flag reports which may be running too frequently.

Use the query timeline view to understand dashboard performance bottlenecks or costly queries, or leverage the Cost and Performance tabs to see how report usage statistics have been trending.

SELECT Mode integration timeline view

On the Run History tab, you’ll see a complete history of all report runs including the users in your company who ran the report.

SELECT Mode integration run history

Learn how to get started in the Mode integration documentation.

Date

We've made several enhancements to the warehouses page to help users explore warehouse costs and performance, and identify opportunities to consolidate warehouses:

  1. Chart added above the warehouses table. Users can filter to a subset of warehouses and plot a variety of different metrics: Spend, Automated Savings, Query Counts and Utilization Efficiency.
  2. Utilization Efficiency has been added as a column to the warehouses table. Users can leverage this to identify warehouses which should be combined to improve their utilization.
SELECT Snowflake warehouses page enhancements
Date

Staying on top of your Snowflake usage can be hard. To help you more easily keep an eye on your spend trends, we created spend digests for Microsoft Teams. Spend digests are a daily/weekly/monthly message sent to a channel of your choice in Teams. They are designed to help you easily keep tabs on your Snowflake spend, see if anything spiked, and understand what caused the spike if so.

Here’s a recent example of the daily spend digest for our Snowflake account:

SELECT Snowflake spend alerts in Microsoft Teams

Anytime something spikes, you can click straight into the relevant section of the SELECT web app (i.e. the warehouse page) and understand what drove the increase (or decrease 🤞).

Head to our setup instructions for more details on how you can enable this feature.

Date

To help users manage their Snowflake spend more effectively, we’ve added a new “Contract Overview” section to SELECT. This page is packed with helpful information to ensure you get the most value out of your credits:

  • Understand your active contract capacity and when your contract expires. The tooltip reveals more information about how much capacity was purchased vs rolled over.
  • Understand how much capacity you’ve used, and the forecasted total usage at the end of the contract.
  • Understand whether you’ll have to renew your contract early and the savings necessary to avoid an early renewal. Alternatively, budget surplus capacity to new workloads that you could bring into Snowflake or other performance optimizations.
SELECT Snowflake contract capacity monitoring

The forecasted total usage is based on a linear regression forecast. Users can adjust the number of days used to create the forecast.

Date

We’ve added the ability to filter and slice by new/existing workloads on the dbt, Looker and Custom workload tabs. This functionality allows users to easily identify new workloads that shipped in the selected time period. With the option to slice by creation recency, users can quickly understand whether a given cost spike was driven by new or existing workloads.

Date

As we expand to support even more integrations, we recognize that we will increasingly provide options for tools that users do not use. By default all possible workload integrations are shown in the sidebar.

SELECT workload cost percent of total

If you’re not using one of the tools we support, you can prevent them from appearing in the sidebar through our new settings options.

SELECT workload cost percent of total
Date

To help users contextualize costs and prioritize workloads to optimize, we’ve added a % of total cost for each workload. If you look at our dbt spend in the image below, 52% of costs come from the top 3 workloads. If we were looking to quickly reduce costs, that’s where we would focus our efforts.

SELECT workload cost percent of total
Date

We’re excited to announce Fivetran support, our first ingestion tool integration. In addition to the monthly active rows charges from Fivetran, Snowflake users must pay for the associated compute costs from Fivetran loading the data into their Snowflake account. The Fivetran integration attributes query costs to each table loaded by Fivetran. Users can leverage this information in conjunction with Fivetran's charges to understand the total cost associated with loading a given data asset.

SELECT Fivetran integration dashboard overview part 1

Similar to our other integrations, users can dive into a variety of historical cost and performance statistics for each Fivetran workload, and see each historical query run.

SELECT Fivetran integration dashboard overview part 2

Follow our Fivetran setup instructions to configure the integration in a few minutes. SELECT only requires the Snowflake user name used by Fivetran to load data.

Date

By default, SELECT only shows data from complete days. We now allow users to view incomplete data in order to analyze workloads or cost data from the most recent day. Days with incomplete data are flagged in the tooltip.

SELECT cost per looker explore

Enable this functionality through the new setting in the UI.

SELECT cost per looker explore
Date

We’ve extended our Looker integration to support viewing costs per Looker explore.

SELECT cost per looker explore

Users can easily diagnose which explores are driving Snowflake costs, and drill into them to understand information like:

  • How often is this explore being used
  • How are performance and costs trending over time
  • Which users are leveraging this explore
  • Query profile characteristics for all historical executions
SELECT cost per looker explore
Date

In order to help users better understand the cost and performance of their dbt project(s), we’ve created a new dbt runs tab. SELECT users can now easily see each run of their dbt project, the command used to invoke the run, as well as the associated costs and duration.

SELECT dbt runs page

As part of this, we’ve released a brand new dbt run page which contains a variety of useful information about a single dbt run, including a model timeline view. Use this view to easily spot bottlenecks in your dbt DAG which are slowing down runs and leading to ineffective warehouse utilization, and then drill into the query profile to understand how to optimize.

Date

By default, SELECT replicates customer’s Snowflake metadata every day at 4AM UTC. We’ve added the functionality for customers to request their metadata refresh to run at a different hour of the day. Reach out to the SELECT team if you’d like to request your metadata sync to occur at a different time of day, or more frequently.

In a future release, we will expose this setting in the UI.

Date

Many of our customers leverage SELECT to monitor multiple Snowflake accounts. We’ve revamped our settings page to segment out organization and account level settings. With this new structure, users can perform multi-account configuration such as enabling alerts across all Snowflake accounts they’ve connected to SELECT.

SELECT organization settings
Date

Data warehouse workloads often follow specific patterns based on the time of day or week. A surge of queries every Monday morning at 9AM, decreased usage on weekends, or a spike over a few hours as an expensive backfill is run. To help Snowflake users better understand their usage, we've added the ability for all timeseries plots to be viewed at the hourly level.

SELECT timeseries granularity dropdown selector

By analyzing trends at a more granular level, users can more easily understand their usage and identify optimization opportunities. Take this reporting warehouse as an example. There's a fairly consistent pattern of the warehosue utilization dropping below 10% every night, presenting a great opportunity for scheduled downsizing.

SELECT now supports hourly granularity in all timeseries plots
Date

Snowflake's serverless features (Snowpipe, Automatic Clustering, Search Optimization, etc.) can quickly become a large portion of customers' spend. Out of the box, Snowflake does not provide any visibility or insight into the drivers behind this spend or how to reduce costs. When one of these services spike, users must manually investigate by writing and running a series of SQL queries against the account_usage views.

As part of our mission to help streamline Snowflake cost exploration and monitoring, we're excited to announce support for serverless workloads. With this launch, we're releasing out of the box support with Snowpipe and Automatic Clustering. Users can now easily understand their serverless spend drivers and monitor costs over time, with no configuration required.

SELECT serverless workloads integration dashboard overview

In addition to this, we're providing tailored recommendations on how these serverless workloads can be optimized. For example, many customers can lower their Snowpipe costs by batching files, or by switching to a scheduled COPY INTO load.

SELECT snowpipe recommendations

Support for other serverless workloads such as Serverless Tasks, Search Optimization, Query Acceleration and Materialized Views will be available in a future release.

Date

When analyzing multi-series charts, users often need to focus on a particular series, or a subset of series. Alternatively, it may be desirable to exclude a single series that contains an outlier. With our latest charting improvements, you can click one or more individual series to just focus on them. Alternatively, you can hold the shift key and the click a series to hide it.

Date

Previously when connecting additional Snowflake accounts, users had to first file a Snowflake support ticket to have them enable the orgadmin role in that account in order for the organization_usage views to become accessible.

This step is no longer necessary, as SELECT will now leverage the organization_usage views from the primary Snowflake account. Refer to our docs for updated instructions.

Date

We're excited to announce our brand new query page, containing SELECT's very own custom query profile. Users can now investigate query performance bottlenecks and access detailed query statistics, all without leaving the app.

The first version of the query profile we've released contains some subtle usability improvements, but largely has the same look and feel as Snowflake's query profile. Stay tuned for future updates where we'll add some powerful new capabilities!

The feature is powered by Snowflake's new get_query_operator_stats which provides programmatic access to the query profile. A big thanks to the team at Snowflake for releasing this functionality and enabling partners to build these new capabilities.

Date

We’re excited to announce Looker support, our first business intelligence (BI) tool integration. As BI tooling can drive significant Snowflake usage, it’s important to understand the cost of each dashboard's queries in these tools.

The Looker integration attributes query costs to dashboards.

SELECT Looker integration dashboard overview

Use the query timeline view to understand performance bottlenecks or costly queries, or leverage the Cost and Performance tabs to see how dashboard usage statistics have been trending. On the Run History tab, you’ll see a complete history of all dashboard runs including which user triggered the run and any error messages they saw.

SELECT Looker integration dashboard overview

Follow our Looker setup instructions to configure the integration in less than 10 minutes.

Date

To help with monitoring performance and understanding the impact of warehouse or query changes, we’ve added three new capabilities to the warehouse performance tab.

Monitor a larger variety of aggregation types (avg, p50 through p99) across any metric.

SELECT warehouse performance change metric

Analyze query times with any aggregation type (min, max, avg, p50 through p99).

SELECT warehouse performance change aggregation type

Monitor query result cache usage rates to better understand performance.

SELECT warehouse performance cache usage
Date

On the warehouse performance tab, we’ve included two new views to help with warehouse sizing:

  1. The number of queries by execution time. Here you can see that over 98% of the queries running on this warehouse are taking less than 1 second to execute.
  2. The number of queries by utilizable warehouse size. Utilizable warehouse size represents the size of warehouse a query can fully utilize. The value is always capped at the query's current warehouse size. Where lots of queries don't utilize the warehouse's size, it indicates that the warehouse is oversized or the queries should run on a smaller warehouse. In this example, over 96% of queries being run on the warehouse aren’t using all 8 nodes available in the warehouse.
SELECT warehouse performance views
Date

To further facilitate identifying workloads and queries of interest, we’ve made all tables in the product sortable. Sort by executions to identify high frequency queries, or by change to identify the workloads with the largest cost fluctuation.

SELECT searchable dropdown selectors
Date

For customers with a large number of users or warehouses in their Snowflake account, finding individual selections can become time consuming.

To enable quick filtering, we’ve added a search parameter to the dropdown filters.

SELECT searchable dropdown selectors
Date

Individuals are now able to invite teammates to SELECT via email directly in the product. Individual user access can now be disabled as well. To access this functionality, click the “Invite a teammate” button in the bottom left or visit the settings page.

Date

Staying on top of your Snowflake usage can be hard. To help you more easily keep an eye on your spend trends, we created Slack spend digests. Spend digests are a daily/weekly/monthly Slack message you can enable in settings. They are designed to help you easily keep tabs on your Snowflake spend, see if anything spiked, and understand what caused the spike if so.

Here’s a recent example of the daily spend digest for our Snowflake account:

SELECT Snowflake spend alerts in slack

Anytime something spikes, you can click straight into the relevant section of the SELECT web app (i.e. the warehouse page) and understand what drove the increase (or decrease 🤞).

Head to our setup instructions for more details on how to get up and running.

Date

Wherever possible, we strive for everything you need to be 1-click away. To help streamline spend investigations, we’ve added new links to the “Spend by User” and “Spend by Database” charts.

In the example below, you can see me figure out what expensive queries Niall was running in the last two weeks, and dive into the most expensive tables in our development environment.

SELECT added chart links
Date

When changing from a daily to weekly/monthly date grain, we now automatically adjust the date range to only include complete weeks/months. This is helpful to avoid having an incomplete time period, which can be very misleading.

SELECT new date range defaults
Date

On the dbt workloads page, you’ll see a new spend overview chart at the top breaking down your dbt spend. Similar to the workloads table, a period over period change is shown allowing you to quickly spot models that spiked, or new ones that shipped. The full run history for each dbt model is also now available, which links out to a dedicated page for each dbt run.

Demo

You can see all this new functionality in action as I investigate a recent spike in our dbt spend:

  • I zoom into the spike on March 11
  • I see that the query_history_enriched_select model had a huge increase in costs that day
  • After opening up that model page, I click on the new “Run History” tab
  • I see that there were a bunch of full refreshes, including many that failed
  • I click one of the failed runs which takes me to the new “dbt Run” page
  • Here I can see the exact SQL ran, which step of the model failed and all relevant execution details
SELECT dbt model history
Date

We’ve added a new “Change” column to all workload views, making it even easier to figure out what caused your spend to change. In the example below, I zoom into a period on our SELECT_BACKEND warehouse and can easily see which workloads spiked and which new workloads were introduced.

SELECT workload change
Date