All the latest updates and improvements to SELECT.
We’ve added more columns to all workloads tables, and given you the option to select which fields you want to see.
Outside of the added query performance stats, some new fields that will be particularly helpful are the “Annualized Cost Change” and “Percent Cost Change”. Use these to identify workloads which spiked in the period and will contribute the highest amount to your annual spend.
As of today, SELECT syncs your Snowflake metadata once per day at 3AM UTC. If you’re ever curious why data is not showing up for a particular day or hour, you can now check when your account was last synced at the bottom of the side bar.
If you hover over the info icon, we’ll show the detailed timestamps for when the sync last finished, and when we have data up until.
Many of our customers ask us if they can query the data we show in SELECT. Next year, we plan to support this by offering a Native Application you can install in your account. In the meantime, we’ve added new data sharing capabilities.
If you’d like to directly access the data shown in SELECT, you can request a data share to be setup for you. We have a limited number of early access spaces, so please reach out to our team if you are interested.
To help users understand how to query the datasets we surface in the data share, we've added some really cool functionality. Once enabled, you can hover over any chart or table in SELECT and you will see a little code icon appear in the top left. Copy the SQL code and run it directly in your Snowflake account. You’ll see the exact data used to populate the UI in SELECT!
Prior to this release, SELECT gave customers the ability to receive Spend Digests on a scheduled basis. A limitation of these is that if Snowflake usage spikes, you won't find out until your next scheduled digest - which could be weeks away for some users.
To help notify you immediately if your usage has spiked, we've added a new "Daily Spend Threshold Alert". In Settings -> Alerts, you can now go in and specify a daily spend threshold. If your daily usage exceeds this amount, you will be alerted. The UI gives you immediate feedback on how often you would have been alerted historical based on the amount you set.
We recommend all users set up a daily spend threshold alert, and then enable the weekly or monthly spend digest in conjunction to catch other changes in your usage patterns.
Learn more in our Alerts documentation.
We’ve updated our Fivetran integration to allow customers to configure multiple Snowflake users used for their Fivetran syncs.
Users can now search for specific workloads by name, and select more than one. Both the spend over time chart and workloads table will automatically update based on the selection. You can go a step further and “slice by” workload name to understand the contribution of each workload over time!
The demo below is for dbt, but the same functionality exists for all workloads (with the exception of Query Patterns, since they don’t have a name).
We’ve expanded support for our Usage Groups feature. SELECT customers can now define usage groups based on the role a query was executed with, dbt meta fields or tags or custom workload metadata fields.
We're quite excited about this release, as it opens the door to much more flexible chargeback/showback models!
We’re excited to announce that SELECT users can now leverage additional metadata associated with query workloads to better understand and attribute their Snowflake consumption. If you’re using our dbt or custom workloads integrations, this will apply to you!
Considering the example dbt_project.yml below, which implements a combination of dbt meta fields and tags.
# dbt_project.yml
name: 'select_dbt'
version: '1.0.0'
config-version: 2
models:
+meta:
team: 'data' # all models get tagged with team = 'data' by default
engineering:
+meta:
team: 'engineering'
marketing:
+meta:
team: 'marketing'
forecasts:
+meta:
team: 'finance'
+tags:
- "sensitive"
- "critical"
The file structure for this dbt project would look something like what is shown below. All resources under the engineering directory would be get a team meta field added with the value engineering. Similarly, the one model in the forecasts directory would get a team field of finance added. It would also have following tags: ["sensitive", "critical"].
.
├── dbt_project.yml
├── packages.yml
└── models/
├── engineering/
| ├── syncs.sql
| └── ai.sql
├── marketing/
| ├── attribition.sql
| └── page_clicks.sql
└── forecasts/
└── forecast.sql
To expose dbt meta fields in SELECT, you can go into the settings and specify which fields you want enabled. No such specification is required for dbt tags.
Once added, you can filter and slice by this metadata throught the UI!
Leverage dbt metadata in your Usage Groups
Looking to do cost chargebacks or showbacks using dbt metadata? You can now flexibly allocate costs using dbt meta fields and tags you've specified using SELECT's Usage Groups feature!
Custom workloads allow users to tag their queries with comments or Snowflake query tags. Here’s an example query comment SELECT has for one of its custom workloads:
select ...
from queries
group by all
order by sum_query_cost desc nulls last
limit 1000
/* {
"select_schema_version": "1.0.0",
"app": "SELECT_WEB_APP",
"workload_id": "DailyWorkloadPoPQuery",
"environment": "prod",
"tenant_id": "scwxhob-ad38017",
"referrer_url": "https://select.dev/app/scwxhob-ad38017/workloads",
"referrer_base_url": "https://select.dev/app/scwxhob-ad38017/workloads?startDate=2023-11-16&endDate=2023-11-22&dateGrain=day&sliceBy=nice_workload_type&sortKey=sum_query_cost&sortOrder=desc&column=name&column=workload_type&column=user_names&column=warehouse_names&column=executions",
"is_admin_user": true,
"user_id": "google-oauth2|115381118870474481840"
}*/
Custom workload fields like app, workload_id and environment are already available for filtering & slicing in the UI out of the box. To enable the same behaviour for other fields you specify in your custom workload query comment/tag, head to Settings → Custom and specify the keys you want enabled.
Once added, you can filter and slice by this metadata throught the UI!
Leverage custom meta keys in your Usage Groups
Looking to do cost chargebacks or showbacks using the metadata you've added to your query? You can now flexibly allocate costs based on custom workload metadata using SELECT's Usage Groups feature!
We don't currently support keys with complex data types
We currently do not support filtering/slicing on custom meta keys which contain complex data types like arrays or objects. If this is a requirement for you, please let us know and we will add this functionality.
select * from credit_cards
/* {
"select_schema_version": "1.0.0",
"app": "generated_tables",
"workload_id": "my_etl_job",
"tags": ["critical", "sensitive"],
"meta": {"task_key": "A1SF249X", "attempt_number": 1} }
*/
Note, you can still add this data to your query comment/tag and perform you own analysis on it. You just want be able to interact with that data in the SELECT UI as of today.
We’re excited to announce Periscope support, our fourth business intelligence (BI) tool integration.
The Periscope integration helps users keep track of Periscope dashboard costs and performance.
Dive into dashboard statistics, cost & performance trends, and understand which charts and users are driving significant usage.
SELECT doesn't require any access to your Periscope account for this feature, if you're using Periscope, you'll get to use the integration's functionality right away.
Learn how to get started in the Periscope integration documentation.
We’re excited to announce Hex support, our third business intelligence (BI) tool integration.
The Hex integration helps users keep track of Hex project costs and performance, and identify unused projects.
Use the query timeline view to understand project query durations and costs, or leverage the Cost and Performance tabs to see how project usage statistics have been trending. On the Run History tab, you’ll see a complete history of all project runs.
SELECT doesn't require any access to your Hex account for this feature, if you're using Hex, you'll get to use the integration's functionality right away.
Learn how to get started in the Hex integration documentation.
We're excited to introduce a brand new feature, Usage Groups. Usage Groups provide a flexible way of creating cost categories within SELECT. They can be used to create showbacks or chargebacks for teams in your organization. That helps:
Learn more in the Usage Groups documentation.
We’re excited to announce Mode support, our second business intelligence (BI) tool integration. As BI tooling can drive significant Snowflake usage, it’s important to understand the cost of each dashboard's queries in these tools.
The Mode integration helps users keep track of dashboard costs and performance, identify unused reports, and flag reports which may be running too frequently.
Use the query timeline view to understand dashboard performance bottlenecks or costly queries, or leverage the Cost and Performance tabs to see how report usage statistics have been trending.
On the Run History tab, you’ll see a complete history of all report runs including the users in your company who ran the report.
Learn how to get started in the Mode integration documentation.
We've made several enhancements to the warehouses page to help users explore warehouse costs and performance, and identify opportunities to consolidate warehouses:
Staying on top of your Snowflake usage can be hard. To help you more easily keep an eye on your spend trends, we created spend digests for Microsoft Teams. Spend digests are a daily/weekly/monthly message sent to a channel of your choice in Teams. They are designed to help you easily keep tabs on your Snowflake spend, see if anything spiked, and understand what caused the spike if so.
Here’s a recent example of the daily spend digest for our Snowflake account:
Anytime something spikes, you can click straight into the relevant section of the SELECT web app (i.e. the warehouse page) and understand what drove the increase (or decrease 🤞).
Head to our setup instructions for more details on how you can enable this feature.
To help users manage their Snowflake spend more effectively, we’ve added a new “Contract Overview” section to SELECT. This page is packed with helpful information to ensure you get the most value out of your credits:
The forecasted total usage is based on a linear regression forecast. Users can adjust the number of days used to create the forecast.
We’ve added the ability to filter and slice by new/existing workloads on the dbt, Looker and Custom workload tabs. This functionality allows users to easily identify new workloads that shipped in the selected time period. With the option to slice by creation recency, users can quickly understand whether a given cost spike was driven by new or existing workloads.
As we expand to support even more integrations, we recognize that we will increasingly provide options for tools that users do not use. By default all possible workload integrations are shown in the sidebar.
If you’re not using one of the tools we support, you can prevent them from appearing in the sidebar through our new settings options.
To help users contextualize costs and prioritize workloads to optimize, we’ve added a % of total cost for each workload. If you look at our dbt spend in the image below, 52% of costs come from the top 3 workloads. If we were looking to quickly reduce costs, that’s where we would focus our efforts.
We’re excited to announce Fivetran support, our first ingestion tool integration. In addition to the monthly active rows charges from Fivetran, Snowflake users must pay for the associated compute costs from Fivetran loading the data into their Snowflake account. The Fivetran integration attributes query costs to each table loaded by Fivetran. Users can leverage this information in conjunction with Fivetran's charges to understand the total cost associated with loading a given data asset.
Similar to our other integrations, users can dive into a variety of historical cost and performance statistics for each Fivetran workload, and see each historical query run.
Follow our Fivetran setup instructions to configure the integration in a few minutes. SELECT only requires the Snowflake user name used by Fivetran to load data.
By default, SELECT only shows data from complete days. We now allow users to view incomplete data in order to analyze workloads or cost data from the most recent day. Days with incomplete data are flagged in the tooltip.
Enable this functionality through the new setting in the UI.
We’ve extended our Looker integration to support viewing costs per Looker explore.
Users can easily diagnose which explores are driving Snowflake costs, and drill into them to understand information like:
In order to help users better understand the cost and performance of their dbt project(s), we’ve created a new dbt runs tab. SELECT users can now easily see each run of their dbt project, the command used to invoke the run, as well as the associated costs and duration.
As part of this, we’ve released a brand new dbt run page which contains a variety of useful information about a single dbt run, including a model timeline view. Use this view to easily spot bottlenecks in your dbt DAG which are slowing down runs and leading to ineffective warehouse utilization, and then drill into the query profile to understand how to optimize.
By default, SELECT replicates customer’s Snowflake metadata every day at 4AM UTC. We’ve added the functionality for customers to request their metadata refresh to run at a different hour of the day. Reach out to the SELECT team if you’d like to request your metadata sync to occur at a different time of day, or more frequently.
In a future release, we will expose this setting in the UI.
Many of our customers leverage SELECT to monitor multiple Snowflake accounts. We’ve revamped our settings page to segment out organization and account level settings. With this new structure, users can perform multi-account configuration such as enabling alerts across all Snowflake accounts they’ve connected to SELECT.
Data warehouse workloads often follow specific patterns based on the time of day or week. A surge of queries every Monday morning at 9AM, decreased usage on weekends, or a spike over a few hours as an expensive backfill is run. To help Snowflake users better understand their usage, we've added the ability for all timeseries plots to be viewed at the hourly level.
By analyzing trends at a more granular level, users can more easily understand their usage and identify optimization opportunities. Take this reporting warehouse as an example. There's a fairly consistent pattern of the warehosue utilization dropping below 10% every night, presenting a great opportunity for scheduled downsizing.
Snowflake's serverless features (Snowpipe, Automatic Clustering, Search Optimization, etc.) can quickly become a large portion of customers' spend. Out of the box, Snowflake does not provide any visibility or insight into the drivers behind this spend or how to reduce costs. When one of these services spike, users must manually investigate by writing and running a series of SQL queries against the account_usage views.
As part of our mission to help streamline Snowflake cost exploration and monitoring, we're excited to announce support for serverless workloads. With this launch, we're releasing out of the box support with Snowpipe and Automatic Clustering. Users can now easily understand their serverless spend drivers and monitor costs over time, with no configuration required.
In addition to this, we're providing tailored recommendations on how these serverless workloads can be optimized. For example, many customers can lower their Snowpipe costs by batching files, or by switching to a scheduled COPY INTO load.
Support for other serverless workloads such as Serverless Tasks, Search Optimization, Query Acceleration and Materialized Views will be available in a future release.
When analyzing multi-series charts, users often need to focus on a particular series, or a subset of series. Alternatively, it may be desirable to exclude a single series that contains an outlier. With our latest charting improvements, you can click one or more individual series to just focus on them. Alternatively, you can hold the shift key and the click a series to hide it.
Previously when connecting additional Snowflake accounts, users had to first file a Snowflake support ticket to have them enable the orgadmin role in that account in order for the organization_usage views to become accessible.
This step is no longer necessary, as SELECT will now leverage the organization_usage views from the primary Snowflake account. Refer to our docs for updated instructions.
We're excited to announce our brand new query page, containing SELECT's very own custom query profile. Users can now investigate query performance bottlenecks and access detailed query statistics, all without leaving the app.
The first version of the query profile we've released contains some subtle usability improvements, but largely has the same look and feel as Snowflake's query profile. Stay tuned for future updates where we'll add some powerful new capabilities!
The feature is powered by Snowflake's new get_query_operator_stats which provides programmatic access to the query profile. A big thanks to the team at Snowflake for releasing this functionality and enabling partners to build these new capabilities.
With our new support for Custom Workloads, users can now tag queries with a supported schema and gain access to a comprehensive dashboard to slice by a variety of useful dimensions, and drill into individual workload’s cost, performance and execution history.
Refer to our documentation for more details on how you can leverage the Custom Workloads feature.
We’re excited to announce Looker support, our first business intelligence (BI) tool integration. As BI tooling can drive significant Snowflake usage, it’s important to understand the cost of each dashboard's queries in these tools.
The Looker integration attributes query costs to dashboards.
Use the query timeline view to understand performance bottlenecks or costly queries, or leverage the Cost and Performance tabs to see how dashboard usage statistics have been trending. On the Run History tab, you’ll see a complete history of all dashboard runs including which user triggered the run and any error messages they saw.
Follow our Looker setup instructions to configure the integration in less than 10 minutes.
To help with monitoring performance and understanding the impact of warehouse or query changes, we’ve added three new capabilities to the warehouse performance tab.
On the warehouse performance tab, we’ve included two new views to help with warehouse sizing:
To further facilitate identifying workloads and queries of interest, we’ve made all tables in the product sortable. Sort by executions to identify high frequency queries, or by change to identify the workloads with the largest cost fluctuation.
For customers with a large number of users or warehouses in their Snowflake account, finding individual selections can become time consuming.
To enable quick filtering, we’ve added a search parameter to the dropdown filters.
Individuals are now able to invite teammates to SELECT via email directly in the product. Individual user access can now be disabled as well. To access this functionality, click the “Invite a teammate” button in the bottom left or visit the settings page.
Staying on top of your Snowflake usage can be hard. To help you more easily keep an eye on your spend trends, we created Slack spend digests. Spend digests are a daily/weekly/monthly Slack message you can enable in settings. They are designed to help you easily keep tabs on your Snowflake spend, see if anything spiked, and understand what caused the spike if so.
Here’s a recent example of the daily spend digest for our Snowflake account:
Anytime something spikes, you can click straight into the relevant section of the SELECT web app (i.e. the warehouse page) and understand what drove the increase (or decrease 🤞).
Head to our setup instructions for more details on how to get up and running.
Wherever possible, we strive for everything you need to be 1-click away. To help streamline spend investigations, we’ve added new links to the “Spend by User” and “Spend by Database” charts.
In the example below, you can see me figure out what expensive queries Niall was running in the last two weeks, and dive into the most expensive tables in our development environment.
When changing from a daily to weekly/monthly date grain, we now automatically adjust the date range to only include complete weeks/months. This is helpful to avoid having an incomplete time period, which can be very misleading.
On the dbt workloads page, you’ll see a new spend overview chart at the top breaking down your dbt spend. Similar to the workloads table, a period over period change is shown allowing you to quickly spot models that spiked, or new ones that shipped. The full run history for each dbt model is also now available, which links out to a dedicated page for each dbt run.
You can see all this new functionality in action as I investigate a recent spike in our dbt spend:
query_history_enriched_select model had a huge increase in costs that day
We’ve added a new “Change” column to all workload views, making it even easier to figure out what caused your spend to change. In the example below, I zoom into a period on our SELECT_BACKEND warehouse and can easily see which workloads spiked and which new workloads were introduced.
