Retape (YC W23)

Saving free money on Snowflake by reducing idle time... In any cost reduction effort, deciding which activity has the highest ROI can be critical. On Snowflake, optimizing queries is the most effective way to reduce costs. However, this requires continuous effort and bandwidth. Minimizing idle time is potentially the highest ROI activity to reduce spend quickly. The first comment has a link to an SQL query to calculate the total credits you are spending on idle time. The primary advantage is that it's a 1-time solution and once implemented, it doesn't require further intervention.

Yesterday was an exciting day for us at Baselit. We launched on Y Combinator’s Launch YC platform and Hacker News at the same time. Pro tip - try not to do it on the same day, unless you love chaos. ?? In the last six months, we’ve tried to learn all there is to it to Snowflake optimization, and then condensed all those learnings into building Baselit (YC W23). We’re really proud of the end result, with happy users who don’t worry about their Snowflake spend anymore!

We did what our YC partners advised us to do - build something people want. In the current economic environment, data teams everywhere want to lower their Snowflake costs. So we decided to build Baselit (YC W23) - fully-automated Snowflake optimization, so that instead of worrying about their spend, teams can now focus on delivering business value. Thanks Y Combinator for sharing our launch!

Baselit (YC W23) automatically reduces Snowflake costs, helping data teams optimize their spend with a single click. Over the past year, Snowflake costs have become a top concern for data teams, as cost optimization is now a business priority. This optimization is usually manual - looking for opportunities like removing workloads and rewriting queries, which can be time-consuming. Founded by Ankit Goyal, Sahil Singla, and Shubham Rana, Baselit automates the optimization of Snowflake warehouses and complements the manual effort of data teams to cut costs. Congrats to the team on the launch!

In an ideal world, we should not have the concept of warehouses in Snowflake. I don’t think I have ever met a data engineer who wants to delve deep into warehouse internals when they are already juggling tasks from different business stakeholders. Ideally, warehouses in Snowflake or DBUs in Databricks are just units of compute. One should be able to define what needs to be done and the underlying data warehouse should be capable enough to figure out the size of machine and the parallelism it needs to complete my task within a particular timeframe and minimum cost. No one cares about the configuration of the warehouse if the task succeeds within the business SLA and it’s priced fairly. Though to be honest, accurately predicting the memory and runtime requirements of an SQL before actually executing is a very tough technical problem to solve. Snowflake has always been famous for ‘it just works’ and my gut says that it’ll be the first to solve this problem (if it isn't solved by academia), but for now, it remains unsolved and the users need to take care of this!?

?? 10 days - 10 features to reduce Snowflake spend 6. Reduce idle with automated agents Spindown of multicluster warehouses is very conservative. Make sure the cluster spins down once the queries are complete. 7. Custom query validation engine To prevent expensive mistakes from happening, write custom rules like Select * on large tables. 8. Horizontal scaling for STANDARD version Middleware to support scaling out of warehouses in the STANDARD version. 9. Rule-based caching policies Instead of hitting Snowflake for every use case, the use cases that can have an acceptable delay in data can be returned from a cache. 10. Automated insights Automated insights engine that skims through the Snowflake metadata and finds the inefficiencies.

Snowflake Optimization - Summary of the week 1. Remove failing workloads Failing workloads are just using credits and not performing any real work. To identify them, use this: https://lnkd.in/gP8bgV-a 2. Identify unused storage Identify and remove unused storage. Here's a blog to identify: https://lnkd.in/gMTrHxmM 3. Set query timeouts Set appropriate query timeouts based on the size of the warehouse and the business use case. 4. Split or merge files before ingestion The appropriate file size for ingestion into Snowflake is 100-250MB. Split or merge files before ingesting to ensure the file size is correct. 5. Use deferred merge If the frequency of ingestion is high, use the concept of deferred merge. Here's a blog: https://lnkd.in/gk4jgz2g

?? Day 10 of shipping 10 features in 10 days to cut down Snowflake spend! ?? Non-AI powered insights! Yeah, I am ending with Non-AI powered insights! On a serious note, we compiled all of my, Ankit Goyal's, Shubham Rana's knowledge of Snowflake and what we learned from our customers, the common mistakes that lead to an increased spend on Snowflake into an automated insight engine. It skims through your Snowflake metadata and keeps finding room for optimization automatically. Sounds interesting? Let's connect!

?? Snowflake Optimization Tip of the Day #10 ? Use Deferred Merge A significant number of customers struggle with ingestion costs, especially with MERGE queries. One of the options to solve this is using deferred merge. The concept is that instead of merging at a high frequency, you can create a base table and a delta table with the same schema that has incoming rows for this table. The downstream queries will be served by a view that is a UNION of these tables. Keep adding data to the delta table and then at some lower frequency, merge the data in the delta table to the base table. This makes sure that you are not constantly writing data and thus reducing cost of ingestion. In the first link is a blog by the data eng team of RevenueCat on how they implemented it.

?? Day 9 of shipping 10 features in 10 days to cut down Snowflake spend! ?? Rule based caching policies Let's say you are ingesting data every 5 minutes to a particular table for a critical pipeline and the same table is being used for analytics as well. The critical pipeline needs absolutely fresh data but does analytics need it too? Snowflake global cache would effectively always be invalidated and analytics results would also be recomputed. This leads to an increase in cost and because results are recomputed, the query also takes a performance hit. We solved this with a rule-based cache within Baselit. Sounds interesting? Let's connect!