Why we migrated from Redshift to Snowflake?

We were using an Amazon Redshift as data warehouse solution before we moved our data platform to cloud. It was obvious choice for our growing IOT data platform. Redshift made it easy for us to pull data from variety of sources using ETL and build a foundation for supporting critical business application. It satisfied our needs at that point of time.

However as we grew and started on boarding variety and volume of data, our data platform became more complex, we began having performance issues like long query wait times, crashes, and lockups, real time api latency issues etc. We’d have to restart our Redshift cluster, kick off manual runs of dbt, manually cancel queries to get things moving.

We might have been able to mitigate these issues if we’d hired a dedicate database administrator to actively manage Redshift cluster, but again it contradicts with our original ideal of adapting fully managed cloud service. At the end of the day, we weren’t able to get the level of reliability we needed from Redshift. So we started looking for other fully managed cloud based solution.

Snowflake's powerful features promised to help our warehouse perform well even if we began to experience heavy loads. Right away, we saw several Snowflake features that we were eager to take advantage of:

Separate compute and storage: Snowflake is a data warehouse-as-a-service, which requires no management and features separate compute, storage, and cloud services that can scale and change independently.

Native support for structured and semi structured data: Snowflake’s data warehouse architecture provides complete relational database support for both structured data, such as CSV files and tables, and semi-structured data, including JSON, Avro, Parquet, etc., all within a single, logically integrated solution.

Data Sharing: We can securely share datasets with anyone in or outside of our organization. This comes in handy now that Stitch is part of Talend. When you share data in Snowflake, it doesn’t move any data from S3, other folks just get access.

Time Travel: It lets us instantly roll back the entire warehouse to any point in time during a chosen retention window.

Today our migration is complete. We now have fresher data, lower query wait times, and improved real time api latency, We're happy to have this migration. Snowflake is capable of delivering more power than we're using,