DoubleCloud’s 13th Product Update
Hey everyone! Victor here with the latest news.
ClickHouse and Apache Kafka compute vertical autoscaling
Finally, the long-awaited autoscaling for compute has arrived. We?are introducing compute vertical autoscaling in?preview for all users of?our Managed ClickHouse and Managed Apache Kafka services.
With disk autoresizing and compute autoscaling combined, you can now stop worrying about your service being under-provisioned during spikes in?load or?high-traffic days and hours. Once enabled, we?will automatically monitor the CPU load on?instances and scale it?up?during sustained high loads on?a?cluster over the last 5?minutes. When the load decreases, it?will automatically scale back down. This feature will help maintain the necessary size of?the cluster and avoid paying for over-provisioned resources when they are not needed.
You can explicitly enable this feature while creating a?cluster or?modifying an?existing?one. You can find more details about this new functionality in?the docs.
Airflow custom container images
You can now customize container images used by?Airflow workers. Airflow is?employed in?various scenarios, such as?orchestrating data pipelines or?machine learning pipelines. In?some cases, you may need to?perform calculations or?use Airflow workers for data transformations, among other tasks. These scenarios often require customizing your workers' runtime environment to?include extra packages like scikit-learn, pandas, or?specific SDKs/drivers for connecting to?external services or?systems.
Now, you can create a?custom container image based on?the default Docker image, push it?to?our registry, and use it?as?the runtime for your workloads.
For more details on?how to?set up?custom images, please refer to?our documentation.
Permissions and users in?ClickHouse
Managing users and privileges in?ClickHouse is?now possible through our console UI, API, and soon via Terraform. You can effortlessly create and manage users and their permissions directly in?the UI?without connecting to?ClickHouse instances through a?terminal. Additionally, you can modify user settings, establish new roles with various privileges, and perform other related tasks through our new?UI. This feature is?accessible from the cluster page under 'Access Control'.
For more information, please check out our step-by-step instructions.
Organization member groups
To?simplify the management of?many users and their permissions, we?have introduced IAM Groups in?the organizational view.
You can now create a?new group, add current members, and assign specific permissions with project-level granularity. For example, the accompanying image illustrates how you can easily create a?user group that can manage clusters in?a?particular project but will only have viewer permissions for the Visualization service.
For more details on?managing groups in?our console, please see here.
Transfer service progressive discounts
One of?the primary uses of?the DoubleCloud Transfer service is?the synchronization of?data from PostgreSQL to?ClickHouse or?from Apache Kafka to?ClickHouse. In?these cases, users typically stream large volumes of?data at?high speeds, ranging from 1?to?10?megabytes per second, which incurred high costs previously.
We?have decided to?revise our pricing structure and introduce dynamic pricing discounts to?align with this usage pattern. Now, the more data you process, the less you?pay. Starting at?$1.20 per 1?million processed rows, the price gradually decreases to?$0.20 after the number of?rows exceeds 400M.
Take a?look at?our new pricing structure:
BigQuery as?target connector
In?addition to?our 21?source and 12?target connectors, we?have launched a?new one: BigQuery. This connector can export data from the ClickHouse or?Apache Kafka service to?BigQuery. Please find more details here.
New ISO 27001:2022 certification
We?at?DoubleCloud take threats to?the availability, integrity, and confidentiality of?our clients’ information seriously. As?such, DoubleCloud is?an?ISO/IEC 27001:2022 certified provider whose Information Security Management System (ISMS) has received third-party accreditation from the International Organization for Standardization (ISO).
This certification demonstrates DoubleCloud’s continued commitment to?information security at?every level. It?ensures that the security of?your data and information has been addressed, implemented, and properly controlled in?all areas of?our organization.
To?view all our certifications or?to?request assessment access, please visit our Trust Center.
Visualization service UI/UX update
We?recently updated our DoubleCloud Visualization service by?incorporating many features from the open-source version released over the last few months. Below is?subset of?things that were changed:
The complete list of?changes is?available here, covering the period from December 21, 2023, to?March 16, 2024.
Quality of?life improvements
Here’s a?list of?some minor but potentially pivotal improvements that simplify workflows and day-to-day tasks: