Shopify Webhooks: Strategies for Optimal Performance in App Development | FireGroup

In the realm of Shopify app development, Shopify webhooks are a familiar concept for developers. Almost every app needs to register Shopify Webhooks to synchronize app data with the store's data. While commonplace, the optimal usage of webhooks and avoiding potential pitfalls often goes unnoticed. In this article, we'll provide insights into the intricacies of Shopify Webhooks and strategies to not just use them but master them for unparalleled performance.

>> Read the full article and explore other #WeShare blogs on our website.

I. Shopify Webhook Overview

Let's begin by understanding what Shopify webhooks are:

• A webhook is a feature allowing a website to automatically notify and send real-time data to systems when a specific event occurs on the website (e.g., customer registration, form submission, purchase, or email sending).
• In Shopify, webhooks enable applications to stay connected to Shopify's data or perform an action after a specific event occurs in the store.
• Webhooks are an efficient solution instead of continuously checking changes in the store's data.

II. How to use Shopify Webhooks

Learn how to navigate the Shopify webhooks setup, understand the basics of GraphQL and REST API registration. Follow step-by-step guides to easily set-up webhooks and stay ahead of the e-commerce game.

1. Register webhooks

Webhooks are registered based on topics, and both REST and GraphQL APIs can be used for registration

• GraphQL

Documentation for the most up-to-date information: here

• REST

Documentation for the most up-to-date information: here

2. Webhook event topic:

To register a webhook, choose a topic from a variety of options covering almost every change related to products, collections, orders, customers, etc. Ensure the app and store data stay synchronized promptly..

• View all supported event topics: https://shopify.dev/docs/api/admin-rest/2023-04/resources/webhook#event-topics

3. Receiving and Handling the Hook:

Shopify sends the webhook with headers and payload to the pre-registered endpoint.

• Verify the webhook: Each webhook request includes a base64-encoded header called X-Shopify-Hmac-SHA256. This header is created using the app's client secret along with the data sent in the request. If you are using PHP, Ruby on Rails, or Sinatra, the header is HTTP_X_SHOPIFY_HMAC_SHA256. To verify the authenticity of the webhook, compare the computed value with the value in the X-Shopify-Hmac-SHA256 or HTTP_X_SHOPIFY_HMAC_SHA256 header. If the header value and the HMAC of the message match, then the webhook is confirmed to be sent from Shopify.
• Update data in the app based on the webhook payload.
• Respond to the webhook: The endpoint receiving the hook must return a status code of 200. Any response outside the 200 range, including HTTP 3XX redirection codes, indicates a failure to receive the webhook.?

Shopify will wait 5 seconds to receive feedback for each request regarding the webhook. If there is no response or an error is returned, Shopify will attempt to reconnect 19 times within the next 48 hours. If there are 19 consecutive failures, the webhook subscription will be automatically deleted. A warning that the subscription will be deleted is sent to the emergency email address for the application developer.

Additional details: https://shopify.dev/docs/apps/webhooks/configuration/https

III. Common Issues with Shopify Webhook

While using webhooks offers the advantage of continuous data synchronization with Shopify's database, there are significant challenges related to app database performance and the risk of outdated data if overly dependent. Some common issues include:

• Spam webhooks: Easy registration may lead to server overload with a large number of hooks.
• Read and write to the database excessively: High request volumes, especially with product/update webhooks, can overload the database.
• Subscription removal: Slow or failed webhook processing may result in automatic removal by Shopify.
• Outdated data: Excessive reliance on webhooks can lead to outdated app data compared to the store.

IV. Strategies for Optimal Performance when using Shopify Webhooks

Below are strategies applied in FireGroup's projects, particularly in the Transcy solution (AI Language Translate), to address and mitigate the impact of webhooks.

1. Restricting spam:

• Separate webhooks into a dedicated service to optimize and limit the impact on other services.
• Register webhooks only when necessary and divide the registration based on essential features.
• Remove unused webhooks.
• Register specific payload data to limit payload size during registration.
• Cache hooks for gradual processing through scheduling, reducing duplicate webhook processing.

2. Restricting database updates:

• Check whether hook data differs from the database before updating.
• Use an additional cache layer for comparison instead of directly selecting from the database.
• Group multiple items for bulk insert and update.
• Separate heavy and frequently hooked resources into a separate database.

3. Restricting outdated data:

• Regularly monitor webhook status in the app admin insights.
• Run a periodic process to update/remove hooks for active users.
• For critical data, implement a periodic process rather than relying solely on webhooks for updates.

V. Practical Lessons from Transcy: Optimizing Shopify Webhook Performance

Embark on a practical journey through the challenges and solutions encountered by our FireGroup Product Team in the Shopify app project, Transcy. With the high volume and processing time of product-related webhooks, our team made several improvements to handle these challenges effectively.

1. Shopify Webhooks: A Necessity for Transcy

Transcy, being an AI translation app, heavily relies on Shopify webhooks to keep product lists updated. The product webhook, encompassing topics like product/create, product/update, and product/delete, generates a substantial number of requests, posing a challenge for efficient processing.

2. Separating Webhook Processing for Improved Stability

Faced with performance issues as user numbers surged, our team opted to move webhook processing to a dedicated server cluster. This strategic move isolated potential disruptions caused by webhook processing, ensuring core app functions remained unaffected even if the webhook cluster experienced difficulties.

3. Optimal Hook Registration and Payload Reduction

To alleviate the strain on the server, we optimized hook registration by subscribing only to essential data for app functionalities, such as product title, description, handle, and content. This reduced the payload on incoming hooks, significantly lightening the server load.

4. Transition to NoSQL for Enhanced Database Performance

Recognizing the strain on the MySQL database, the team explored NoSQL, specifically MongoDB, for improved scalability. The complete migration of product data storage to MongoDB alleviated CPU overload issues, ensuring smoother handling of large datasets.

5. Kafka Integration for Efficient Hook Handling

To further optimize webhook processing, the team implemented Kafka as a buffering mechanism. Instead of processing hooks immediately, the webhook server pushes data to Kafka, allowing for a more controlled and efficient handling of incoming requests. Kafka acts as a buffer, reducing pressure on the webhook server and minimizing data duplication.

6. Scheduled Batch Processing for Database Updates

A scheduled crontab runs every minute to scan and process data from Kafka into the database. By marking entries with product IDs, the team efficiently handles and overrides duplicated data, reducing the need for redundant data processing.

7. Python Worker Queues for Database Updates

Worker queues, implemented in Python, facilitate the gradual update of data from Kafka into the database. Before updating, a query is performed, leveraging Redis for data caching, to check for changes compared to existing app data. This process minimizes unnecessary database writes, focusing only on essential updates.

Conclusion

The concepts and optimizations discussed here provide a foundation for working effectively with Shopify Webhooks. Applying these strategies to Shopify app projects can ensure rapid synchronization, minimizing the risks associated with webhooks. Continuous improvement and exploration of new solutions remain crucial as user engagement with the app grows.

Written by Dao Vo Sy – Tech Lead, FireGroup Technology

Embrace the opportunity to be part of our cutting-edge projects and tech-driven journey, join us now at https://firegroup.io/careers/

You may also want to read:

Optimizing PageSpeed to Enhance User Experience in 2023