Client-Side Error Handling: "Curl"-ing with Confidence

The curl command (short for "Client URL"), is a versatile command-line tool used for transferring data to and from a server. In infrastructure management, it is frequently employed in deployment scripts to initiate changes in administered systems, such as submitting configurations, creating or modifying resources, and starting tasks.

With the advent of cloud-based and self-healing systems, there is a tendency to trust more in stability than it probably should be and overlook potential unavailability issues. While these problems are more common in traditional physical and legacy environments, even contemporary infrastructure is not immune to network unreliability (consider distributed systems connected over the Internet). Services may not always be able to handle requests with 100% reliability. Therefore, being prepared for outages and errors and taking appropriate actions is a best practice to follow.

In this article, I present a pattern that can make the use of curl more robust in addressing similar issues.

The forms of errors

When sending a request to a service, the following errors may typically occur:

The service responds with an error message indicating the reason:

- User error: The request is malformed or unprocessable (e.g., the entity the request refers to is missing).

- System error: The request failed due to infrastructure issues (e.g., the web proxy is unable to reach the backend service) or functional errors (e.g., a bug or improper error handling in the backend code).

The request times out (the underlying system is unable to respond within a reasonable time).

The connection cannot be established to the target system or gets broken (e.g., physical layer issues, firewall settings, network congestion).

Handling errors on the client side

Handling errors is always a case-by-case matter.

Key considerations include:

• Is the problem intermittent or terminal?
• Is it possible to retry the request in case of a failure?

1. User errors are usually not retriable on their own; additional action is needed to recover (e.g., fixing the request or the target system's state).
2. With a timed-out request, changes may have already occurred in the target system, so retrying may not be safe.
3. System errors might be safe to retry, but not always (the target system may have already started to change its state, which may not be retried).
4. Refused connections could be retried (e.g., the next request might be routed to a different, healthy backend instance).

There is no one-size-fits-all solution. In the example below, a REST API is invoked with a JSON payload. Curl is equipped with error handling, security, timeout and retry settings, which can serve as a starting point for implementing your own:

# Create the JSON payload
REQUEST_PAYLOAD=$(cat <<EOF
{
  "attr": "$SOME_ENV_VAR",
  ...
}
EOF
)

SERVER_URL=https://api.some-awesome-service.net/v1/some-entity/

# Send the request to the REST API
set +e  # do not exit the script immediately on errors
HTTP_STATUS=$(curl -w "%{http_code}" -o /tmp/curl_response_body -s -S -i \
  --retry-connrefused --connect-timeout 10 \
  --fail-with-body --retry 3 --retry-delay 5 \
  -m 10 \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  -d "$REQUEST_PAYLOAD" \
  $SERVER_URL)
CURL_RETURN_CODE=$?
set -e  # exit on errors

if [ "$CURL_RETURN_CODE" -ne 0 ]; then
    echo $(date) "Failed to reach the API or process the payload. (HTTP Status: $HTTP_STATUS)" >&2
    [ -f /tmp/curl_response_body ] && cat /tmp/curl_response_body >&2
    rm /tmp/curl_response_body || true   # cleanup
    exit 1
fi

rm /tmp/curl_response_body || true     # Clean-up:

# the client code continues...        

Explanation of the used curl Flags

Response Handling

• -w "%{http_code}": This flag specifies what information to display after a completed operation. In this case, it outputs the HTTP status code of the response. The response code is captured into the HTTP_STATUS variable and evaluated later in the script. Long version: --write-out "%{http_code}"
• -o /tmp/curl_response_body: This flag writes the output to a file instead of stdout. Long version: --output /tmp/curl_response_body
• -s: This flag makes curl operate in silent mode, which means it won’t show the progress meter or error messages. Long version: --silent. Use the -s or --silent option along with the -S or --show-error option. The -s option suppresses the progress meter, while the -S option ensures that errors are still shown.
• -i or --include: This option tells curl to include the HTTP response headers in the output along with the response body. If you only want to see the headers without the body, you can use the -I or --head option.
• --fail-with-body: This flag makes curl fail on server errors while still outputting the response body.

Retries

• --retry-connrefused: This flag makes curl retry on connection refused errors.
• --connect-timeout 10: This flag sets the maximum time in seconds that curl will wait for a connection to be established.
• --retry 3: This flag specifies the number of retries if the request fails (no retry on user errors - HTTP 4xx).
• --retry-delay 5: This flag specifies the delay between retries in seconds.

Timeout

• --max-time or -m: This sets the maximum time in seconds that the entire operation is allowed to take.

Security

• -H "Authorization: Bearer $TOKEN": This flag adds an extra header to the request. In this case, it adds the Authorization header with a Bearer token. Long version: --header "Authorization: Bearer $TOKEN"

Request Content

• -H "Content-Type: application/json": This flag adds the Content-Type header to specify that the payload is in JSON format. Long version: --header "Content-Type: application/json"
• -X POST: This flag specifies the request method to use when communicating with the HTTP server. In this case, it is a POST request. Long version: --request POST
• -d "$PAYLOAD": This flag sends the specified data in a POST request to the HTTP server. Long version: --data "$REQUEST_PAYLOAD"

Conclusion

In summary, the curl command is an invaluable tool for managing data transfers in infrastructure management.

By understanding and utilizing its various flags for error handling, retries, timeouts, security, and request content, you can significantly enhance the robustness and reliability of your deployment scripts.

This article has provided a detailed explanation of these flags and demonstrated how to effectively handle common issues that may arise during data transfers. By implementing these practices, you can ensure smoother operations and better resilience against potential errors and outages in your systems.