How do you ensure your microservices are reliable?

1. Standardize error codes and keep it consistent. Avoids confusion and helps with debugging. 2. Use appropriate http status code. Helps clients build relevant error handling. 3. Provide additional diagnostics and error messages in response. Helps with investigation and fix the issues only seen in the wild. 4. API documentation at best. Leaves no room for ambiguities in client integration.

At design stage, 1. make sure potential failure point and pattern identified, clearly document expected client behavior also. 2. Ensure schema is extensible. At implementation 1. All defined failure cases need be covered in UT and IT, to prevent contract breaking change from shipping 2. Dive deep into dependencies used, to unveil hidden failure patterns from upstream and handle according to contract with downstream. 3. Instrument enough, but not too noisy to avoid hiding info in flood At operation stage 1. Have canary runs often && monitored 2. Have runbook && operational tools ready 3. Train team to support effectively 4. Retrospective for failures in operation, implementation and design Re-iterate

1. Define retry strategy and pattern (retryable vs non-retryable, exponential backoff with cap, # of retries, etc) , avoid too aggressive retry to self DDOS, while not too long to breach SLA/SLO 2. Timeout is the evil, avoid it all. Timeout gives no information where the failure happens (incoming/outgoing connection, or process etc). Instead give meaningful error in case a logic step times out.

This does not include a case where microservice returns invalid or unexpected data which needs to be handled by the requesting system

Clearly define CR open, half-open and close transition strategy, condition and timeframe. CR is a protective mechanism with the goal to maintain high availability/throughput/etc (Your business goal), don't define CR to contradict with the goal

1. Observability is key in operation stage. Having enough instrument at the key points, but not too much to confuse engineers during critical events. 2. leverage standard tools and mechanisms to reduce learning curve (OTel, distributed tracing, logging, metrics system)

scalability and the agility to scale are two things. Design should cover both aspect, implementation must not undermine the design, operation tools and mechanism must be kept up-to-date and ready

any resilient service or app survives a system reboot and/or does not inherently depend on other (wildly distributed) services but will patiently wait until upstream services are ready, at the expense of delivery/response time. Network protocols are designed to be resilient, and so should upper-layer services be too, like eg smtp is implemented.

I'd like to note another one thing which rises up reliability microservices of my org: it's using a round-robin load balancing hardware/software.