Fahrenheit 451 - RESTful APIs - The Secret to HTTP Status Codes

This blog post is classified and will self destruct within 100 years. Read with caution, you may learn something, and above all... Burn after reading.

Introduction

The humble HTTP status code is often unloved and usually misused when used as part of a RESTful API. SOAP returns errors through 200, which has led people migrating to REST to commit some basic errors. The problem with this approach is that unless you use status codes appropriately, your API will be difficult to consume.

In order to mitigate this problem, this article attempts to provide a subset of these codes for standard RESTful API usage. There's no "one true answer" as Dr. Roy Fielding's doctorial paper (which you can read here) doesn't really go into detail on this, so there is no "canon", just sensible best-practice.

Before we start in earnest, it's important to understand that there are five “bags” of HTTP status codes:

• 1XX – Informational
• 2XX – Success
• 3XX – Redirects
• 4XX – Client Errors
• 5XX – Server Errors

Understanding these bags is the first step on the path to success, understanding that "client and server" map to the "Client-Server Architecture" constraint of REST very well is key.

The excellent article at: https://www.codetinkerer.com/2015/12/04/choosing-an-http-status-code.html shows some great flowcharts for choosing appropriate status codes. I’ll include the top-level flowchart here, but I’d urge you to read the linked article as it’s pretty definitive.

If you want a fairly comprehensive list, the Wikipedia article https://en.wikipedia.org/wiki/List_of_HTTP_status_codes does as good a job as any. I’d bookmark this page. It doesn’t feature the useful flowcharts in the first link though, so I’d read both.

In general, don’t aim to implement all the codes, it makes your API too “busy” and implementing clients will have to deal with too many variants. Just implement what you need, but implement it well.

It’s also worth noting that these codes are either temporary (such as 307 – Temporary Redirect) or permanent (such as 308 – Permanent Redirect). Your HTTP client should not request something returning a permanent code a second time, allowing for caching.

I’ll also provide logging hints throughout, as being able to use something like ELK with your RESTful API is a very useful thing in production.

1XX – Informational

This is the smallest bag, with only four codes in it. You won’t usually need to use these at all unless you’re implementing a Web Server (which, for the rest of the article, we’ll assume you’re not).

I’m not going to go into further detail on these as it’s largely of no use to you. If you need to log these, do it at the TRACE or DEBUG level. Do not clog up your logs with information you’re not interested in.

2XX – Success

Another small bag of codes and you’ll only be interested in a subset of them. Log these at the INFO level (at least the URL and headers, but maybe not the body). You’ll often want to be able to dashboard successful requests, so if you log them at the TRACE or DEBUG level, they may not get logged in production.

200 – OK – This code is your general success response. You must only return 200 if the request is genuinely ok. Do not… I repeat, do not, tunnel errors through 200s. Your logs will miss them, your API consumers will be confused. For GETs, a 200 response should be considered for caching. I’d log any 200 bodies at the TRACE or DEBUG level.

201 – Created –If somebody POSTs or PUTs a new resource which is then persisted, this is the response to return, along with a link to the new resource in the “location” header in the response. See https://httpstatuses.com/201. Like 200, I’d put the body at TRACE or DEBUG level.

202 – Accepted – If you’re implementing asynchronous requests then this is your code for “I’ve heard you, please check back later for your result”. You may send back a body that includes a link to a “status” server and the current status (e.g. PENDING). Please see https://httpstatuses.com/202. Again, TRACE or DEBUG for the body.

204 – No Content – If you’re performing an action (such as a PUT) and there’s nothing to send in return of post, use this status code. You should include an ETag header representing the latest version/hash to prevent subsequent HEAD requests to get the latest one.

The other 2XX codes are not as useful to the RESTafarian, but feel free to look them up if you’re interested.

3XX – Redirects

A medium sized bag of redirection and location based status codes. Unless you’re implementing an HTTP Client or a Web Server, you won’t really need to use these. Log them at the TRACE or DEBUG level as you’re probably not interested in dashboarding these in production.

The most interesting for our purposes is:

304 – Not Modified – Your cache should return this if a resource has not been modified, allowing for HEAD requests to be useful.

4XX – Client Errors

The largest bag of codes, and the most useful for our purposes. These are your meat and potatoes; the client is expected to send in bad information from time to time. Client errors should not be logged as errors in your logging as it makes monitoring harder and is an expected interaction. I’d consider logging them at the INFO or WARN level. You may want to log the body at DEBUG level for these requests. You probably don’t want to see a lot of body information in production logs as you may end up eating up a lot of storage.

Validation of Requests

400 – Bad Content – The request is malformed e.g. un-parsable JSON.

409 –  Conflict – The ETag passed in doesn’t match the one on the server, therefore you’re trying to PUT stale data. This is used for optimistic locking.

422 – Unprocessable Entity – The request is well formed but fails validation e.g. a Date of Birth that’s in the future.

423 – Locked – The resource is currently locked by part of the system.

Trouble Getting Content

404 – Not Found – The classic code you’ll all have heard of. But, just because it doesn’t exist now, doesn’t mean it won’t in the future. This is a temporary code.

410 – Gone – Something existed, but it is now gone. This is a permanent code and should be used when a resource is permanently deleted.

414 – Request URI too Long – If the client decides to send in an enormous query-string that isn’t to your liking, you can politely decline to service the request for the resource.

415 – Unsupported Media Type – The request provides or is asking for a MIME type you don’t support for this method e.g. Accept: application/octet-stream when you’re only serving application/json. Don’t log the body for this one, it may be malicious.

Auth and Auth based Codes

401 – Unauthorised – Actually means “unauthenticated” as it is a request to authenticate before attempting again.

402 – Payment Required – You may be authenticated, but have not paid your dues. This is officially “reserved for future use”.

403 – Forbidden – You’re authenticated, but not allowed to see this resource. Further authentication (unless with different credentials i.e. a different user) will not resolve the issue, you should not try this link again.

405 – Method not Allowed – If you try to PUT on a GET only resource, this is what you should receive. Using the OPTIONS verb should return the acceptable verbs for this resource. Don’t log the body for this one as it’s not allowed.

407 – Proxy Authentication Required – Similar to 401, but you must supply a Proxy-Authenticate header field in the subsequent attempt.

451 – Unavailable for Legal Reasons – The redaction code, inspired by Ray Bradbury’s novel Fahrenheit 451, which you should read, it’s a classic. Do not log any data other than the URL and headers, as the data may be sensitive.

5XX – Server Errors

You should be careful sending back server errors, especially 500 (Internal Server Error). Too many of these on certain cloud providers and your service may end up being torn down and recreated. This is slow and expensive (compared to returning a 4XX error).

Also, don’t use exception handling when sending a return code will do, this is a) general good practice – not controlling program flow through exceptions and b) much more performance (1,000 times quicker to return a value than rely on exceptions which unwind the stack and log out a huge wad of information.

Log 5XX errors at the WARN or ERROR level, so they are picked up in production. You may want to WARN on upstream timeouts and ERROR for genuine 500 errors. You may log the body at the DEBUG or INFO level as you’re more likely to be interested in the details of a 500 error as you may have found a bug in your system!

500 – Internal Server Error – Your server has thrown an exception somewhere, this is your problem to solve and you should have monitoring and alerting on to spot when it’s happening. Ensure you log the body so you can figure out what went wrong.

501 – Not Implemented – You shouldn’t need to throw this very often as it’s almost always covered by one of the 4XX codes.

503 – Service Unavailable – Return this when your service is down or is too busy. It’s better than accepting something you can’t process.

504 – Gateway Timeout – An upstream server didn’t respond in time, that’s not your fault, but you now cannot service the request, so you should inform your API consumer so they can back-off and retry at a later time.

511 – Network Authentication Required – Use 401 – Unauthorised instead. You shouldn’t need this code.

Dealing with Errors

You should return a list of errors with an error condition, this should always be a list, even if you only have one error, so they can be parsed consistently by the client e.g.

Provide a machine-readable code, the field in error and a human-readable message as a hint to the consumer and for easier diagnosis of problems in logs. Provide the consumer with the ability to fix their requests, it will cut down on support requests.

Conclusion

HTTP Status codes aren't that hard to understand, you just need to spend the time learning them. Ensure you use something like Swagger/OpenAPI to document the codes you'll be returning. Write your usage of status codes into your API standards and ensure everybody reads them.

As long as you stick to the standards (i.e. do NOT make your own ones up, HTTP clients and libraries will not know how to handle them) and do not tunnel errors through 200, you should be ok. Keep it simple, be consistent, keep it documented.