The QUERY HTTP Method
LinkedIn article formatting isn't great - for properly formatted quotes and code inserts, check out this article as originally published on my blog, 4lex.nz
Introducing QUERY
The IETF has published a document detailing the QUERY verb. The Query verb neatly solves the problem of asking APIs for LOTS of data or?conditional?data. In other words, QUERY enables API providers to provide a means for clients to ask the server about data, where the client isn’t sure what data is available.
A worked example
Working in Agri-tech, I often want to find out information about animals. Unfortunately, that information can live across many systems, with many identifiers (primary keys or other). In one context I’ve experienced, Dairy Animals can be identified and referenced by five or more different identifiers - so what do you do if you don’t know all of them?
In my case, I want to provide an API with the identifiers I DO know and ask for the API to return any identifiers it has that are linked to the ones I submitted. There are a few ways the API could implement such functionality.
For a complete animal:
HTTP GET /animals/${the_identifier_i_know}?identifierType=${the_identifier_type}
Based on this request URL, I would expect to provide an identifier and receive back the entire animal entity. Animal entities can be of substantial size! Serialising all this data can be time-consuming and wasteful for a busy API, particularly when we know we’re just exploring and want only a subset of that data. We’d be discarding 90% of the data returned.
For an animal’s identifiers only:
HTTP GET /animal/identifier?identifierType=${the_identifier_i_know}
I interpret this URL as only providing me with identifying details about the animal. Optionally, I can request a certain type of identifier. If I only care about a singular animal, this is probably all you need. No?QUERY?verb is necessary. What If I want to query a collection of animals? There are potentially many thousands of animals in a group for which I require data. I’d encounter a significant HTTP overhead, making that many requests to this endpoint. What we need is an API for a collection of animals!
A collection of dairy cows is a herd. I don’t care about herds; I want identifiers for an arbitrary group of animals for my specific use case.
REST guidelines ask that you model your APIs as
/{collection_one}/{collection_one_identifier}/{collection_two}/{collection_two_identifier}?and so on.
Under this guidance, we cannot submit multiple identifiers for a single collection. So if we had 3000 animals, for example, the request might be:
/animals?identifier=id1,id2,id3 ... etc?- it doesn’t make sense! Putting many identifiers in a URI comes with its work, but crucially, if you’re asking for lots of data and are forced to encode that in the URI, you could run into length limits preventing you from encoding all of your request. Most servers will typically limit the length of your encoded URI to 2048 characters or less. Those servers will decline your request if the URI is too long.
Most folks get around this by defining a?POST?endpoint that breaks RESTful principles. Here’s an example of a?POST?request:
HTTP POST /animals/search?with a body:
{
"knownAnimalIdentifier": [
"id1",
"id2",
"id3",
"id4",
"id5"
]
}
You could put all sorts of freaky logic or conditionals in this body; there are far fewer limits and, therefore, more risk of pushing the concept outside sensible boundaries. For example, this might also be valid:
{
"knownAnimalIdentifier":[
"id1",
"id3",
"id3"
],
"where":{
"healthTreatment":{
"condition":"mastitis",
"product":"amyzin"
}
}
}
You could pull all sorts of stunts here. The above looks suspiciously like GraphQL to me - the key message here is that?POST-ing arbitrary search data to an API imposes very few limits on otherwise conscientious and responsible developers.
Further, using a?POST?doesn’t match the intent of the request. REST guideline defines a?POST?as NOT idempotent -?POST?means you’re creating a new resource every time you make the request. With our above requests, the intent is to search for data, not create resources, but we don’t have a good way of modelling that! You could argue that when you submit a search via?POST?you are asking the server to create a new resource containing your results - the issue with that, is that you’re not creating a new resource. At best you are creating a new projection of resources that already exist. You’re not?POST-ing a new letter into the letterbox, you’re asking the mailman to pass you some letters that may or may not exist.
To do a search we need to do the?HTTP?GET?POST?GET?dance. First?GET?enough data to formulate your search query, then?POST?to create a query resource, then optionally?GET?the resource you just created.
Some folks will implement an API that directly returns the results to you. You’d need to code for latency, dropped connections and retries. For long-lived queries, you’d need a solution for the underlying data changing as the query progresses (similar to pagination where page 3 changes after you retrieve it but before you complete paging). If the API indirectly returns your data to you (i.e. immediately returns a link to the newly created search result resource), you have to go and?GET?that resource.
Using?POST?requests for search functionality solves one problem - lack of flexibility in the?GET?verb - but introduces another. Using?POST?server cannot cache or re-use results, nor can it deal gracefully with common failure modes (retrying ten times creates ten resources! All those resources might be different!)
The Query Verb
Julian Reschke, Ashok Malhotra and James M Snell have authored a draft RFC that describes the?QUERY verb.
QUERY?solves the problems of your typical search workflow.
The draft spec defines a?QUERY?as a means of making a safe, idempotent request that contains content. It’s worth your time to read the entire article (it’s about a five-minute read).
The important parts for me (i.e. order and emphasis mine) as a system designer are:
Before, I would have to make a?GET?to some URL with an enormous query string OR create arbitrary numbers of un-cacheable new resources; now, I can use the?QUERY?request. Furthermore, I can put whatever data I need in the body of my request, safe in the knowledge that implemented to spec; I’m going to get a quick response that I can cache in line with the server’s guidance.
It’s a small and subtle change. Implementation-wise, you could change your existing?POST?search endpoint to a?QUERY?and leave the same body. Migrate current?GET?endpoints by shifting query/path parameter content into the body and change the verb.
The most significant improvement relates to intent and clarity. Using a?QUERY?verb signals my intention to request data based on conditions that I supply.?QUERY?marks that request as idempotent and indicates the intent of the caller.
POST?shows intent to create a new resource.?GET?indicates intent to retrieve a resource for whom the caller already has an identifier. Neither?POST?nor?GET?indicate an intention to request data about which the caller is uncertain. The semantics of the verb and the request matter because they clarify intent. Computers will do what we tell them to do. Not what we intend for them to do.
Clarity of intent in an API is crucial to the success of that API. When designing that API, we want to make it as easy as possible to map the intent of the caller to appropriate functionality in the API. Introducing new verbs to clarify the caller’s desires and better match those needs (i.e. “What endpoint should I call to do X?”) to the correct part of an API brings developers joy.
Conclusion
The?QUERY?verb is brand new and in draft with the IETF - you can check it out?here. It clarifies the calling system’s intent and enables behaviour more suited to interactions where you don’t know what specific resources are available, but you do know something about those resources (e.g. the first and last names of a person, or their phone number, email address or some combination thereof). QUERY allows for idempotency, caching and is easy to adopt incrementally.