How can you design RESTful APIs that support long polling and server-sent events?

For applications that need updates almost instantly, long polling is a technique that enables real-time communication between a client and a server. Although it offers a straightforward method of achieving real-time functionality, WebSockets and other more recent options might be more effective in this regard. Nevertheless, Long Polling can still be a useful option for accomplishing asynchronous communication between clients and servers in specific situations or when working with legacy systems.

REST Long Polling is an asynchronous communication method where the client sends a request, and the server keeps the connection open until data is available or a timeout occurs. Advantage: It reduces network overhead and enhances responsiveness by avoiding constant connection reestablishment. Disadvantage: Potential latency in real-time data delivery; servers may face scalability issues due to long-open connections, consuming resources. Consider these trade-offs when deciding to implement Long Polling in a system.

Why would you use long polling when websockets are available? Isn't long polling simply the backup transport for websockets?

Long polling, a client-server communication technique, involves the client sending a request, and the server keeping the connection open until data is available or a timeout occurs. This minimizes the need for repeated polling, optimizing bandwidth usage. Implementable via standard HTTP methods, it provides real-time data updates. Yet, challenges like heightened server load, latency, and compatibility concerns with proxies and firewalls should be considered. While offering immediate data transmission, long polling demands careful assessment for its impact on server performance and potential obstacles in certain network environments.

This approach minimizes connection overhead by eliminating the need for repeated connection setups, enhancing efficiency compared to techniques like long polling. SSE's support for automatic reconnection and event buffering further bolsters reliability. However, considerations include limitations on binary data support, challenges with CORS requests, and slightly lower browser compatibility when compared to long polling. Despite these constraints, SSE emerges as a powerful tool for real-time data transmission, providing developers with a robust and efficient mechanism for server-client communication in web applications.

Yes we need to secure the project before exposing to end user and for that we require to choose a technique which suits and we should design taking this in consideration

Consider incorporating versioning in your API to ensure backward compatibility as it evolves. Employing HATEOAS (Hypermedia as the Engine of Application State) can enhance discoverability by including links in responses, guiding clients through the application flow. Lastly, implement meaningful error handling with clear messages to assist developers in troubleshooting and resolving issues effectively.

To have added confidence in integration testing, tests can be made more deterministic by using a fixed configuration and mocked server that responses at a fixed time. This makes it much simpler to automate validations in a safe and reliable manner.

If long polling is a requirement I think the application could be reviewed, maybe needs an improved architecture to reduce the request and response time? Considering other options: In modern web applications the use of other protocols like MQTT, WebSocket, Kafka, DDS or implementing Pub/Sub with Redis, RabbitMQ, aws SQS for background processing can provide better experiences since it provides an asynchronous data flow with a persistent connection. It probably is more code and overkill on small projects but using http for requests with an unexpected response time doesn't seem appropriate.