Pradip Dharam

How FastAPI handles 1000 client hits in a second

FastAPI async event loop, short-long polling, web sockets and server sent events

Photo by MARIOLA GROBELSKA on Unsplash

API’s Asynchronous processing can be done by these protocols. These are the methods used to perform the asynchronous / real-time communication between a client and server.

1. Short polling
2. Long polling
3. Web sockets
4. Server Sent Events.

During the old times. API was hit on the server and connection used to get created between client and server for each of the API hit. Limited number of API’s could be processed because of limited number of connections can be maintained with the server. System was not scalable for handling high number of requests from clients. This was refered to as Synchronous approach.

Evolution where API request sent to server is now queued at server side and token is shared with respective client for that request queued. Now, how does client going to access the response of the placed request?

Concept of polling came into picture where client and server need not maintain the connection. Instead client places the API request with server, server then returns the token to the client.

Short polling where client repeatedly sends API requests to the server at fixed intervals (e.g. every 2 seconds) to check if request is processed and the response of the placed request is available. Server either responds with the response data or message like request is still processing “no response”. Short interval between repetitive API request to check whether placed request has response returned by server, creates the performance bottleneck. Long interval creates the latency even if the request returned the response even in smaller time.

Long polling where client does not send repetitive requests. Client keeps the connection open when API request is placed with server, say for 30 seconds, client keeps listening. If within 30 seconds response is not received, then response with empty data is shared with client. Client understands that the request is still processing, it then creates new connection request to keep listening whether server generated the response, as soon as response is ready, client receives the response. It leads to 100’s or 1000’s of open connections, some may lead to timeouts leading to increased load on server, the performance bottleneck.

Efficient solution is needed than the polling. The Web sockets where full duplex communication maintained, both client and server are listening upon request is placed to server. Web sockets leads to disadvantage in terms of good amount of resource consumption because of web sockets stateful nature, duplex long lived connections does not scale much for 1000’s of API hits to the server

The event loop helps scale the high number of API hits. Imagine the single waiter (the event loop), takes the order, queues the order to the kitchen but not wait until order is ready and attends the next table to take the order. He returns to the kitchen when order is ready to serve.

Fast API uses asyncio as an event loop which handles many tasks concurrently (concurrency blogs below) without having to bring many threads into picture by efficiently switching between requests. It switches back to another request if one request is waiting for disk or network I/O. The Uvicorn is an engine that manages the event loop.

Concurrency and parallelism
You woke up in the morning. You need to get ready for office. You see notification on chat message on your mobile to…medium.com

await instructs asyncio event loop to switch back to another task till the await task is getting processed. async keyword helps define asunchronous route.

from fastapi import FastAPI
import asyncio

app = FastAPI()

# Simulated async task (e.g., database call or external API)
async def do_slow_work():
    print("Starting slow async work...")
    await asyncio.sleep(5)  # Simulate async I/O task
    print("Slow async work completed.")
    return "Result from slow work"

# Async route that awaits the task
@app.get("/process")
async def process():
    print("Handling request...")
    result = await do_slow_work()  # await another async function
    print("Request handled.")
    return {"message": result}

# In Shell Script
uvicorn main:app --reload

# Disclaimer: This code was generated with assistance from chatgpt,
# an AI-powered coding assistant

Event loop asyncio helps FastAPI handle 1000 client hits in a second.

Feel free to reach out on LinkedIn for networking.

Apart from short polling, long polling, web sockets, there is one more which was not supported in http protocol is Server Sent Events.