Design a GPU inference serving system that accepts synchronous HTTP requests containing natural-language prompts and returns one complete model response per request.
The model-serving API accepts batches containing between 1 and 100 prompts. A GPU can process only one batch at a time, and each batch takes approximately 100 ms regardless of whether it contains one request or one hundred. This makes **batch occupancy the primary throughput and…
