Sometime ago we faced a challenge when I was working at Agumbe while developing our AI gateway solution, we noticed our response timing going haywire with some resquests taking 17 secs to execute. That's not an issue with the API, it's a result of the architectural decision we took that we needed to rethink.
Most good startups while developing a product re-utilise the existing services which may not be well equipped to provide you the performance you desire which happened in our case, a Gateway demands low latency and blazing fast responses and our REST based inter service communication added up to the latency which was already bad from the llm providers side. Hence, we shifted to gRPCs.
A Representational State Transfer (REST) API architecture is built around "resources."
Think of a resource as a noun: a specific object or thing in your system. For example, if you are building an AI app, your resources might be /users, /prompts, or /models. To interact with these things, you use standard internet verbs that everyone already knows:
GET to read a resourcePOST to create a new resourceDELETE to remove a resource
In a standard setup, every time the Agumbe gateway needed to talk to an internal microservice, it sent a standard REST request over the web.
This works great for simple web apps. But inside the AI gateway, it became a massive bottleneck for two simple reasons:
1. The "JSON Tax" (Heavy Text Packets)
REST talks in JSON. JSON is awesome because humans can read it easily. But computers don't naturally speak JSON. Like come on JSON is practically english if you look at it properly.
Every time Service A wanted to talk to Service B, it had to turn its code into a giant text string (JSON), send it over the wire, and then Service B had to parse that text back into code. When you are dealing with huge AI payloads and LLM responses, translating text back and forth eats up precious time and CPU power.
2. Network Overhead Adds Up
REST usually runs over HTTP.
Modern REST APIs can absolutely perform well, but for high-frequency service-to-service communication there is still overhead associated with HTTP request handling, headers, and text-based payloads.
One request might only cost a few milliseconds.
Five services calling one another?
Now you've multiplied that overhead several times.
When the external LLM is already introducing latency, adding extra delay inside your own system only makes the problem worse.
Enter gRPC: The solution I would have Implemented.
To fix the 17-second problem, I would have switched our internal communication to gRPC (Google Remote Procedure Call). (we were on a GKE machine that's why the gRPC ;) )
If REST is like mailing a physical letter through the post office, gRPC is like a direct intercom system between your servers. Instead of dealing with URLs and web verbs, a service can call a function on another server directly, almost like the code was sitting on the exact same machine. Here is how gRPC makes your microservices life better:
- Binary Data (Protocol Buffers): gRPC doesn't use JSON. It compresses data into a tight, tiny binary format (zeros and ones) called Protocol Buffers. It drops the size of the data packets drastically, and computers can read it instantly without any slow text translation.
- A Multi-Lane Highway (HTTP/2): gRPC runs on HTTP/2. This allows our gateway to send and receive hundreds of internal requests at the exact same time over a single, always-open connection. No more waiting in traffic lines.
- Strict Rules (The Code Contract): With gRPC, you define your data structures upfront in a simple file. If a developer accidentally changes a variable name in a background service, the system won't even compile. This stops hidden bugs from crashing your production gateway.
REST vs gRPC Isn't a Competition
One lesson this experience taught me is that engineering isn't about finding the "best" technology.
It's about understanding trade-offs.
REST is simple.
It's easy to test.
Every language supports it.
It's ideal for public APIs that developers interact with directly.
gRPC shines somewhere else.
It excels at fast, reliable communication between machines.
Those are two very different problems.
Trying to replace REST everywhere would be just as much of a mistake as using REST everywhere.
Why the Choice Matters
Even though I left the job before I could get the chance to solve this system design issue, but it taught me something: use the right tool for the specific job.
Nobody needs to throw REST away completely. The system still uses REST for the public-facing APIs because it is easy for external developers to plug into and test.
But for the internal microservices where speed, low latency, and machine-to-machine performance are critical; gRPC was the obvious savior. It has the ability to stripe away the self-inflicted delays and give you a solid foundation to handle heavy AI workloads.
If you are building an internal system where every millisecond counts, look past the comfort of JSON and give gRPC(or RPC :)) a shot.
Comments
Post a Comment