Few days ago we were faced with a challenge in Agumbe while developing our AI gateway solution, we noticed our response timing going haywire with some resquests taking 17 secs to execute. That's not an issue with the API, it's a result of the architectural decision we took that we needed to rethink.
Most good startups while developing a product re-utilise the existing services which may not be well equipped to provide you the performance you desire which happened in our case, a Gateway demands low latency and blazing fast responses and our REST based inter service communication added up to the latency which was already bad from the llm providers side. Hence, we shifted to gRPCs.
A Representational State Transfer API architecture
Comments
Post a Comment