Developers building complex AI apps - how are you solving these challenges?

Hey everyone👋!

I've been working on integrating OpenAI into my React JS apps for about 14 months now. My main app is a pretty complex application (a no-code app development platform), so the outputs I've been looking for from AI are always JSON, often multiple layers of nesting, and little room for error in terms of formatting.

I've built a few others as well, which were simpler but had similar requirements.

I encountered a lot of roadblocks and ended up coming up with my own custom solutions / internal tools to make things work well.

My personal issues fall into these categories:

JSON Formatting
Ensuring AI predictably responds in my custom JSON format. OpenAI's function calling helped a ton with this, but didn't solve everything on it's own without the other things below (it still fails and returns invalid results, often).

Validation / Auto-Retry
I had to build a system to validate all AI responses, and then auto-retry up to X times hoping to get a valid response. This does of course increase the cost significantly if there are multiple failures, but I don't think there's any way around this one.

Request Timeout Limit
I use Heroku for my front-end and it maxes at 30 seconds. From my research it seems like this is pretty common. Since AI responses could be very long-running, I ended up having to hack together a combination of a Websocket, Amazon API Gateway and Lambda function to accomplish a system free of request timeout limitations. Has anyone dealt with this in a better / easier way? Am I missing something 🤣?

Chaining Multiple Requests
After a lot of trial and error, getting frustrated with the AI responses being really inconsistent, I came to the conclusion that if you give AI one clear task, it will probably do it well, but if you give it lot of different tasks and things to consider at the same time, it's likely to mess up one or more of those things.

This isn't a 'challenge' anymore in the same way the others were, since I think my solution is pretty good, but the hard part is deciding on where to 'split up' your desired result into different data structures and then how to pass that into the next step as context. This is very time-consuming.

I'm just curious if anyone out there has been going through any of the same problems and wants to discuss different solutions?