TIL not using slow external connections in critical places of my app
Today, in the analytics system of https://seofy.io, I found a decrease in conversion from visiting the main page of the site to adding a site, so I began to investigate the reasons. I tried to add the site myself and hung up waiting for the server to respond and redirect to the next page of the user path funnel. Waited for about 30 seconds.
The culprit was Guzzle, who sent a request to an external resource and was waiting for a response, and the external resource was loaded so much that he had no time to respond.
As a result, I transferred this Guzzle request and Telegram alerts to the Laravel Model Observer, rewriting the code as follows:
exec('curl -s -X POST https://api.telegram.org/bot'.env('TELEGRAM_BOT_TOKEN').'/sendMessage -d chat_id='.env('TELEGRAM_GROUP_ID').' -d text="'.$text_to_send.'" --connect-timeout 5 > /dev/null 2>&1 &');
exec('curl -X POST http://'.env('BOT_ADDRESS','localhost').':8051/domains/'.$domain->id.'/start --connect-timeout 5 > /dev/null 2>&1 &');
Now, this logic is launched from the command line, without waiting for the result.
After: it's like 5 seconds needed to show the first website and content audit results to the user.
Before: loading... waiting... wtf...
Here is a video demo https://www.loom.com/share/5862c390abc54b1d901fab70feec90ec
I wonder what you think about it and how you would act in this situation.
You should never rely on the reliability of external servers and thus never call them synchronously. I had issues even with super stable platforms like Facebook. When you have even moderate traffic to your website (say 1000 users an hour) and external platform slows down from 100ms to 3s, it will make your platform suffer significantly as there will be a hanging connection sitting there for 30 times longer.
How to solve this issue:
3a. If you have a multi-tenant service design an external flag where you can quickly deploy a version of your service without a call to the slow external one until they resolve a problem. This will allow you to call things synchronously and quickly react when external services are slow. They will be slow from time to time, it is just a matter of time.
Hi @iamarsenibragimov
Noticed you're using Laravel. This is exactly what Queues are for. If the HTTP request isn't critical for the user's response just put the request into a queue and let it run asynchronously.
This way you could also go back to using Guzzle rather than exec() requests or even better if you're using Laravel 7 you now have the built-in HTTP facade.
Cheers.
Where I work, we've had the platform go down due to external dependencies. Where possible, either:
Both those methods will make your website snappy, and process the slow request in the background. At size, you quickly learn that relying on anyone out of your network is a terrible idea. And yes, Stripe included.