Hello there!
I have set up integration with Twilio in order to send WhatsApp notification messages to my workers.
Twilio doesn't support bulk sending. So the recommended way is to loop over the array of receivers' phone numbers and send them message one by one.
The thing is that there could be a few hundreds of receivers (~500), and if I make a loop that sends them message one by one it could take quite some time and probably apply a heavy load on the frontend part of the app. Because from what I see there is no meaningful way to do such operation from the backend (query library) part.
My question is what would be the best way to handle this scenario?
1 Like
Hey @Top_Admin
What I found works best with a very similar situation is to utilize retool workflows
you can create a workflow that is triggered via a webhook and send the least amount information to it possible. for example an array of ids to the users and inside the work flow
1- create a query to get the phone numbers for all the users
2- create a function that hits the Twilio API and takes the needed arguments for one user
3- create a loop that goes over all your users and triggers the function with javaScript.
the benefit of following the above method is the level of control you have over your data. you can add as much filters and make it as error proof as possible. you can even add an error handler that sends you message on slack to notify you when there's a failure for some users.
2 Likes
Hey @Mahmoud! Thanks so much for your response. It really makes sense to me, because workflows are actually suited for this kind of job.
I will try to summarize what you just suggested in order to make sure that I get it right.
A) Basically I will send a POST request to Retool Workflow Webhook from my Retool App with the list of all the phone numbers that I need to send the message to.
B) Retool Workflow will send the message by Looping the request to Twilio API
One more question:
If I need to send messages to 500 of receivers – what would be the best way to do it concurrently? If I understand it correctly looping will send the message one after another, i.e. sequentially.
My best guess is to trigger Workflow 5 times with 100 phone numbers, so that 5 workflows can execute concurrently. What do you think?
1 Like
@Top_Admin
For Point A I found it best when I send only the id or basically the least amount of information possible to be functional.
for the following reasons
1- it helps with the performance in front facing UI: you won't have to run the query that will get the information needed for the workflow on the client side.
2- as the workflow grows and you need more information (names of the users to contact for example) it will be much easier to just do that in one place at the workflow and leave the app functioning as is.
This is all just personal experiences. it might be more suitable for you to send the list of phone numbers.
As for the concurrency question:
Good news is loops in retool workflows execute in parallel by default.
They use
Promise.all() to resolve all the values in parallel.
so there is no need to trigger the same workflow more than once.
However, one thing that you will have to take care of ,as the number of users grow, is the rate limit of the API you're working with.
you might have to work in batches or execute sequentially to get around the rate limit.
2 Likes
@Mahmoud Thank you so much for your very considerate & thorough answers. Much appreciated!
You are an absolute star!
1 Like
