Queuing tasks with prioritization rules

This is a more general question on how the solution could be designed.

The goal is to have an API which would take some a text or a list of texts, process it and return transformed ones. How should I approach it so that whenever a text is requested, all the requests that contains list of texts would wait and one text would be processed first?

Answer

This question is a bit generic, therefore I’m going to answer with a stack that I developed in a past project. I would use the redis-server function of streaming(https://redis.io/topics/streams-intro). You can have a stream for a list of texts and a stream for texts, and when your algorithm is searching for new tasks he first checks if the stream of the list of texts has any pending package. I recommend the usage of redis-server because is a robust solution, having access to a server dedicated for this service, instead of using celery. But I have never used celery for this purpose, and can’t advocate for him on this topic.