for loop generator with a sentinel

I have an application with one producer and many consumers, and a queue, which communicates them. The consumer should collect some data from queue, let’s say qsize()/number_of_consumers, but it must stop it work when a sentinel appears.

I have such a code:

frame = 0
elems_max = 10
while frame is not None:
    frames = []
    for _ in range(elems_max):
        frame = queue_in.get()
        if frame:
            frames.append(frame)
        else:
            break
   process_data(frames)

As You can see None is a sentinel for this queue, and when it appears I wan’t to break my working process. I also want to get more then one element for data processing.

What is the fastest method to achieve this [in python 3.5]?

Answer

I understand that you want to break your outer while when encountering a None.

You can hold a boolean variable that is True while the while must execute, and False when it should stop.

This would look like this:

frame = 0
elems_max = 10
running = True
while running and frame is not None:
    frames = []
    for _ in range(elems_max):
        frame = queue_in.get()
        if frame is not None:
            frames.append(frame)
        else:
            running = False
            break
   process_data(frames)

The break instruction will break the inner for, but not the outer while. However, having set running to False, the while loop will stop.


Based on your comment.

It is not possible to include a break statement in a comprehension, nor an else clause, as you wanted to do:

frames = [f for i in range(elems_max) if queue_in.get() is not None else break]

However, you can build your list, and then remove all the elements after a None:

frames = [queue_in.get() for _ in range(elems_max)]

try:
    noneId = frames.find(None)
    frames = frames[:noneId]
except ValueError:
    pass

This is not very efficient, because potentially many elements will be appended in frames for nothing. I would prefer a manual construction, to avoid this hazard.


One more solution, based on a generator. This might not be what you expected, but the syntax is rather simple, so you may like it.

The idea is to wrap the getting of the data inside of a generator, that breaks on a None value:

def queue_data_generator(queue, count):
    for _ in range(count):
        item = queue.get()
        if item is None:
            raise StopIteration
        else:
            yield item

Then, instantiate this generator, and simply iterate over it:

g = queue_data_generator(queue_in, elems_max)
frames = [frame for frame in g]

The frames list will contain all the frames contained in queue_in, until the first None. The usage is rather simple, but you have to setup it by defining the generator. I think it’s pretty elegant though.