Long polling in BroadwaySQS

By default BroadwaySQS doesn’t wait for a message to arrive if no messages are available in the queue. Also, it has a default 5 seconds duration for which the producer waits before making a request for more messages. This may result in few second’s gap between enqueuing a message and processing it. For many cases this is good enough, but sometimes one may need a message to be processed immediately after enqueueing. In such cases you should consider using long polling.

To achieve it in BroadwaySQS just set wait_time_seconds to e.g. 10 seconds (or any value from 1 to 20) and receive_interval to 0 in producer’s options.

Example:

def start_link(_opts) do
  producer = Application.get_env(:scanner, :broadway_producer, BroadwaySQS.Producer)
  queue_url = Application.get_env(:ex_aws_sqs, :queue_url)

  Broadway.start_link(__MODULE__,
    name: __MODULE__,
    producers: [
      default: [
        module: {
          producer,
          queue_url: queue_url, wait_time_seconds: 20, receive_interval: 0
        },
        stages: 1
      ]
    ],
    processors: [
      default: [stages: 50]
    ]
  )
end

Tweet