Buffers and working with rate

Buffers for asynchronous stages

In this section we will discuss internal buffers that are introduced as an optimization when using asynchronous stages.

To run a stage asynchronously it has to be marked explicitly as such using the .async() method. Being run asynchronously means that a stage, after handing out an element to its downstream consumer is able to immediately process the next message. To demonstrate what we mean by this, let's take a look at the following example:

Source.from(Arrays.asList(1, 2, 3))
  .map(i -> {System.out.println("A: " + i); return i;}).async()
  .map(i -> {System.out.println("B: " + i); return i;}).async()
  .map(i -> {System.out.println("C: " + i); return i;}).async()
  .runWith(Sink.ignore(), mat);

Running the above example, one of the possible outputs looks like this:

A: 1
A: 2
B: 1
A: 3
B: 2
C: 1
B: 3
C: 2
C: 3

Note that the order is not A:1, B:1, C:1, A:2, B:2, C:2, which would correspond to the normal fused synchronous execution model of flows where an element completely passes through the processing pipeline before the next element enters the flow. The next element is processed by an asynchronous stage as soon as it is emitted the previous one.

While pipelining in general increases throughput, in practice there is a cost of passing an element through the asynchronous (and therefore thread crossing) boundary which is significant. To amortize this cost Akka Streams uses a windowed, batching backpressure strategy internally. It is windowed because as opposed to a Stop-And-Wait protocol multiple elements might be "in-flight" concurrently with requests for elements. It is also batching because a new element is not immediately requested once an element has been drained from the window-buffer but multiple elements are requested after multiple elements have been drained. This batching strategy reduces the communication cost of propagating the backpressure signal through the asynchronous boundary.

While this internal protocol is mostly invisible to the user (apart form its throughput increasing effects) there are situations when these details get exposed. In all of our previous examples we always assumed that the rate of the processing chain is strictly coordinated through the backpressure signal causing all stages to process no faster than the throughput of the connected chain. There are tools in Akka Streams however that enable the rates of different segments of a processing chain to be "detached" or to define the maximum throughput of the stream through external timing sources. These situations are exactly those where the internal batching buffering strategy suddenly becomes non-transparent.

akka.stream.materializer.max-input-buffer-size = 16

final Materializer materializer = ActorMaterializer.create(
  ActorMaterializerSettings.create(system)
    .withInputBuffer(64, 64), system);

final Flow<Integer, Integer, NotUsed> flow1 =
  Flow.of(Integer.class)
  .map(elem -> elem * 2).async()
  .addAttributes(Attributes.inputBuffer(1, 1)); // the buffer size of this map is 1
final Flow<Integer, Integer, NotUsed> flow2 =
  flow1.via(
    Flow.of(Integer.class)
    .map(elem -> elem / 2)).async(); // the buffer size of this map is the default

final FiniteDuration oneSecond =
    FiniteDuration.create(1, TimeUnit.SECONDS);
final Source<String, Cancellable> msgSource =
    Source.tick(oneSecond, oneSecond, "message!");
final Source<String, Cancellable> tickSource =
    Source.tick(oneSecond.mul(3), oneSecond.mul(3), "tick");
final Flow<String, Integer, NotUsed> conflate =
    Flow.of(String.class).conflateWithSeed(
        first -> 1, (count, elem) -> count + 1);

RunnableGraph.fromGraph(GraphDSL.create(b -> {
  // this is the asynchronous stage in this graph
  final FanInShape2<String, Integer, Integer> zipper =
      b.add(ZipWith.create((String tick, Integer count) -> count).async());
  b.from(b.add(msgSource)).via(b.add(conflate)).toInlet(zipper.in1());
  b.from(b.add(tickSource)).toInlet(zipper.in0());
  b.from(zipper.out()).to(b.add(Sink.foreach(elem -> System.out.println(elem))));
  return ClosedShape.getInstance();
})).run(mat);

Buffers in Akka Streams

In this section we will discuss explicit user defined buffers that are part of the domain logic of the stream processing pipeline of an application.

The example below will ensure that 1000 jobs (but not more) are dequeued from an external (imaginary) system and stored locally in memory - relieving the external system:

// Getting a stream of jobs from an imaginary external system as a Source
final Source<Job, NotUsed> jobs = inboundJobsConnector;
jobs.buffer(1000, OverflowStrategy.backpressure());

The next example will also queue up 1000 jobs locally, but if there are more jobs waiting in the imaginary external systems, it makes space for the new element by dropping one element from the tail of the buffer. Dropping from the tail is a very common strategy but it must be noted that this will drop the youngest waiting job. If some "fairness" is desired in the sense that we want to be nice to jobs that has been waiting for long, then this option can be useful.

jobs.buffer(1000, OverflowStrategy.dropTail());

Instead of dropping the youngest element from the tail of the buffer a new element can be dropped without enqueueing it to the buffer at all.

jobs.buffer(1000, OverflowStrategy.dropNew());

Here is another example with a queue of 1000 jobs, but it makes space for the new element by dropping one element from the head of the buffer. This is the oldest waiting job. This is the preferred strategy if jobs are expected to be resent if not processed in a certain period. The oldest element will be retransmitted soon, (in fact a retransmitted duplicate might be already in the queue!) so it makes sense to drop it first.

jobs.buffer(1000, OverflowStrategy.dropHead());

Compared to the dropping strategies above, dropBuffer drops all the 1000 jobs it has enqueued once the buffer gets full. This aggressive strategy is useful when dropping jobs is preferred to delaying jobs.

jobs.buffer(1000, OverflowStrategy.dropBuffer());

If our imaginary external job provider is a client using our API, we might want to enforce that the client cannot have more than 1000 queued jobs otherwise we consider it flooding and terminate the connection. This is easily achievable by the error strategy which simply fails the stream once the buffer gets full.

jobs.buffer(1000, OverflowStrategy.fail());

Rate transformation

final Flow<Double, Tuple3<Double, Double, Integer>, NotUsed> statsFlow =
  Flow.of(Double.class)
    .conflateWithSeed(elem -> Collections.singletonList(elem), (acc, elem) -> {
      return Stream
        .concat(acc.stream(), Collections.singletonList(elem).stream())
        .collect(Collectors.toList());
    })
    .map(s -> {
      final Double mean = s.stream().mapToDouble(d -> d).sum() / s.size();
      final DoubleStream se = s.stream().mapToDouble(x -> Math.pow(x - mean, 2));
      final Double stdDev = Math.sqrt(se.sum() / s.size());
      return new Tuple3<>(stdDev, mean, s.size());
    });

final Double p = 0.01;
final Flow<Double, Double, NotUsed> sampleFlow = Flow.of(Double.class)
  .conflateWithSeed(elem -> Collections.singletonList(elem), (acc, elem) -> {
    if (r.nextDouble() < p) {
      return Stream
        .concat(acc.stream(), Collections.singletonList(elem).stream())
        .collect(Collectors.toList());
    }
    return acc;
  })
  .mapConcat(d -> d);

final Flow<Double, Double, NotUsed> lastFlow = Flow.of(Double.class)
  .expand(in -> Stream.iterate(in, i -> i).iterator());

	final Flow<Double, Pair<Double, Integer>, NotUsed> driftFlow = Flow.of(Double.class)
      .expand(d -> Stream.iterate(0, i -> i + 1).map(i -> new Pair<>(d, i)).iterator());