streaming response body to a streaming request body with java 11 HttpClient

I’m trying stream the data from an HTTP (GET) response to another HTTP (POST) request. With old HttpURLConnection I would take the responses OutputStream, read parts into a buffer and write them to the requests InputStream.

I’ve already managed to do the same with HttpClient in Java 11 by creating my own Publisher that is used in the POST to write the request body. The GET request has a BodyHandler with ofByteArrayConsumer that sends the chunks to the custom Publisher which itself then sends the chunks to the subscribing HTTP POST request.

But I think this is not the correct approach as it looks like there is something in the API that looks like this could be done directly without implementing publishers and subscribers myself.

There is HttpResponse.BodyHandlers.ofPublisher() which returns a Publisher<List<ByteBuffer> which I can use for the HTTP GET request. Unfortunately for my POST request, there is HttpRequest.BodyPublishers.fromPublisher which expects a Publisher<? extends ByteBuffer> so it seems that the fromPublisher only works for a publisher that holds a complete ByteBuffer and not one that sends several ByteBuffers for parts of the data.

Do I miss something here to be able to connect the BodyPublisher from one request to the other?

Answer

You’re not missing anything. This is simply a use case that is not supported out of the box for now. Though the mapping from ByteBuffer to List<ByteBuffer> is trivial, the inverse mapping is less so. One easy (if not optimal) way to adapt from one to the other could be to collect all the buffers in the list into a single buffer – possibly combining HttpResponse.BodyHandlers.ofPublisher() with HttpResponse.BodyHandlers.buffering() if you want to control the amount of bytes in each published List<ByteBuffer> that you receive from upstream.

Leave a Reply

Your email address will not be published. Required fields are marked *