Kafka producer hangs if topic doesn’t exist

I’m new to Kafka and trying to implement a simple producer, sending data to a topic. If the topic doesn’t exist, I want to handle the sutiation as an exception.

private Producer<UUID, Object> producer = createProducer(); 

private static Producer createProducer() {
    Properties props = new Properties();
    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
            "mybootstrapserveraddress");
    props.put(ProducerConfig.CLIENT_ID_CONFIG, "ADAPTER");
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
            UUIDSerializer.class);
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
            JsonSerializer.class);
    props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG,
            1000);
    props.put(ProducerConfig.RETRIES_CONFIG,
            1);
    props.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG,
            1000);
    return new KafkaProducer<>(props);
}

public void send(Event event, String topic){
    try {
        UUID key = UUID.randomUUID();
        producer.send(new ProducerRecord<>(topic, key , event), (rm, ex) -> {
            if (ex != null) {
                log.warn("Error sending message with key {}n{}", new Object[]{key, ex.getMessage()});
            } else {
                log.info( "Partition for key-value {} is {}", new Object[]{key, rm.partition()});
            }
        });
    } catch (Exception e) {
        log.error("Failed to send message ",e);
    } finally {
        producer.flush();
    }
}

However, if the topic doesn’t exist, the message continues to be polled. Timeouts and retries from ProducerConfig are ignored.

[kafka-producer-network-thread | ADAPTER] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=ADAPTER] Error while fetching metadata with correlation id 6 : {my-example-topic2=UNKNOWN_TOPIC_OR_PARTITION}
[kafka-producer-network-thread | ADAPTER] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=ADAPTER] Error while fetching metadata with correlation id 7 : {my-example-topic2=UNKNOWN_TOPIC_OR_PARTITION}
[kafka-producer-network-thread | ADAPTER] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=ADAPTER] Error while fetching metadata with correlation id 8 : {my-example-topic2=UNKNOWN_TOPIC_OR_PARTITION}

I do not want to check that the topic exist through AdminClient. Kafka document https://kafka.apache.org/documentation/#producerconfigs was of no help.

Is there a way to fix the issue?

Answer

When the topic doesn’t exist the retry for getting metadata should ends after 60 seconds by default raising a timeout exception at the end. The producer config parameter related to that is max.block.ms (default 60000). As far as I know, there is no way to have feedback earlier than reducing this timeout or using the AdminClient (which is something you don’t want to do).

Leave a Reply

Your email address will not be published. Required fields are marked *