Delay in Kafka Log4j appender

So here is my setup, and this is all being done in Eclipse

  1. I am trying to implement the Kafka Log4j appender to pump messages into Kafka.

  2. I have some java code setup to create the log messages:

    public static void main(String[] args) {
    
    SimpleDateFormat sdf = new SimpleDateFormat();
    
    log.debug("Debug message at "+sdf.format(new Date()));
    
    log.info("Info message at "+sdf.format(new Date()));
    
    log.error("Error Message at "+sdf.format(new Date()));
    
    log.fatal("Fatal Message at "+sdf.format(new Date()));
    
    log.warn("Warn Message at "+sdf.format(new Date()));
    
    log.trace("Trace Message at "+sdf.format(new Date()));
    }
    
  3. here are my log4j.properties

    log4j.rootLogger=TRACE, stdout
    log4j.appender.stdout=org.apache.log4j.ConsoleAppender
    log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
    log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% %m%n
    
    log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
    log4j.appender.KAFKA.BrokerList=localhost:9092
    log4j.appender.KAFKA.Topic=kfkLogs
    log4j.appender.KAFKA.producer.type=async
    log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
    log4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n
    
    log4j.logger.logGen.TestLog4j=TRACE, KAFKA
    
  4. In another instance of Eclipse, i have my Kafka consumer code running which basically prints out any messages it receieves

Here is my problem,

When i execute my log producer, i do not always see the messages in the consumer. sometimes i see the messages and sometimes i do not. Sometimes i can execute the producer repeatedly for a few seconds and each time i see the messages, but other times, the consumer only gets them once and not for the next minute or so. then it starts receiving again. so the messages that i pushed in the meantime are lost.

Is there anyway that there can be a delay when Kafka published the messages ?

Can it have something to do with my Kafka or Zookeeper configurations ?

I am executing everything on localhost

Please advise.

Answer

log4j.appender.KAFKA.producer.type=sync

fixed it.

Leave a Reply

Your email address will not be published. Required fields are marked *