How to call FileIO.Write.via(Contextful, Contextful) in Scala

I’m using Apache Beam with Scala. To create new ParquetIO.Sink instances with the respective schemas for the different types of streaming data, I’m trying to call .via(Contextful, Contextful) on FileIO.Write. However, IntelliJ cannot find the overloaded method that I’m trying to use and raises this error: Cannot resolve overloaded method 'via'.

FileIO
  .writeDynamic[String, DeserializedEvent]()
  .by(new UDFs.PartitionByEventName())
  .withDestinationCoder(StringUtf8Coder.of())
  .withNumShards(numShards)
  .withNaming(new UDFs.NameFiles())
  .via(
    Contextful.fn[DeserializedEvent, GenericRecord](
      new UDFs.EventToGenericRecord() // SerializableFunction[DeserializedEvent, String]
    ),
    Contextful.fn[String, ParquetIO.Sink](
      new UDFs.SinkParquet() // SerializableFunction[String, ParquetIO.Sink]
    )
  )
  .to(path)

What’s the issue here?

Thanks

Answer

Scala’s type inference system and overload resolution system don’t work together quite perfectly. In this case, the overloads of the via() function are preventing type inference from working properly. Why does scala fail to compile when method is overloaded in a seemingly unrelated way? has a good explanation of the problem.

In your case, you should just need to change the type params of the second Contextful.fn to [String, FileIO.Sink] to make it match the signature of the desired overload.

Leave a Reply

Your email address will not be published. Required fields are marked *