StructField issue at compilation when specifying table schema

I want to specify a schema for a table I do it in JAVA using the following code :

StructType schema =  new StructType(List
                (
                        StructField("id", NumericType,  true),
                        StructField("text", StringType,  true),
                        StructField("user", StringType,  true)
                ));

BUT NumericType, StringType give error saying “expecting expression”. How to solve this issue?

Thank you.

Answer

Your code contains some mistakes. You need to use new keyword to create StructField and you cannot use List(StructField, ...). Also, there is no NumericType in spark sql types, you probably want to use IntegerType. See available Spark DataTypes.

That said, you can try with this :

StructType schema = new StructType(new StructField[]{
                new StructField("id", IntegerType, true, null),
                new StructField("text", StringType, true, null),
                new StructField("user", StringType, true, null)
        });

Or using static import with the provided methods from DataTypes factory:

import static org.apache.spark.sql.types.DataTypes.*;

StructType schema = createStructType(Arrays.asList(
                createStructField("id", IntegerType, true),
                createStructField("text", StringType, true),
                createStructField("user", StringType, true)
        ));

Leave a Reply

Your email address will not be published. Required fields are marked *