org.apache.spark.sql.json

Class JsonRDD

Object

org.apache.spark.sql.json.JsonRDD

All Implemented Interfaces:

Logging

public class JsonRDD
extends Object
implements Logging

Constructor Summary

Constructors

Constructor and Description
JsonRDD()

Method Summary

Methods

Modifier and Type	Method and Description
static org.apache.spark.sql.catalyst.types.DataType	compatibleType(org.apache.spark.sql.catalyst.types.DataType t1, org.apache.spark.sql.catalyst.types.DataType t2) Returns the most general data type for two given data types.
static Object	enforceCorrectType(Object value, org.apache.spark.sql.catalyst.types.DataType desiredType)
static org.apache.spark.sql.catalyst.types.StructType	inferSchema(RDD<String> json, double samplingRatio, String columnNameOfCorruptRecords)
static RDD<org.apache.spark.sql.catalyst.expressions.Row>	jsonStringToRow(RDD<String> json, org.apache.spark.sql.catalyst.types.StructType schema, String columnNameOfCorruptRecords)
static org.apache.spark.sql.catalyst.types.StructType	nullTypeToStringType(org.apache.spark.sql.catalyst.types.StructType struct)
static String	rowToJSON(org.apache.spark.sql.catalyst.types.StructType rowSchema, com.fasterxml.jackson.core.JsonFactory jsonFactory, org.apache.spark.sql.catalyst.expressions.Row row) Transforms a single Row to JSON using Jackson

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

JsonRDD

public JsonRDD()

Method Detail

jsonStringToRow

public static RDD<org.apache.spark.sql.catalyst.expressions.Row> jsonStringToRow(RDD<String> json,
                                                                 org.apache.spark.sql.catalyst.types.StructType schema,
                                                                 String columnNameOfCorruptRecords)

inferSchema

public static org.apache.spark.sql.catalyst.types.StructType inferSchema(RDD<String> json,
                                                         double samplingRatio,
                                                         String columnNameOfCorruptRecords)

nullTypeToStringType

public static org.apache.spark.sql.catalyst.types.StructType nullTypeToStringType(org.apache.spark.sql.catalyst.types.StructType struct)

compatibleType

public static org.apache.spark.sql.catalyst.types.DataType compatibleType(org.apache.spark.sql.catalyst.types.DataType t1,
                                                          org.apache.spark.sql.catalyst.types.DataType t2)

Returns the most general data type for two given data types.

enforceCorrectType

public static Object enforceCorrectType(Object value,
                        org.apache.spark.sql.catalyst.types.DataType desiredType)

rowToJSON

public static String rowToJSON(org.apache.spark.sql.catalyst.types.StructType rowSchema,
               com.fasterxml.jackson.core.JsonFactory jsonFactory,
               org.apache.spark.sql.catalyst.expressions.Row row)

Transforms a single Row to JSON using Jackson

Parameters:

jsonFactory - a JsonFactory object to construct a JsonGenerator

rowSchema - the schema object used for conversion

row - The row to convert