org.apache.spark.sql.jdbc

Class JDBCRDD

Object

org.apache.spark.rdd.RDD<Row>

org.apache.spark.sql.jdbc.JDBCRDD

All Implemented Interfaces:

java.io.Serializable, Logging

public class JDBCRDD
extends RDD<Row>

An RDD representing a table in a database accessed via JDBC. Both the driver code and the workers must be able to access the database; the driver needs to fetch the schema while the workers need to fetch the data.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	JDBCRDD.BinaryConversion$
class	JDBCRDD.BinaryLongConversion$
class	JDBCRDD.BooleanConversion$
class	JDBCRDD.DateConversion$
class	JDBCRDD.DecimalConversion$
class	JDBCRDD.DoubleConversion$
class	JDBCRDD.FloatConversion$
class	JDBCRDD.IntegerConversion$
class	JDBCRDD.JDBCConversion
class	JDBCRDD.LongConversion$
class	JDBCRDD.StringConversion$
class	JDBCRDD.TimestampConversion$

Constructor Summary

Constructors

Constructor and Description
JDBCRDD(SparkContext sc, scala.Function0<java.sql.Connection> getConnection, StructType schema, String fqTable, String[] columns, Filter[] filters, Partition[] partitions)

Method Summary

Methods

Modifier and Type	Method and Description
JDBCRDD.BinaryConversion$	BinaryConversion() Accessor for nested Scala object
JDBCRDD.BinaryLongConversion$	BinaryLongConversion() Accessor for nested Scala object
JDBCRDD.BooleanConversion$	BooleanConversion() Accessor for nested Scala object
scala.collection.Iterator<Row>	compute(Partition thePart, TaskContext context) Runs the SQL query against the JDBC driver.
JDBCRDD.DateConversion$	DateConversion() Accessor for nested Scala object
JDBCRDD.DecimalConversion$	DecimalConversion() Accessor for nested Scala object
JDBCRDD.DoubleConversion$	DoubleConversion() Accessor for nested Scala object
JDBCRDD.FloatConversion$	FloatConversion() Accessor for nested Scala object
static scala.Function0<java.sql.Connection>	getConnector(String driver, String url, java.util.Properties properties) Given a driver string and an url, return a function that loads the specified driver string then returns a connection to the JDBC url.
JDBCRDD.JDBCConversion[]	getConversions(StructType schema) Maps a StructType to a type tag list.
Partition[]	getPartitions() Retrieve the list of partitions corresponding to this RDD.
JDBCRDD.IntegerConversion$	IntegerConversion() Accessor for nested Scala object
JDBCRDD.LongConversion$	LongConversion() Accessor for nested Scala object
static StructType	resolveTable(String url, String table, java.util.Properties properties) Takes a (schema, table) specification and returns the table's Catalyst schema.
static RDD<Row>	scanTable(SparkContext sc, StructType schema, String driver, String url, java.util.Properties properties, String fqTable, String[] requiredColumns, Filter[] filters, Partition[] parts) Build and return JDBCRDD from the given information.
JDBCRDD.StringConversion$	StringConversion() Accessor for nested Scala object
JDBCRDD.TimestampConversion$	TimestampConversion() Accessor for nested Scala object

Methods inherited from class org.apache.spark.rdd.RDD

aggregate, cache, cartesian, checkpoint, checkpointData, coalesce, collect, collect, collectPartitions, computeOrReadCheckpoint, conf, context, count, countApprox, countApproxDistinct, countApproxDistinct, countByValue, countByValueApprox, creationSite, dependencies, distinct, distinct, doCheckpoint, doubleRDDToDoubleRDDFunctions, elementClassTag, filter, filterWith, first, flatMap, flatMapWith, fold, foreach, foreachPartition, foreachWith, getCheckpointFile, getCreationSite, getNarrowAncestors, getStorageLevel, glom, groupBy, groupBy, groupBy, id, intersection, intersection, intersection, isCheckpointed, isEmpty, iterator, keyBy, map, mapPartitions, mapPartitionsWithContext, mapPartitionsWithIndex, mapPartitionsWithSplit, mapWith, markCheckpointed, max, min, name, numericRDDToDoubleRDDFunctions, partitioner, partitions, persist, persist, pipe, pipe, pipe, preferredLocations, randomSplit, rddToAsyncRDDActions, rddToOrderedRDDFunctions, rddToPairRDDFunctions, rddToSequenceFileRDDFunctions, reduce, repartition, retag, retag, sample, saveAsObjectFile, saveAsTextFile, saveAsTextFile, setName, sortBy, sparkContext, subtract, subtract, subtract, take, takeOrdered, takeSample, toArray, toDebugString, toJavaRDD, toLocalIterator, top, toString, treeAggregate, treeReduce, union, unpersist, zip, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipWithIndex, zipWithUniqueId

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

JDBCRDD

public JDBCRDD(SparkContext sc,
       scala.Function0<java.sql.Connection> getConnection,
       StructType schema,
       String fqTable,
       String[] columns,
       Filter[] filters,
       Partition[] partitions)

Method Detail

resolveTable

public static StructType resolveTable(String url,
                      String table,
                      java.util.Properties properties)

Takes a (schema, table) specification and returns the table's Catalyst schema.

Parameters:

url - - The JDBC url to fetch information from.

table - - The table name of the desired table. This may also be a SQL query wrapped in parentheses.

Returns:

A StructType giving the table's Catalyst schema.

Throws:

SQLException - if the table specification is garbage.

SQLException - if the table contains an unsupported type.

getConnector

public static scala.Function0<java.sql.Connection> getConnector(String driver,
                                                String url,
                                                java.util.Properties properties)

Given a driver string and an url, return a function that loads the specified driver string then returns a connection to the JDBC url. getConnector is run on the driver code, while the function it returns is run on the executor.

Parameters:

driver - - The class name of the JDBC driver for the given url.

url - - The JDBC url to connect to.

Returns:

A function that loads the driver and connects to the url.

scanTable

public static RDD<Row> scanTable(SparkContext sc,
                 StructType schema,
                 String driver,
                 String url,
                 java.util.Properties properties,
                 String fqTable,
                 String[] requiredColumns,
                 Filter[] filters,
                 Partition[] parts)

Build and return JDBCRDD from the given information.

Parameters:

sc - - Your SparkContext.

schema - - The Catalyst schema of the underlying database table.

driver - - The class name of the JDBC driver for the given url.

url - - The JDBC url to connect to.

fqTable - - The fully-qualified table name (or paren'd SQL query) to use.

requiredColumns - - The names of the columns to SELECT.

filters - - The filters to include in all WHERE clauses.

parts - - An array of JDBCPartitions specifying partition ids and per-partition WHERE clauses.

Returns:

An RDD representing "SELECT requiredColumns FROM fqTable".

getPartitions

public Partition[] getPartitions()

Retrieve the list of partitions corresponding to this RDD.

BooleanConversion

public JDBCRDD.BooleanConversion$ BooleanConversion()

Accessor for nested Scala object

DateConversion

public JDBCRDD.DateConversion$ DateConversion()

Accessor for nested Scala object

DecimalConversion

public JDBCRDD.DecimalConversion$ DecimalConversion()

Accessor for nested Scala object

DoubleConversion

public JDBCRDD.DoubleConversion$ DoubleConversion()

Accessor for nested Scala object

FloatConversion

public JDBCRDD.FloatConversion$ FloatConversion()

Accessor for nested Scala object

IntegerConversion

public JDBCRDD.IntegerConversion$ IntegerConversion()

Accessor for nested Scala object

LongConversion

public JDBCRDD.LongConversion$ LongConversion()

Accessor for nested Scala object

BinaryLongConversion

public JDBCRDD.BinaryLongConversion$ BinaryLongConversion()

Accessor for nested Scala object

StringConversion

public JDBCRDD.StringConversion$ StringConversion()

Accessor for nested Scala object

TimestampConversion

public JDBCRDD.TimestampConversion$ TimestampConversion()

Accessor for nested Scala object

BinaryConversion

public JDBCRDD.BinaryConversion$ BinaryConversion()

Accessor for nested Scala object

getConversions

public JDBCRDD.JDBCConversion[] getConversions(StructType schema)

Maps a StructType to a type tag list.

compute

public scala.collection.Iterator<Row> compute(Partition thePart,
                                     TaskContext context)

Runs the SQL query against the JDBC driver.

Specified by:

compute in class RDD<Row>