org.apache.spark.sql.hive.client

Interface HiveClient

public interface HiveClient

An externally visible interface to the Hive client. This interface is shared across both the internal and external classloaders for a given version of Hive and thus must expose only shared classes.

Method Summary

Modifier and Type	Method and Description
void	addJar(String path) Add a jar into class loader
void	alterDatabase(org.apache.spark.sql.catalyst.catalog.CatalogDatabase database) Alter a database whose name matches the one specified in database, assuming it exists.
void	alterFunction(String db, org.apache.spark.sql.catalyst.catalog.CatalogFunction func) Alter a function whose name matches the one specified in `func`, assuming it exists.
void	alterPartitions(String db, String table, scala.collection.Seq<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition> newParts) Alter one or more table partitions whose specs match the ones specified in newParts, assuming the partitions exist.
void	alterTable(org.apache.spark.sql.catalyst.catalog.CatalogTable table) Alter a table whose name matches the one specified in `table`, assuming it exists.
void	alterTable(String dbName, String tableName, org.apache.spark.sql.catalyst.catalog.CatalogTable table) Updates the given table with new metadata, optionally renaming the table or moving across different database.
void	alterTableDataSchema(String dbName, String tableName, StructType newDataSchema, scala.collection.immutable.Map<String,String> schemaProps) Updates the given table with a new data schema and table properties, and keep everything else unchanged.
void	createDatabase(org.apache.spark.sql.catalyst.catalog.CatalogDatabase database, boolean ignoreIfExists) Creates a new database with the given name.
void	createFunction(String db, org.apache.spark.sql.catalyst.catalog.CatalogFunction func) Create a function in an existing database.
void	createPartitions(String db, String table, scala.collection.Seq<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition> parts, boolean ignoreIfExists) Create one or many partitions in the given table.
void	createTable(org.apache.spark.sql.catalyst.catalog.CatalogTable table, boolean ignoreIfExists) Creates a table with the given metadata.
boolean	databaseExists(String dbName) Return whether a table/view with the specified name exists.
void	dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade) Drop the specified database, if it exists.
void	dropFunction(String db, String name) Drop an existing function in the database.
void	dropPartitions(String db, String table, scala.collection.Seq<scala.collection.immutable.Map<String,String>> specs, boolean ignoreIfNotExists, boolean purge, boolean retainData) Drop one or many partitions in the given table, assuming they exist.
void	dropTable(String dbName, String tableName, boolean ignoreIfNotExists, boolean purge) Drop the specified table.
boolean	functionExists(String db, String name) Return whether a function exists in the specified database.
String	getConf(String key, String defaultValue) Returns the configuration for the given key in the current session.
org.apache.spark.sql.catalyst.catalog.CatalogDatabase	getDatabase(String name) Returns the metadata for specified database, throwing an exception if it doesn't exist
org.apache.spark.sql.catalyst.catalog.CatalogFunction	getFunction(String db, String name) Return an existing function in the database, assuming it exists.
scala.Option<org.apache.spark.sql.catalyst.catalog.CatalogFunction>	getFunctionOption(String db, String name) Return an existing function in the database, or None if it doesn't exist.
org.apache.spark.sql.catalyst.catalog.CatalogTablePartition	getPartition(String dbName, String tableName, scala.collection.immutable.Map<String,String> spec) Returns the specified partition, or throws `NoSuchPartitionException`.
scala.collection.Seq<String>	getPartitionNames(org.apache.spark.sql.catalyst.catalog.CatalogTable table, scala.Option<scala.collection.immutable.Map<String,String>> partialSpec) Returns the partition names for the given table that match the supplied partition spec.
scala.Option<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition>	getPartitionOption(org.apache.spark.sql.catalyst.catalog.CatalogTable table, scala.collection.immutable.Map<String,String> spec) Returns the specified partition or None if it does not exist.
scala.Option<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition>	getPartitionOption(String db, String table, scala.collection.immutable.Map<String,String> spec) Returns the specified partition or None if it does not exist.
scala.collection.Seq<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition>	getPartitions(org.apache.spark.sql.catalyst.catalog.CatalogTable catalogTable, scala.Option<scala.collection.immutable.Map<String,String>> partialSpec) Returns the partitions for the given table that match the supplied partition spec.
scala.collection.Seq<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition>	getPartitions(String db, String table, scala.Option<scala.collection.immutable.Map<String,String>> partialSpec) Returns the partitions for the given table that match the supplied partition spec.
scala.collection.Seq<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition>	getPartitionsByFilter(org.apache.spark.sql.catalyst.catalog.CatalogTable catalogTable, scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression> predicates) Returns partitions filtered by predicates for the given table.
Object	getState() Return the associated Hive SessionState of this HiveClientImpl
org.apache.spark.sql.catalyst.catalog.CatalogTable	getTable(String dbName, String tableName) Returns the specified table, or throws `NoSuchTableException`.
scala.Option<org.apache.spark.sql.catalyst.catalog.CatalogTable>	getTableOption(String dbName, String tableName) Returns the metadata for the specified table or None if it doesn't exist.
scala.collection.Seq<org.apache.spark.sql.catalyst.catalog.CatalogTable>	getTablesByName(String dbName, scala.collection.Seq<String> tableNames) Returns metadata of existing permanent tables/views for given names.
scala.collection.Seq<String>	listDatabases(String pattern) List the names of all the databases that match the specified pattern.
scala.collection.Seq<String>	listFunctions(String db, String pattern) Return the names of all functions that match the given pattern in the database.
scala.collection.Seq<String>	listTables(String dbName) Returns the names of all tables in the given database.
scala.collection.Seq<String>	listTables(String dbName, String pattern) Returns the names of tables in the given database that matches the given pattern.
scala.collection.Seq<String>	listTablesByType(String dbName, String pattern, org.apache.spark.sql.catalyst.catalog.CatalogTableType tableType) Returns the names of tables with specific tableType in the given database that matches the given pattern.
void	loadDynamicPartitions(String loadPath, String dbName, String tableName, java.util.LinkedHashMap<String,String> partSpec, boolean replace, int numDP) Loads new dynamic partitions into an existing table.
void	loadPartition(String loadPath, String dbName, String tableName, java.util.LinkedHashMap<String,String> partSpec, boolean replace, boolean inheritTableSpecs, boolean isSrcLocal) Loads a static partition into an existing table.
void	loadTable(String loadPath, String tableName, boolean replace, boolean isSrcLocal) Loads data into an existing table.
HiveClient	newSession() Return a HiveClient as new session, that will share the class loader and Hive client
void	renameFunction(String db, String oldName, String newName) Rename an existing function in the database.
void	renamePartitions(String db, String table, scala.collection.Seq<scala.collection.immutable.Map<String,String>> specs, scala.collection.Seq<scala.collection.immutable.Map<String,String>> newSpecs) Rename one or many existing table partitions, assuming they exist.
void	reset() Used for testing only.
scala.collection.Seq<String>	runSqlHive(String sql) Runs a HiveQL command using Hive, returning the results as a list of strings.
void	setCurrentDatabase(String databaseName) Sets the name of current database.
void	setError(java.io.PrintStream stream)
void	setInfo(java.io.PrintStream stream)
void	setOut(java.io.PrintStream stream)
boolean	tableExists(String dbName, String tableName) Return whether a table/view with the specified name exists.
String	userName() Returns the user name which is used as owner for Hive table.
org.apache.spark.sql.hive.client.HiveVersion	version() Returns the Hive Version of this client.
<A> A	withHiveState(scala.Function0<A> f) Run a function within Hive state (SessionState, HiveConf, Hive client and class loader)

Method Detail

addJar

void addJar(String path)

Add a jar into class loader

alterDatabase

void alterDatabase(org.apache.spark.sql.catalyst.catalog.CatalogDatabase database)

Alter a database whose name matches the one specified in database, assuming it exists.

Parameters:

database - (undocumented)

alterFunction

void alterFunction(String db,
                   org.apache.spark.sql.catalyst.catalog.CatalogFunction func)

Alter a function whose name matches the one specified in `func`, assuming it exists.

alterPartitions

void alterPartitions(String db,
                     String table,
                     scala.collection.Seq<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition> newParts)

Alter one or more table partitions whose specs match the ones specified in newParts, assuming the partitions exist.

Parameters:

db - (undocumented)

table - (undocumented)

newParts - (undocumented)

alterTable

void alterTable(org.apache.spark.sql.catalyst.catalog.CatalogTable table)

Alter a table whose name matches the one specified in `table`, assuming it exists.

alterTable

void alterTable(String dbName,
                String tableName,
                org.apache.spark.sql.catalyst.catalog.CatalogTable table)

Updates the given table with new metadata, optionally renaming the table or moving across different database.

Parameters:

dbName - (undocumented)

tableName - (undocumented)

table - (undocumented)

alterTableDataSchema

void alterTableDataSchema(String dbName,
                          String tableName,
                          StructType newDataSchema,
                          scala.collection.immutable.Map<String,String> schemaProps)

Updates the given table with a new data schema and table properties, and keep everything else unchanged.

TODO(cloud-fan): it's a little hacky to introduce the schema table properties here in HiveClient, but we don't have a cleaner solution now.

Parameters:

dbName - (undocumented)

tableName - (undocumented)

newDataSchema - (undocumented)

schemaProps - (undocumented)

createDatabase

void createDatabase(org.apache.spark.sql.catalyst.catalog.CatalogDatabase database,
                    boolean ignoreIfExists)

Creates a new database with the given name.

createFunction

void createFunction(String db,
                    org.apache.spark.sql.catalyst.catalog.CatalogFunction func)

Create a function in an existing database.

createPartitions

void createPartitions(String db,
                      String table,
                      scala.collection.Seq<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition> parts,
                      boolean ignoreIfExists)

Create one or many partitions in the given table.

Parameters:

db - (undocumented)

table - (undocumented)

parts - (undocumented)

ignoreIfExists - (undocumented)

createTable

void createTable(org.apache.spark.sql.catalyst.catalog.CatalogTable table,
                 boolean ignoreIfExists)

Creates a table with the given metadata.

databaseExists

boolean databaseExists(String dbName)

Return whether a table/view with the specified name exists.

dropDatabase

void dropDatabase(String name,
                  boolean ignoreIfNotExists,
                  boolean cascade)

Drop the specified database, if it exists.

Parameters:

name - database to drop

ignoreIfNotExists - if true, do not throw error if the database does not exist

cascade - whether to remove all associated objects such as tables and functions

dropFunction

void dropFunction(String db,
                  String name)

Drop an existing function in the database.

dropPartitions

void dropPartitions(String db,
                    String table,
                    scala.collection.Seq<scala.collection.immutable.Map<String,String>> specs,
                    boolean ignoreIfNotExists,
                    boolean purge,
                    boolean retainData)

Drop one or many partitions in the given table, assuming they exist.

Parameters:

db - (undocumented)

table - (undocumented)

specs - (undocumented)

ignoreIfNotExists - (undocumented)

purge - (undocumented)

retainData - (undocumented)

dropTable

void dropTable(String dbName,
               String tableName,
               boolean ignoreIfNotExists,
               boolean purge)

Drop the specified table.

functionExists

boolean functionExists(String db,
                       String name)

Return whether a function exists in the specified database.

getConf

String getConf(String key,
               String defaultValue)

Returns the configuration for the given key in the current session.

getDatabase

org.apache.spark.sql.catalyst.catalog.CatalogDatabase getDatabase(String name)

Returns the metadata for specified database, throwing an exception if it doesn't exist

getFunction

org.apache.spark.sql.catalyst.catalog.CatalogFunction getFunction(String db,
                                                                  String name)

Return an existing function in the database, assuming it exists.

getFunctionOption

scala.Option<org.apache.spark.sql.catalyst.catalog.CatalogFunction> getFunctionOption(String db,
                                                                                      String name)

Return an existing function in the database, or None if it doesn't exist.

getPartition

org.apache.spark.sql.catalyst.catalog.CatalogTablePartition getPartition(String dbName,
                                                                         String tableName,
                                                                         scala.collection.immutable.Map<String,String> spec)

Returns the specified partition, or throws `NoSuchPartitionException`.

getPartitionNames

scala.collection.Seq<String> getPartitionNames(org.apache.spark.sql.catalyst.catalog.CatalogTable table,
                                               scala.Option<scala.collection.immutable.Map<String,String>> partialSpec)

Returns the partition names for the given table that match the supplied partition spec. If no partition spec is specified, all partitions are returned.

The returned sequence is sorted as strings.

Parameters:

table - (undocumented)

partialSpec - (undocumented)

Returns:

(undocumented)

getPartitionOption

scala.Option<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition> getPartitionOption(String db,
                                                                                             String table,
                                                                                             scala.collection.immutable.Map<String,String> spec)

Returns the specified partition or None if it does not exist.

getPartitionOption

scala.Option<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition> getPartitionOption(org.apache.spark.sql.catalyst.catalog.CatalogTable table,
                                                                                             scala.collection.immutable.Map<String,String> spec)

Returns the specified partition or None if it does not exist.

getPartitions

scala.collection.Seq<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition> getPartitions(String db,
                                                                                                String table,
                                                                                                scala.Option<scala.collection.immutable.Map<String,String>> partialSpec)

Returns the partitions for the given table that match the supplied partition spec. If no partition spec is specified, all partitions are returned.

Parameters:

db - (undocumented)

table - (undocumented)

partialSpec - (undocumented)

Returns:

(undocumented)

getPartitions

scala.collection.Seq<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition> getPartitions(org.apache.spark.sql.catalyst.catalog.CatalogTable catalogTable,
                                                                                                scala.Option<scala.collection.immutable.Map<String,String>> partialSpec)

Returns the partitions for the given table that match the supplied partition spec. If no partition spec is specified, all partitions are returned.

Parameters:

catalogTable - (undocumented)

partialSpec - (undocumented)

Returns:

(undocumented)

getPartitionsByFilter

scala.collection.Seq<org.apache.spark.sql.catalyst.catalog.CatalogTablePartition> getPartitionsByFilter(org.apache.spark.sql.catalyst.catalog.CatalogTable catalogTable,
                                                                                                        scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression> predicates)

Returns partitions filtered by predicates for the given table.

getState

Object getState()

Return the associated Hive SessionState of this HiveClientImpl

Returns:

Any not SessionState to avoid linkage error

getTable

org.apache.spark.sql.catalyst.catalog.CatalogTable getTable(String dbName,
                                                            String tableName)

Returns the specified table, or throws `NoSuchTableException`.

getTableOption

scala.Option<org.apache.spark.sql.catalyst.catalog.CatalogTable> getTableOption(String dbName,
                                                                                String tableName)

Returns the metadata for the specified table or None if it doesn't exist.

getTablesByName

scala.collection.Seq<org.apache.spark.sql.catalyst.catalog.CatalogTable> getTablesByName(String dbName,
                                                                                         scala.collection.Seq<String> tableNames)

Returns metadata of existing permanent tables/views for given names.

listDatabases

scala.collection.Seq<String> listDatabases(String pattern)

List the names of all the databases that match the specified pattern.

listFunctions

scala.collection.Seq<String> listFunctions(String db,
                                           String pattern)

Return the names of all functions that match the given pattern in the database.

listTables

scala.collection.Seq<String> listTables(String dbName)

Returns the names of all tables in the given database.

listTables

scala.collection.Seq<String> listTables(String dbName,
                                        String pattern)

Returns the names of tables in the given database that matches the given pattern.

listTablesByType

scala.collection.Seq<String> listTablesByType(String dbName,
                                              String pattern,
                                              org.apache.spark.sql.catalyst.catalog.CatalogTableType tableType)

Returns the names of tables with specific tableType in the given database that matches the given pattern.

Parameters:

dbName - (undocumented)

pattern - (undocumented)

tableType - (undocumented)

Returns:

(undocumented)

loadDynamicPartitions

void loadDynamicPartitions(String loadPath,
                           String dbName,
                           String tableName,
                           java.util.LinkedHashMap<String,String> partSpec,
                           boolean replace,
                           int numDP)

Loads new dynamic partitions into an existing table.

loadPartition

void loadPartition(String loadPath,
                   String dbName,
                   String tableName,
                   java.util.LinkedHashMap<String,String> partSpec,
                   boolean replace,
                   boolean inheritTableSpecs,
                   boolean isSrcLocal)

Loads a static partition into an existing table.

loadTable

void loadTable(String loadPath,
               String tableName,
               boolean replace,
               boolean isSrcLocal)

Loads data into an existing table.

newSession

HiveClient newSession()

Return a HiveClient as new session, that will share the class loader and Hive client

renameFunction

void renameFunction(String db,
                    String oldName,
                    String newName)

Rename an existing function in the database.

renamePartitions

void renamePartitions(String db,
                      String table,
                      scala.collection.Seq<scala.collection.immutable.Map<String,String>> specs,
                      scala.collection.Seq<scala.collection.immutable.Map<String,String>> newSpecs)

Rename one or many existing table partitions, assuming they exist.

Parameters:

db - (undocumented)

table - (undocumented)

specs - (undocumented)

newSpecs - (undocumented)

reset

void reset()

Used for testing only. Removes all metadata from this instance of Hive.

runSqlHive

scala.collection.Seq<String> runSqlHive(String sql)

Runs a HiveQL command using Hive, returning the results as a list of strings. Each row will result in one string.

Parameters:

sql - (undocumented)

Returns:

(undocumented)

setCurrentDatabase

void setCurrentDatabase(String databaseName)

Sets the name of current database.

setError

void setError(java.io.PrintStream stream)

setInfo

void setInfo(java.io.PrintStream stream)

setOut

void setOut(java.io.PrintStream stream)

tableExists

boolean tableExists(String dbName,
                    String tableName)

Return whether a table/view with the specified name exists.

userName

String userName()

Returns the user name which is used as owner for Hive table.

version

org.apache.spark.sql.hive.client.HiveVersion version()

Returns the Hive Version of this client.

withHiveState

<A> A withHiveState(scala.Function0<A> f)

Run a function within Hive state (SessionState, HiveConf, Hive client and class loader)