Home | 简体中文 | 繁体中文 | 杂文 | 打赏(Donations) | ITEYE 博客 | OSChina 博客 | Facebook | Linkedin | 知乎专栏 | Search | About

19.3. sqoop2-shell

进入 sqoop2-shell

		
[hadoop@netkiller ~]$ sqoop2-shell 
Setting conf dir: /srv/apache-sqoop/bin/../conf
Sqoop home directory: /srv/apache-sqoop
Sqoop Shell: Type 'help' or '\h' for help.

sqoop:000>
		
		

Sqoop client script:

		
sqoop2-shell /path/to/your/script.sqoop
		
		

19.3.1. show version

			
sqoop:000> show version
client version:
  Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb 
  Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
			
			
			
sqoop:000> show version --all 
client version:
  Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb 
  Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
0    [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
server version:
  Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb 
  Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
API versions:
  [v1]			
			
			

19.3.2. set

19.3.2.1. server

				
sqoop:000> set server --host master --port 12000 --webapp sqoop
Server is set successfully			
				
				

19.3.2.2. 要设置可查看具体出错信息

				
sqoop:000> set option --name verbose --value true
Verbose option was changed to true				
				
				

19.3.3. show connector

sqoop:000> show connector
0    [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+------------------------+---------+------------------------------------------------------------+----------------------+
|          Name          | Version |                           Class                            | Supported Directions |
+------------------------+---------+------------------------------------------------------------+----------------------+
| generic-jdbc-connector | 1.99.7  | org.apache.sqoop.connector.jdbc.GenericJdbcConnector       | FROM/TO              |
| kite-connector         | 1.99.7  | org.apache.sqoop.connector.kite.KiteConnector              | FROM/TO              |
| oracle-jdbc-connector  | 1.99.7  | org.apache.sqoop.connector.jdbc.oracle.OracleJdbcConnector | FROM/TO              |
| ftp-connector          | 1.99.7  | org.apache.sqoop.connector.ftp.FtpConnector                | TO                   |
| hdfs-connector         | 1.99.7  | org.apache.sqoop.connector.hdfs.HdfsConnector              | FROM/TO              |
| kafka-connector        | 1.99.7  | org.apache.sqoop.connector.kafka.KafkaConnector            | TO                   |
| sftp-connector         | 1.99.7  | org.apache.sqoop.connector.sftp.SftpConnector              | TO                   |
+------------------------+---------+------------------------------------------------------------+----------------------+
sqoop:000>			
sqoop list-databases --connect  jdbc:mysql://192.168.1.1:3306/ --username root --password 123456
			
			
sqoop:000> show connector --all
7 connector(s) to show: 
Connector with Name: generic-jdbc-connector 
  Class: 1.99.7
  Version: FROM/TO
  Supported Directions {4}
    link config 1:
      Name: linkConfig
      Label: Database connection
      Help: Contains configuration that is required to establish connection with your database server.
      Input 1:
        Name: linkConfig.jdbcDriver
        Label: Driver class
        Help: Fully qualified class name of the JDBC driver that will be used for establishing this connection. Check documentation for instructions how to make the driver's jar files available to Sqoop 2 server.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 128
      Input 2:
        Name: linkConfig.connectionString
        Label: Connection String
        Help: JDBC connection string associated with your database server.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 2000
      Input 3:
        Name: linkConfig.username
        Label: Username
        Help: Username to be used for connection to the database server.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 40
      Input 4:
        Name: linkConfig.password
        Label: Password
        Help: Password to be used for connection to the database server.
        Type: STRING
        Sensitive: true
        Editable By: ANY
        Overrides: 
        Size: 40
      Input 5:
        Name: linkConfig.fetchSize
        Label: Fetch Size
        Help: Optional hint specifying requested JDBC fetch size.
        Type: INTEGER
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 6:
        Name: linkConfig.jdbcProperties
        Label: Connection Properties
        Help: Key-value pairs that should be passed down to JDBC driver when establishing connection.
        Type: MAP
        Sensitive: false
        Editable By: ANY
        Overrides: 
    link config 2:
      Name: dialect
      Label: SQL Dialect
      Help: Database dialect that should be used for generated queries.
      Input 1:
        Name: dialect.identifierEnclose
        Label: Identifier enclose
        Help: Character(s) that should be used to enclose table name, schema or column names.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 5
    FROM Job config 1:
      Name: fromJobConfig
      Label: Database source
      Help: Specifies source and way how the data should be fetched from source database.
      Input 1:
        Name: fromJobConfig.schemaName
        Label: Schema name
        Help: Schema name if the table is not stored in default schema. Note: Not all database systems understands the concept of schema.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 50
      Input 2:
        Name: fromJobConfig.tableName
        Label: Table name
        Help: Input table name from from which data will be retrieved.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 50
      Input 3:
        Name: fromJobConfig.sql
        Label: SQL statement
        Help: Import data from given query's results set rather then static table.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 2000
      Input 4:
        Name: fromJobConfig.columnList
        Label: Column names
        Help: Subset of columns that should be retrieved from source table.
        Type: LIST
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 5:
        Name: fromJobConfig.partitionColumn
        Label: Partition column
        Help: Input column that should be use to split the import into independent parallel processes. This column will be used in condition of generated queries.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 50
      Input 6:
        Name: fromJobConfig.allowNullValueInPartitionColumn
        Label: Partition column nullable
        Help: Set true if partition column can contain NULL value.
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 7:
        Name: fromJobConfig.boundaryQuery
        Label: Boundary query
        Help: Customize query to retrieve minimal and maximal value of partition column.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 50
    FROM Job config 2:
      Name: incrementalRead
      Label: Incremental read
      Help: Configures optional incremental read from the database where source data are changing over time and only new changes need to be re-imported.
      Input 1:
        Name: incrementalRead.checkColumn
        Label: Check column
        Help: Column that is checked during incremental read for new values.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 50
      Input 2:
        Name: incrementalRead.lastValue
        Label: Last value
        Help: Last imported value, job will read only newer values.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: -1
    TO Job config 1:
      Name: toJobConfig
      Label: Database target
      Help: Describes target destination and way how data should be persisted on the RDBMS system.
      Input 1:
        Name: toJobConfig.schemaName
        Label: Schema name
        Help: Schema name if the table is not stored in default schema. Note: Not all database systems understands the concept of schema.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 50
      Input 2:
        Name: toJobConfig.tableName
        Label: Table name
        Help: Destination table name to store transfer results.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 2000
      Input 3:
        Name: toJobConfig.columnList
        Label: Column names
        Help: Subset of columns that will will be written to. Omitted columns have to either allow NULL values or have defined default value.
        Type: LIST
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 4:
        Name: toJobConfig.stageTableName
        Label: Staging table
        Help: Name of table with same structure as final table that should be used as a staging destination. Data will be directly written to final table if no staging table is specified.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 2000
      Input 5:
        Name: toJobConfig.shouldClearStageTable
        Label: Clear stage table
        Help: If set to true, staging table will be wiped out upon job start.
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
Connector with Name: kite-connector 
  Class: 1.99.7
  Version: FROM/TO
  Supported Directions {4}
    link config 1:
      Name: linkConfig
      Label: Global configuration
      Help: Global configuration options that will be used for both from and to sides.
      Input 1:
        Name: linkConfig.authority
        Label: HDFS host and port
        Help: Optional to override HDFS file system location.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
      Input 2:
        Name: linkConfig.confDir
        Label: Hadoop conf directory
        Help: Directory with Hadoop configuration files. This directory will be added to the classpath.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
    FROM Job config 1:
      Name: fromJobConfig
      Label: Source configuration
      Help: Configuration options relevant to source dataset.
      Input 1:
        Name: fromJobConfig.uri
        Label: dataset:hdfs://namespace/table
        Help: Kite Dataset URI from which data will be read.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
    TO Job config 1:
      Name: toJobConfig
      Label: Target configuration
      Help: Configuration options relevant to target dataset.
      Input 1:
        Name: toJobConfig.uri
        Label: Dataset URI
        Help: Kite Dataset URI where should be data written to.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
      Input 2:
        Name: toJobConfig.fileFormat
        Label: File format
        Help: Storage format that should be used when creating new dataset.
        Type: ENUM
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Possible values: CSV,AVRO,PARQUET
Connector with Name: oracle-jdbc-connector 
  Class: 1.99.7
  Version: FROM/TO
  Supported Directions {4}
    link config 1:
      Name: connectionConfig
      Label: Oracle connection configuration
      Help: You must supply the information requested in order to create an Oracle connection object.
      Input 1:
        Name: connectionConfig.connectionString
        Label: JDBC connection string
        Help: Enter the value of JDBC connection string to be used by this connector for creating Oracle connections.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 128
      Input 2:
        Name: connectionConfig.username
        Label: Username
        Help: Enter the username to be used for connecting to the database.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 40
      Input 3:
        Name: connectionConfig.password
        Label: Password
        Help: Enter the password to be used for connecting to the database.
        Type: STRING
        Sensitive: true
        Editable By: ANY
        Overrides: 
        Size: 40
      Input 4:
        Name: connectionConfig.jdbcProperties
        Label: JDBC connection properties
        Help: Enter any JDBC properties that should be supplied during the creation of connection.
        Type: MAP
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 5:
        Name: connectionConfig.timeZone
        Label: Session time zone
        Help: timeZone
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: -1
      Input 6:
        Name: connectionConfig.actionName
        Label: Session action name
        Help: actionName
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: -1
      Input 7:
        Name: connectionConfig.fetchSize
        Label: JDBC fetch size
        Help: fetchSize
        Type: INTEGER
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 8:
        Name: connectionConfig.initializationStatements
        Label: Session initialization statements
        Help: initializationStatements
        Type: LIST
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 9:
        Name: connectionConfig.jdbcUrlVerbatim
        Label: Use JDBC connection string verbatim
        Help: jdbcUrlVerbatim
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 10:
        Name: connectionConfig.racServiceName
        Label: RAC service name
        Help: racServiceName
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: -1
    FROM Job config 1:
      Name: fromJobConfig
      Label: From Oracle configuration
      Help: You must supply the information requested in order to create the FROM part of the job object.
      Input 1:
        Name: fromJobConfig.tableName
        Label: Table name
        Help: tableName
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 2000
      Input 2:
        Name: fromJobConfig.columns
        Label: Columns
        Help: Columns
        Type: LIST
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 3:
        Name: fromJobConfig.consistentRead
        Label: Consistent read
        Help: consistentRead
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 4:
        Name: fromJobConfig.consistentReadScn
        Label: Consistent read SCN
        Help: consistentReadScn
        Type: LONG
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 5:
        Name: fromJobConfig.partitionList
        Label: Partitions
        Help: partitionList
        Type: LIST
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 6:
        Name: fromJobConfig.dataChunkMethod
        Label: Data chunk method
        Help: dataChunkMethod
        Type: ENUM
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Possible values: ROWID,PARTITION
      Input 7:
        Name: fromJobConfig.dataChunkAllocationMethod
        Label: Data chunk allocation method
        Help: dataChunkAllocationMethod
        Type: ENUM
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Possible values: ROUNDROBIN,SEQUENTIAL,RANDOM
      Input 8:
        Name: fromJobConfig.whereClauseLocation
        Label: Where clause location
        Help: whereClauseLocation
        Type: ENUM
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Possible values: SUBSPLIT,SPLIT
      Input 9:
        Name: fromJobConfig.omitLobColumns
        Label: Omit LOB columns
        Help: omitLobColumns
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 10:
        Name: fromJobConfig.queryHint
        Label: Query hint
        Help: queryHint
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 2000
      Input 11:
        Name: fromJobConfig.conditions
        Label: Conditions
        Help: conditions
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 2000
    TO Job config 1:
      Name: toJobConfig
      Label: To database configuration
      Help: You must supply the information requested in order to create the TO part of the job object.
      Input 1:
        Name: toJobConfig.tableName
        Label: Table name
        Help: Table name to write data into
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 2000
      Input 2:
        Name: toJobConfig.columns
        Label: Columns
        Help: Columns
        Type: LIST
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 3:
        Name: toJobConfig.templateTable
        Label: Template table name
        Help: templateTable
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 2000
      Input 4:
        Name: toJobConfig.partitioned
        Label: Partitioned
        Help: partitioned
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 5:
        Name: toJobConfig.nologging
        Label: Nologging
        Help: nologging
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 6:
        Name: toJobConfig.updateKey
        Label: Update key columns
        Help: updateKey
        Type: LIST
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 7:
        Name: toJobConfig.updateMerge
        Label: Merge updates
        Help: updateMerge
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 8:
        Name: toJobConfig.dropTableIfExists
        Label: Drop table if exists
        Help: dropTableIfExists
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 9:
        Name: toJobConfig.storageClause
        Label: Template table storage clause
        Help: storageClause
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 2000
      Input 10:
        Name: toJobConfig.temporaryStorageClause
        Label: Temporary table storage clause
        Help: temporaryStorageClause
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 2000
      Input 11:
        Name: toJobConfig.appendValuesHint
        Label: Append values hint usage
        Help: appendValuesHint
        Type: ENUM
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Possible values: AUTO,ON,OFF
      Input 12:
        Name: toJobConfig.parallel
        Label: Parallel
        Help: parallel
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
Connector with Name: ftp-connector 
  Class: 1.99.7
  Version: TO
  Supported Directions {4}
    link config 1:
      Name: linkConfig
      Label: FTP Server configuration
      Help: Parameters required to connect to an FTP server.
      Input 1:
        Name: linkConfig.server
        Label: Hostname
        Help: Hostname for the FTP server.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 256
      Input 2:
        Name: linkConfig.port
        Label: Port
        Help: Port for the FTP server. Connector will use 21 if omitted.
        Type: INTEGER
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 3:
        Name: linkConfig.username
        Label: Username
        Help: Username that will be used to authenticate connection to the FTP Server.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 256
      Input 4:
        Name: linkConfig.password
        Label: Password
        Help: Password that will be used to authenticate connection to the FTP Server.
        Type: STRING
        Sensitive: true
        Editable By: ANY
        Overrides: 
        Size: 256
    TO Job config 1:
      Name: toJobConfig
      Label: Output configuration
      Help: Parameters required to store data on the FTP server.
      Input 1:
        Name: toJobConfig.outputDirectory
        Label: Output directory
        Help: Directory on the FTP server to write data to.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 260
Connector with Name: hdfs-connector 
  Class: 1.99.7
  Version: FROM/TO
  Supported Directions {4}
    link config 1:
      Name: linkConfig
      Label: HDFS cluster
      Help: Contains configuration required to connect to your HDFS cluster.
      Input 1:
        Name: linkConfig.uri
        Label: URI
        Help: Namenode URI for your cluster.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
      Input 2:
        Name: linkConfig.confDir
        Label: Conf directory
        Help: Directory on Sqoop server machine with hdfs configuration files (hdfs-site.xml, ...). This connector will load all files ending with -site.xml.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
      Input 3:
        Name: linkConfig.configOverrides
        Label: Additional configs:
        Help: Additional configuration that will be set on HDFS Configuration object, possibly overriding any keys loaded from configuration files.
        Type: MAP
        Sensitive: false
        Editable By: ANY
        Overrides: 
    FROM Job config 1:
      Name: fromJobConfig
      Label: Input configuration
      Help: Specifies information required to get data from HDFS.
      Input 1:
        Name: fromJobConfig.inputDirectory
        Label: Input directory
        Help: Input directory containing files that should be transferred.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
      Input 2:
        Name: fromJobConfig.overrideNullValue
        Label: Override null value
        Help: If set to true, then the null value will be overridden with the value set in Null value.
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 3:
        Name: fromJobConfig.nullValue
        Label: Null value
        Help: For file formats that doesn't have native representation of NULL (as for example text file) use this particular string to decode NULL value.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
    FROM Job config 2:
      Name: incremental
      Label: Incremental import
      Help: Information relevant for incremental reading from HDFS.
      Input 1:
        Name: incremental.incrementalType
        Label: Incremental type
        Help: Type of incremental import.
        Type: ENUM
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Possible values: NONE,NEW_FILES
      Input 2:
        Name: incremental.lastImportedDate
        Label: Last imported date
        Help: Datetime stamp of last read file. Next job execution will read only files that have been created after this point in time.
        Type: DATETIME
        Sensitive: false
        Editable By: ANY
        Overrides: 
    TO Job config 1:
      Name: toJobConfig
      Label: Target configuration
      Help: Configuration describing where and how the resulting data should be stored.
      Input 1:
        Name: toJobConfig.overrideNullValue
        Label: Override null value
        Help: If set to true, then the null value will be overridden with the value set in Null value.
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 2:
        Name: toJobConfig.nullValue
        Label: Null value
        Help: For file formats that doesn't have native representation of NULL (as for example text file) use this particular string to encode NULL value.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
      Input 3:
        Name: toJobConfig.outputFormat
        Label: File format
        Help: File format that should be used for transferred data.
        Type: ENUM
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Possible values: TEXT_FILE,SEQUENCE_FILE,PARQUET_FILE
      Input 4:
        Name: toJobConfig.compression
        Label: Compression codec
        Help: Compression codec that should be use to compress transferred data.
        Type: ENUM
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Possible values: NONE,DEFAULT,DEFLATE,GZIP,BZIP2,LZO,LZ4,SNAPPY,CUSTOM
      Input 5:
        Name: toJobConfig.customCompression
        Label: Custom codec
        Help: Fully qualified class name with Hadoop codec implementation that should be used if none of the build-in options are suitable.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
      Input 6:
        Name: toJobConfig.outputDirectory
        Label: Output directory
        Help: HDFS directory where transferred data will be written to.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
      Input 7:
        Name: toJobConfig.appendMode
        Label: Append mode
        Help: If set to false, job will fail if output directory already exists. If set to true then imported data will be stored to already existing and possibly non empty directory.
        Type: BOOLEAN
        Sensitive: false
        Editable By: ANY
        Overrides: 
Connector with Name: kafka-connector 
  Class: 1.99.7
  Version: TO
  Supported Directions {4}
    link config 1:
      Name: linkConfig
      Label: Kafka cluster
      Help: Configuration options describing Kafka cluster.
      Input 1:
        Name: linkConfig.brokerList
        Label: Kafka brokers
        Help: Comma-separated list of Kafka brokers in the form of host:port. It doesn't need to contain all brokers, but at least two are recommended for high availability
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 1024
      Input 2:
        Name: linkConfig.zookeeperConnect
        Label: Zookeeper quorum
        Help: Address of Zookeeper used by the Kafka cluster. Usually host:port. Multiple zookeeper nodes are supported. If Kafka is stored in its own znode use host:portkafka
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
    TO Job config 1:
      Name: toJobConfig
      Label: Output configuration
      Help: Configuration necessary when writing data to Kafka.
      Input 1:
        Name: toJobConfig.topic
        Label: Topic
        Help: Name of Kafka topic where data will be written into.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
Connector with Name: sftp-connector 
  Class: 1.99.7
  Version: TO
  Supported Directions {4}
    link config 1:
      Name: linkConfig
      Label: FTP Server configuration
      Help: Parameters required to connect to an SFTP server.
      Input 1:
        Name: linkConfig.server
        Label: Hostname
        Help: Hostname of the SFTP server.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 255
      Input 2:
        Name: linkConfig.port
        Label: Port
        Help: Port for the SFTP server. Connector will use 22 if omitted.
        Type: INTEGER
        Sensitive: false
        Editable By: ANY
        Overrides: 
      Input 3:
        Name: linkConfig.username
        Label: Username
        Help: Username that will be used to authenticate connection to SFTP serer.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 256
      Input 4:
        Name: linkConfig.password
        Label: Password
        Help: Password that will be used to authenticate connection to the FTP Server.
        Type: STRING
        Sensitive: true
        Editable By: ANY
        Overrides: 
        Size: 256
    TO Job config 1:
      Name: toJobConfig
      Label: Output configuration
      Help: Parameters required to store data on the SFTP server.
      Input 1:
        Name: toJobConfig.outputDirectory
        Label: Output directory
        Help: Directory on the SFTP server to write data to.
        Type: STRING
        Sensitive: false
        Editable By: ANY
        Overrides: 
        Size: 260
sqoop:000>			
			
			

19.3.4. link

19.3.4.1. hdfs-connector

				
sqoop:000> create link -connector hdfs-connector
0    [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Creating link for connector with name hdfs-connector
Please fill following values to create new link object
Name: hdfs

HDFS cluster

URI: hdfs://127.0.0.1:9000
Conf directory: 
Additional configs:: 
There are currently 0 values in the map:
entry# 
New link was successfully created with validation status OK and name hdfs
sqoop:000> 
				
				

				
sqoop:000> show link
+------+----------------+---------+
| Name | Connector Name | Enabled |
+------+----------------+---------+
| hdfs | hdfs-connector | true    |
+------+----------------+---------+			
				
				

19.3.4.2. generic-jdbc-connector

				
sqoop:000> create link -connector generic-jdbc-connector
Creating link for connector with name generic-jdbc-connector
Please fill following values to create new link object
Name: mysql

Database connection

Driver class: com.mysql.jdbc.Driver
Connection String: jdbc:mysql://127.0.0.1:3306/test
Username: test
Password: ****
Fetch Size: 
Connection Properties: 
There are currently 0 values in the map:
entry# 

SQL Dialect

Identifier enclose: 
New link was successfully created with validation status OK and name mysql		
				
				

				
sqoop:000> show link
+-------+------------------------+---------+
| Name  |     Connector Name     | Enabled |
+-------+------------------------+---------+
| mysql | generic-jdbc-connector | true    |
| hdfs  | hdfs-connector         | true    |
+-------+------------------------+---------+			
				
				

19.3.5. job

19.3.5.1. create job

				
sqoop:000> create job -f "mysql" -t "hdfs"
Creating job for links with from name mysql and to name hdfs
Please fill following values to create new job object
Name: from-mysql-to-hdfs

Database source

Schema name: test
Table name: member
SQL statement: 
Column names: 
There are currently 0 values in the list:
element# 
Partition column: 
Partition column nullable: 
Boundary query: 

Incremental read

Check column: 
Last value: 

Target configuration

Override null value: 
Null value: 
File format: 
  0 : TEXT_FILE
  1 : SEQUENCE_FILE
  2 : PARQUET_FILE
Choose: 0
Compression codec: 
  0 : NONE
  1 : DEFAULT
  2 : DEFLATE
  3 : GZIP
  4 : BZIP2
  5 : LZO
  6 : LZ4
  7 : SNAPPY
  8 : CUSTOM
Choose: 0
Custom codec: 
Output directory: /sqoop/member
Append mode: 

Throttling resources

Extractors: 
Loaders: 

Classpath configuration

Extra mapper jars: 
There are currently 0 values in the list:
element# 
New job was successfully created with validation status OK  and name from-mysql-to-hdfs
				
				

19.3.5.2. show job

				
sqoop:000> show job
+----+--------------------+--------------------------------+-----------------------+---------+
| Id |        Name        |         From Connector         |     To Connector      | Enabled |
+----+--------------------+--------------------------------+-----------------------+---------+
| 1  | from-mysql-to-hdfs | mysql (generic-jdbc-connector) | hdfs (hdfs-connector) | true    |
+----+--------------------+--------------------------------+-----------------------+---------+				
				
				

19.3.5.3. start job

				
sqoop:000> start job -n from-mysql-to-hdfs		

sqoop:000> start job -n from-mysql-to-hdfs
Submission details
Job Name: from-mysql-to-hdfs
Server URL: http://localhost:12000/sqoop/
Created by: hadoop
Creation date: 2017-07-22 23:18:02 CST
Lastly updated by: hadoop
External ID: job_1499236611045_0001
	http://iZj6ciilv2rcpgauqg2uuwZ:8088/proxy/application_1499236611045_0001/
2017-07-22 23:18:02 CST: BOOTING  - Progress is not available
				
				

启动后进入HDFS查看导入情况

				
[hadoop@netkiller ~]$ hdfs dfs -ls /sqoop	

[hadoop@netkiller ~]$ hdfs dfs -ls /member
Found 10 items
-rw-r--r--   3 hadoop supergroup          0 2017-07-22 23:18 /member/310af608-5533-4bc2-bfb8-eaa45470b04d.txt
-rw-r--r--   3 hadoop supergroup         48 2017-07-22 23:18 /member/36bc39a5-bc73-4065-a361-ff2d61c4922c.txt
-rw-r--r--   3 hadoop supergroup          0 2017-07-22 23:18 /member/3e855400-84a9-422d-b50c-1baa9666a719.txt
-rw-r--r--   3 hadoop supergroup        140 2017-07-22 23:18 /member/3e8dad92-e0f1-4a74-a337-642cf4e6d634.txt
-rw-r--r--   3 hadoop supergroup         55 2017-07-22 23:18 /member/4a9f47f1-0413-4149-a93a-ed8b51efbc87.txt
-rw-r--r--   3 hadoop supergroup          0 2017-07-22 23:18 /member/4dc5bfe7-1cd9-4d9b-96a8-07e82ed79a71.txt
-rw-r--r--   3 hadoop supergroup          0 2017-07-22 23:18 /member/60dbcc60-61f2-4433-af39-1dfdfc048940.txt
-rw-r--r--   3 hadoop supergroup          0 2017-07-22 23:18 /member/6d02ed89-94d9-4d4b-87ed-d5da9d2bf9fe.txt
-rw-r--r--   3 hadoop supergroup        209 2017-07-22 23:18 /member/cf7b7185-3ab6-4077-943a-26228b769c57.txt
-rw-r--r--   3 hadoop supergroup          0 2017-07-22 23:18 /member/f2e0780d-ad33-4b35-a1c7-b3fbc23e303d.txt	
				
				

19.3.5.4. status job

				
sqoop:000> status job -n from-mysql-to-hdfs		
				
				

19.3.6. update

19.3.6.1. link

				
sqoop:000> update link -n  mysql
Updating link with name mysql
Please update link:
Name: mysql

Database connection

Driver class: com.mysql.jdbc.Driver
Connection String: jdbc:mysql://127.0.0.1:3306/test
Username: test
Password: ****
Fetch Size: 
Connection Properties: 
There are currently 0 values in the map:
entry# 

SQL Dialect

Identifier enclose:  
link was successfully updated with status OK