sqoop从oracle导入hdfs,Sqoop 1.99.3基础操作--导入Oracle的数据到HDFS

逃离我推掉我的手 2023-01-18 05:26 254阅读 0赞

第一步：进入客户端Shell fulong@FBI008:~$ sqoop.sh client Sqoop home directory: /home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200 Sqoop Shell: Type help or h for help. sqoop:000 set server --host FBI003 --port 12000 --webapp sqoop Server is set

第一步：进入客户端Shell

fulong@FBI008:~$ sqoop.sh client

Sqoop home directory: /home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200

Sqoop Shell: Type 'help' or 'h' for help.

sqoop:000> set server --host FBI003 --port 12000 --webapp sqoop

Server is set successfully

sqoop:000> show version --all

client version:

Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b

Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013

server version:

Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b

Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013

Protocol version:

\[1\]

第二步：创建连接

sqoop:000> create connection --cid 001

Creating connection for connector with id 1

Please fill following values to create new connection object

Name: 147patents

Connection configuration

JDBC Driver Class: oracle.jdbc.driver.OracleDriver

JDBC Connection String: jdbc:oracle:thin:@192.168.0.147:1521:ORCLGBK

Username: zhaobiao

Password: \*\*\*\*\*\*

JDBC Connection Properties:

There are currently 0 values in the map:

entry\#

Security related configuration options

Max connections: 100

New connection was successfully created with validation status FINE and persistent id 1

第三步：创建Job

我这里顺便尝试了一下update命令，所以第一遍创建job时输入了错误的tablename：

sqoop:000> create job

Required argument --xid is missing.

sqoop:000> create job --xid 1 --type import

Creating job for connection with id 1

Please fill following values to create new job object

Name: ImportPatents

Database configuration

Schema name: zhaobiao

Table name: zhaobiao

Table SQL statement:

Table column names:

Partition column name:

Nulls in partition column:

Boundary query:

Output configuration

Storage type:

0 : HDFS

Choose: 0

Output format:

0 : TEXT\_FILE

1 : SEQUENCE\_FILE

Choose: 0

Compression format:

0 : NONE

1 : DEFAULT

2 : DEFLATE

3 : GZIP

4 : BZIP2

5 : LZO

6 : LZ4

7 : SNAPPY

Choose: 0

Output directory: /Data/zhaobiao

Throttling resources

Extractors:

Loaders:

New job was successfully created with validation status FINE  and persistent id 1

sqoop:000> update job

Required argument --jid is missing.

sqoop:000> update job --jid 1

Updating job with id 1

Please update job metadata:

Name: ImportPatents

Database configuration

Schema name: zhaobiao

Table name: CMS\_NEWS\_0625

Table SQL statement:

Table column names:

Partition column name:

Nulls in partition column:

Boundary query:

Output configuration

Storage type:

0 : HDFS

Choose: 0

Output format:

0 : TEXT\_FILE

1 : SEQUENCE\_FILE

Choose: 0

Compression format:

0 : NONE

1 : DEFAULT

2 : DEFLATE

3 : GZIP

4 : BZIP2

5 : LZO

6 : LZ4

7 : SNAPPY

Choose: 0

Output directory: /Data/zhaobiao

Throttling resources

Extractors:

Loaders:

Job was successfully updated with status FINE

总结：

1.create job时必须指定一个连接id

2.create job完成后，自动分配了一个job id

3.update job时，，需要指定job id

4.创建Job过程中指定的Output directory指定的是HDFS中的路径

第四步：执行job

sqoop:000> start job --jid 1

2014-06-27 12:44:17 CST: FAILURE\_ON\_SUBMIT

Exception: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://fulonghadoop/Data/zhaobiao already exists

sqoop:000> start job --jid 1

Submission details

Job ID: 1

Server URL: :12000/sqoop/

Created by: fulong

Creation date: 2014-06-27 12:44:53 CST

Lastly updated by: fulong

External ID: job\_1403248336137\_0005

:8088/proxy/application\_1403248336137\_0005/

2014-06-27 12:44:53 CST: BOOTING  - Progress is not available

sqoop:000> status job --jid 1

Submission details

Job ID: 1

Server URL: :12000/sqoop/

Created by: fulong

Creation date: 2014-06-27 12:44:53 CST

Lastly updated by: fulong

External ID: job\_1403248336137\_0005

:8088/proxy/application\_1403248336137\_0005/

2014-06-27 12:50:11 CST: RUNNING  - 0.00 %

sqoop:000> status job --jid 1

Submission details

Job ID: 1

Server URL: :12000/sqoop/

Created by: fulong

Creation date: 2014-06-27 12:44:53 CST

Lastly updated by: fulong

External ID: job\_1403248336137\_0005

:8088/proxy/application\_1403248336137\_0005/

2014-06-27 13:36:49 CST: RUNNING  - 25.00 %

sqoop:000> status job --jid 1

Submission details

Job ID: 1

Server URL: :12000/sqoop/

Created by: fulong

Creation date: 2014-06-27 12:44:53 CST

Lastly updated by: fulong

External ID: job\_1403248336137\_0005

:8088/proxy/application\_1403248336137\_0005/

2014-06-27 13:39:24 CST: RUNNING  - 35.00 %

sqoop:000> status job --jid 1

Submission details

Job ID: 1

Server URL: :12000/sqoop/

Created by: fulong

Creation date: 2014-06-27 12:44:53 CST

Lastly updated by: fulong

External ID: job\_1403248336137\_0005

:8088/proxy/application\_1403248336137\_0005/

2014-06-27 13:47:19 CST: RUNNING  - 40.00 %

sqoop:000> status job --jid 1

Submission details

Job ID: 1

Server URL: :12000/sqoop/

Created by: fulong

Creation date: 2014-06-27 12:44:53 CST

Lastly updated by: fulong

External ID: job\_1403248336137\_0005

:8088/proxy/application\_1403248336137\_0005/

2014-06-27 14:08:36 CST: SUCCEEDED

Counters:

org.apache.hadoop.mapreduce.JobCounter

SLOTS\_MILLIS\_MAPS: 16499226

TOTAL\_LAUNCHED\_MAPS: 10

SLOTS\_MILLIS\_REDUCES: 0

OTHER\_LOCAL\_MAPS: 15

org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter

BYTES\_WRITTEN: 421226094

org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter

BYTES\_READ: 0

org.apache.hadoop.mapreduce.TaskCounter

MAP\_INPUT\_RECORDS: 0

MERGED\_MAP\_OUTPUTS: 0

PHYSICAL\_MEMORY\_BYTES: 1185689600

SPILLED\_RECORDS: 0

COMMITTED\_HEAP\_BYTES: 627179520

CPU\_MILLISECONDS: 307570

FAILED\_SHUFFLE: 0

VIRTUAL\_MEMORY\_BYTES: 6542311424

SPLIT\_RAW\_BYTES: 1437

MAP\_OUTPUT\_RECORDS: 644083

GC\_TIME\_MILLIS: 5279

org.apache.hadoop.mapreduce.FileSystemCounter

FILE\_WRITE\_OPS: 0

FILE\_READ\_OPS: 0

FILE\_LARGE\_READ\_OPS: 0

FILE\_BYTES\_READ: 0

HDFS\_BYTES\_READ: 1437

FILE\_BYTES\_WRITTEN: 931780

HDFS\_LARGE\_READ\_OPS: 0

HDFS\_WRITE\_OPS: 20

HDFS\_READ\_OPS: 40

HDFS\_BYTES\_WRITTEN: 431907904

org.apache.sqoop.submission.counter.SqoopCounters

ROWS\_READ: 644083

Job executed successfully

接下来的问题：如何导入数据到Hbase中？

sqoop从oracle导入hdfs,Sqoop 1.99.3基础操作--导入Oracle的数据到HDFS

发表评论取消回复

还没有评论，来说两句吧...

相关阅读

相关 Hadoop sqoop从MySQL导入数据到HDFS

相关 Sqoop 从MySQL 导入导出HDFS

相关 sqoop从oracle导入hdfs,Sqoop 1.99.3基础操作--导入Oracle的数据到HDFS

相关 sqoop增量导入到_sqoop的使用之import导入到HDFS

相关 Sqoop2 从MySQL导入数据到Hadoop HDFS

相关用sqoop导入数据到HIVE和HDFS中

相关 Sqoop从mysql导入数据至HDFS操作（sqoop初级）

相关 Sqoop将hdfs上的数据导入到MySQL

相关 sqoop 从mysql导入数据到 hadoop出错

相关从Oracle导入数据到Hive中