Sqoop之增量导入数据之Shell脚本之手把手教学--------------对着写领导再也不用担心你加班了

不想加班到凌晨，又想一键导入昨晚的数据，那咋办呢？ 定时启动Shell脚本。 废话少说，直接上：

首先先在mysql创建一个表用来测试：

mysql> create table myorder(ordid int primary key not null auto_increment,orderno varchar(20),orderdate date); Query OK, 0 rows affected (0.02 sec) mysql> insert into myorder(orderno,orderdate) values('dd0001','2020-6-25'),('dd0002','2020-6-25'),('dd000)3','2020-6-26'),('dd0004','2020-6-26'); Query OK, 4 rows affected (0.00 sec) Records: 4 Duplicates: 0 Warnings: 0 mysql> select * from myorder; +-------+---------+------------+ | ordid | orderno | orderdate | +-------+---------+------------+ | 1 | dd0001 | 2020-06-25 | | 2 | dd0002 | 2020-06-25 | | 3 | dd0003 | 2020-06-26 | | 4 | dd0004 | 2020-06-26 | +-------+---------+------------+ 4 rows in set (0.00 sec)

写这玩意遇到的小难点： 1. shell去除字符串中所有空格：

[root@zjw myshl]# echo '1999-9-9' | sed 's/-//g' 199999

2.Shell怎么获得当前时间 带格式：

nowdate=$(date --date='1 day ago' '+%Y-%m-%d')

3.hdfs怎么取出文件名后几个

[root@zjw myshl]# hdfs dfs -ls /mydata | awk '{print $8}' | cut -d '/' -f3 | grep -v "[0-9]\{4\})" 20/06/26 14:46:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 0610 test

大概就这几个要点，脚本正文：

#! /bin/bash # get current date nowdate=$(date --date='1 day ago' '+%Y-%m-%d') # if exists mydata folder, delete folder where folder'name not confirm 8 number hdfs dfs -test -e /mydata/ if [ $? -eq 0 ];then files=`hdfs dfs -ls /mydata | awk '{print $8}' | cut -d '/' -f3 | grep -v "[0-9]\{4\})"` for fl in $files do hdfs dfs -rmr /mydata/$fl done else hdfs dfs -mkdir -p /mydata/ fi foldername=`echo $nowdate|sed 's/-//g'` #import mysql data to hdfs sqoop import --connect jdbc:mysql://192.168.56.100:3306/mydemo --driver com.mysql.jdbc.Driver --username root --password 1234 --query "select * from myorder where orderdate="\'$nowdate\'" and \$CONDITIONS" --target-dir /mydata/$foldername -m 1

授权之后简单运行一下：

直接芜湖。