Hive迁移老数据-动态分区

现在有个hive表 dw_revisit_user_d ,创建的时候没有加partitioned by,现在想拓展表中的datestr当分区列。要怎么保证原来数据不丢,并且让原来的数据按datestr分区,以后的数据也按datestr分区?

9DA3BBF8-75B6-4788-8EE9-015C14D84DDD.png

我们可以使用select..insert + 动态分区解决问题

步骤

1
2
3
4
5
6
7
8
9
10
11
12
13
set hive.exec.dynamic.partition.mode=nonstrict;

create table t_1(datestr string,u_id string,acc_cnt bigint);
insert into table t_1 values('2019-07-17','1',22);
insert into table t_1 values('2019-07-17','2',24);
insert into table t_1 values('2019-07-16','3',255);

insert overwrite table t_2 partition (datestr)
select u_id,acc_cnt,datestr from t_1 where datestr = '2019-07-17';

insert overwrite table t_2 partition (datestr)
select u_id,acc_cnt,datestr from t_1 where datestr = '2019-07-16';

加上下面这句话,不带where直接自动匹配datestr

1
2
3
set hive.exec.dynamici.partition=true;
insert overwrite table t_2 partition (datestr)
select u_id,acc_cnt,datestr from t_1;

之后删掉t_1 修改t_2为t_1就行了


Hive迁移老数据-动态分区
http://example.com/2019/07/18/2019–07-17-迁移老数据动态分区/
Author
Hoey
Posted on
July 18, 2019
Licensed under