多表关联只有主表才能触发分区更新,被关联表变动都是全量,如何解决

Viewed 68

sql如下,如果客户表变化才能按分区刷,如果是地址表变化,都是全量,有没有办法解决呢。就想刷新的时候占用少点资源和时间

-- 物化视图
CREATE MATERIALIZED VIEW dwd.dwd_cus_customer_address
BUILD IMMEDIATE
       REFRESH AUTO ON COMMIT
    PARTITION BY ( DATE_TRUNC(cus_create_time,'month'))
DISTRIBUTED BY hash(cus_id)  BUCKETS 16
PROPERTIES ('replication_num' = '1',
    'storage_medium' = 'ssd'
    )
AS

SELECT
    cus.id AS cus_id,
    cus.create_time AS cus_create_time,
    cus.name AS cus_name,
    cus.gender_type AS cus_gender_type,
    cus.phone AS cus_phone,
    cus.birthday AS cus_birthday,
    cus.audit_status AS cus_audit_status,
    cus.source_type AS cus_source_type,
    cus.wx_open_id AS cus_wx_open_id,
    cus.wx_union_id AS cus_wx_union_id,
    cus.wx_head_portrait AS cus_wx_head_portrait,
    cus.wx_nickname AS cus_wx_nickname,
    cus.sys_dept_id AS cus_sys_dept_id,
    cus.revision AS cus_revision,
    cus.create_by AS cus_create_by,
    cus.create_by_name AS cus_create_by_name,
    cus.create_by_type AS cus_create_by_type,
    cus.update_by AS cus_update_by,
    cus.update_by_name AS cus_update_by_name,
    cus.update_by_type AS cus_update_by_type,
    cus.update_time AS cus_update_time,
    cus.delete_by AS cus_delete_by,
    cus.delete_by_name AS cus_delete_by_name,
    cus.delete_by_type AS cus_delete_by_type,
    cus.delete_time AS cus_delete_time,
    cus.delete_status AS cus_delete_status,
    cus.sort_num AS cus_sort_num,
    cus.remark AS cus_remark,
    cus.spare_id AS cus_spare_id,
    cus.provice AS cus_provice,
    cus.city AS cus_city,
    cus.area AS cus_area,
    cus.detail_distribution_address AS cus_detail_distribution_address,
    cus.customer_stock AS cus_customer_stock,
    cus.customer_stock_zero_time AS cus_customer_stock_zero_time,
    cus.fresh_milk_order_total AS cus_fresh_milk_order_total,
    cus.fresh_milk_give_total AS cus_fresh_milk_give_total,
    cus.fresh_milk_sent_total AS cus_fresh_milk_sent_total,
    cus.exchange_use_fresh_milk_total AS cus_exchange_use_fresh_milk_total,
    cus.distribution_fresh_milk_total AS cus_distribution_fresh_milk_total,
    cus.temporary_add_milk_total AS cus_temporary_add_milk_total,
    cus.long_milk_give_total AS cus_long_milk_give_total,
    cus.long_milk_sent_total AS cus_long_milk_sent_total,
    cus.expiration_time AS cus_expiration_time,
    cus.should_send AS cus_should_send,
    cus.fresh_milk_road_way_sent_total AS cus_fresh_milk_road_way_sent_total,
    cus.labeled AS cus_labeled,
    cus.origin_stock AS cus_origin_stock,
    cus.is_import AS cus_is_import,
    cus.lock_stock AS cus_lock_stock,
    cus.qw_external_user_id AS cus_qw_external_user_id,
    cus.qw_remark AS cus_qw_remark,
    cusaddr.id AS cusaddr_id,
    cusaddr.create_time AS cusaddr_create_time,
    cusaddr.cxr_customer_id AS cusaddr_cxr_customer_id,
    cusaddr.sys_dept_id AS cusaddr_sys_dept_id,
    cusaddr.cxr_residential_quarters_id AS cusaddr_cxr_residential_quarters_id,
    cusaddr.receiver_name AS cusaddr_receiver_name,
    cusaddr.receiver_phone AS cusaddr_receiver_phone,
    cusaddr.detail_distribution_address AS cusaddr_detail_distribution_address,
    cusaddr.sys_area_id AS cusaddr_sys_area_id,
    cusaddr.cxr_site_id AS cusaddr_cxr_site_id,
    cusaddr.cxr_employee_id AS cusaddr_cxr_employee_id,
    cusaddr.is_show_am_distribution AS cusaddr_is_show_am_distribution,
    cusaddr.am_distribution_status AS cusaddr_am_distribution_status,
    cusaddr.am_distribution_start_delivery_time AS cusaddr_am_distribution_start_delivery_time,
    cusaddr.am_distribution_suspend_start_time AS cusaddr_am_distribution_suspend_start_time,
    cusaddr.am_distribution_suspend_end_time AS cusaddr_am_distribution_suspend_end_time,
    cusaddr.am_distribution_info AS cusaddr_am_distribution_info,
    cusaddr.is_show_pm_distribution AS cusaddr_is_show_pm_distribution,
    cusaddr.pm_distribution_status AS cusaddr_pm_distribution_status,
    cusaddr.pm_distribution_start_delivery_time AS cusaddr_pm_distribution_start_delivery_time,
    cusaddr.pm_distribution_suspend_start_time AS cusaddr_pm_distribution_suspend_start_time,
    cusaddr.pm_distribution_suspend_end_time AS cusaddr_pm_distribution_suspend_end_time,
    cusaddr.pm_distribution_info AS cusaddr_pm_distribution_info,
    cusaddr.revision AS cusaddr_revision,
    cusaddr.create_by AS cusaddr_create_by,
    cusaddr.create_by_name AS cusaddr_create_by_name,
    cusaddr.create_by_type AS cusaddr_create_by_type,
    cusaddr.update_by AS cusaddr_update_by,
    cusaddr.update_by_name AS cusaddr_update_by_name,
    cusaddr.update_by_type AS cusaddr_update_by_type,
    cusaddr.update_time AS cusaddr_update_time,
    cusaddr.delete_by AS cusaddr_delete_by,
    cusaddr.delete_by_name AS cusaddr_delete_by_name,
    cusaddr.delete_by_type AS cusaddr_delete_by_type,
    cusaddr.delete_time AS cusaddr_delete_time,
    cusaddr.delete_status AS cusaddr_delete_status,
    cusaddr.sort_num AS cusaddr_sort_num,
    cusaddr.remark AS cusaddr_remark,
    cusaddr.spare_id AS cusaddr_spare_id,
    cusaddr.provice AS cusaddr_provice,
    cusaddr.city AS cusaddr_city,
    cusaddr.area AS cusaddr_area,
    cusaddr.defalut_account_address AS cusaddr_defalut_account_address,
    cusaddr.am_friday AS cusaddr_am_friday,
    cusaddr.am_monday AS cusaddr_am_monday,
    cusaddr.am_tuesday AS cusaddr_am_tuesday,
    cusaddr.am_saturday AS cusaddr_am_saturday,
    cusaddr.am_thursday AS cusaddr_am_thursday,
    cusaddr.am_wednesday AS cusaddr_am_wednesday,
    cusaddr.pm_friday AS cusaddr_pm_friday,
    cusaddr.pm_monday AS cusaddr_pm_monday,
    cusaddr.pm_tuesday AS cusaddr_pm_tuesday,
    cusaddr.pm_saturday AS cusaddr_pm_saturday,
    cusaddr.pm_thursday AS cusaddr_pm_thursday,
    cusaddr.pm_wednesday AS cusaddr_pm_wednesday,
    cusaddr.address_serial_number AS cusaddr_address_serial_number,
    cusaddr.change_status AS cusaddr_change_status,
    cusaddr.distribution_district_name AS cusaddr_distribution_district_name,
    cusaddr.longitude AS cusaddr_longitude,
    cusaddr.latitude AS cusaddr_latitude,
    cusaddr.level AS cusaddr_level,
    cusaddr.first_road_way_id AS cusaddr_first_road_way_id,
    cusaddr.clean_time AS cusaddr_clean_time,
    cusaddr.is_cleaned AS cusaddr_is_cleaned,
    cusaddr.clean_msg AS cusaddr_clean_msg
FROM ry_cloud.ods_cxr_customer cus
         LEFT JOIN ry_cloud.ods_cxr_customer_address cusaddr ON cus.id = cusaddr.cxr_customer_id;
2 Answers

客户表数据刷新,可能只是单个partition的数据,你这里的地址表发生变化,可能涉及到的不止一个partition吧。

ods_cxr_customer_address这个表的数据一般是新增还是历史数据修改,如果是新增的话可以创建物化视图的时候指定 excluded_trigger_tables 属性,指定了这个属性后,此属性里面表数据变化不再触发刷新。可能会导致一定的数据延迟,刷新后,数据就是正确的了。

https://doris.apache.org/zh-CN/docs/dev/sql-manual/sql-statements/table-and-view/async-materialized-view/CREATE-ASYNC-MATERIALIZED-VIEW

ods_cxr_customer_address这个表的数据是历史数据修改,因为不确定会影响物化视图的那些分区,所以会全量刷新。