flinkcdc使用记录

当前yaml：pipeline连接方式

source:
  type: mysql
  hostname: localhost
  port: 3306
  username: root
  password: 123456
  tables: app_db.\.*
  server-id: 5400-5404
  server-time-zone: UTC

sink:
  type: starrocks
  name: StarRocks Sink
  jdbc-url: jdbc:mysql://127.0.0.1:9030
  load-url: 127.0.0.1:8080
  username: root
  password: ""
  table.create.properties.replication_num: 1

pipeline:
  name: Sync MySQL Database to StarRocks
  parallelism: 2

1.修改mysql字段名，starrocks中不会自动修改

alter table test.employees change salary sala decimal(10,2)

1.mysql修改表字段名后，flink任务挂掉。后续对mysql操作都不会同步到sr

2.flink任务挂掉后，重启任务前删除mysql表中一行数据，重启任务后 sr并不会删除历史数据。（

• initial （默认）：在第一次启动时对受监视的数据库表执行初始快照，并继续读取最新的 binlog。

）注意：此时，修改参数为  scan.startup.mode: earliest-offset  跳过快照从最早的belong读取，flink任务会挂掉。

transform:
  - source-table: test.ws
    projection: id,ts,vc,localtimestamp as new_timestamp  #增加新的一列
    description: append calculated columns based on source table

- source-table前两个缩进，projection四个缩进

完整的

 source:
     type: mysql
     hostname: localhost
     port: 3306
     username: root
     password: 123456
     tables: app_db.\.*

   sink:
     type: doris
     fenodes: 127.0.0.1:8030
     username: root
     password: ""

   transform:
     - source-table: adb.web_order01
       projection: \*, format('%S', product_name) as product_name
       filter: addone(id) > 10 AND order_id > 100
       description: project fields and filter
     - source-table: adb.web_order02
       projection: \*, format('%S', product_name) as product_name
       filter: addone(id) > 20 AND order_id > 200
       description: project fields and filter

   route:
     - source-table: app_db.orders
       sink-table: ods_db.ods_orders
     - source-table: app_db.shipments
       sink-table: ods_db.ods_shipments
     - source-table: app_db.products
       sink-table: ods_db.ods_products

   pipeline:
     name: Sync MySQL Database to Doris
     parallelism: 2
     user-defined-function:
       - name: addone
         classpath: com.example.functions.AddOneFunctionClass
       - name: format
         classpath: com.example.functions.FormatFunctionClass