DolphinScheduler配置Seatunnel同步任务出错(从MySQL同步数据到Doris)

DolphinSScheduler版本3.2.0, Seatunnel版本2.3.4

错误日志

定义好工作流后, 上线运行报错, 关键日志入下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[INFO] 2025-02-16 13:02:00.427 -0800 - tenantCode :root, task dir:/tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3
[INFO] 2025-02-16 13:02:00.429 -0800 - generate script file:/tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3/seatunnel_2_3.conf
[INFO] 2025-02-16 13:02:00.433 -0800 - SeaTunnel task command: ${SEATUNNEL_HOME}/bin/seatunnel.sh --config /tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3/seatunnel_2_3.conf --deploy-mode local
[INFO] 2025-02-16 13:02:00.436 -0800 - Final Shell file is : 
#!/bin/bash
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
${SEATUNNEL_HOME}/bin/seatunnel.sh --config /tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3/seatunnel_2_3.conf --deploy-mode local
[INFO] 2025-02-16 13:02:00.439 -0800 - Executing shell command : sudo -u root -i /tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3/2_3.sh
[INFO] 2025-02-16 13:02:00.483 -0800 - process start, process id is: 37662
[INFO] 2025-02-16 13:02:02.484 -0800 -  -> 
	/tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3/2_3.sh: line 4: /bin/seatunnel.sh: No such file or directory
[INFO] 2025-02-16 13:02:02.491 -0800 - process has exited. execute path:/tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3, processId:37662 ,exitStatusCode:127 ,processWaitForStatus:true ,processExitValue:127
[INFO] 2025-02-16 13:02:02.493 -0800 - ***********************************************************************************************
[INFO] 2025-02-16 13:02:02.505 -0800 - *********************************  Finalize task instance  ************************************
[INFO] 2025-02-16 13:02:02.506 -0800 - ***********************************************************************************************
[INFO] 2025-02-16 13:02:02.509 -0800 - Upload output files: [] successfully
[INFO] 2025-02-16 13:02:02.512 -0800 - Send task execute status: FAILURE to master : 192.168.10.132:1234
[INFO] 2025-02-16 13:02:02.514 -0800 - Remove the current task execute context from worker cache
[INFO] 2025-02-16 13:02:02.515 -0800 - The current execute mode isn't develop mode, will clear the task execute file: /tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3
[INFO] 2025-02-16 13:02:02.519 -0800 - Success clear the task execute file: /tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3
[INFO] 2025-02-16 13:02:02.520 -0800 - FINALIZE_SESSION

解决尝试

先检查DolphinScheduler的Seatunnel环境变量是否配置正确

1
echo $SEATUNNEL_HOME

发现输出为空, 但我已经在dolphinscheduler_env.sh配置了export SEATUNNEL_HOME=${SEATUNNEL_HOME:-/opt/software/seatunnel-2.3.4}, 于是决定重启DolphinScheduler, 重启后再次尝试, 环境依旧未生效, 进入dolphinscheduler_env.sh所在目录, 手动加载文件

1
source dolphinscheduler_env.sh

手动加载后运行echo $SEATUNNEL_HOME, 成功显示/opt/software/seatunnel-2.3.4, 再次重启DolphinScheduler, 运行依旧报错

意识到应该是我修改过的dolphinscheduler_env.sh文件没有生效, 应该重新执行install.sh脚本, 重新安装DolphinScheduler, 由于3.2.0是新的第一个大版本, 于是决定回退到3.1.9

重装Dolphinscheduler

停止ds服务

1
./stop-all.sh

删除dolphinscheduler目录

1
rm -rf /opt/software/dolphinscheduler-3.2.0

删除ds元数据库

1
DROP DATABASE dolphinscheduler;

配置dolphinscheduler_env.sh, 根据自己的安装目录和服务器ip和账户密码配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
cd /opt/software/dolphinscheduler-3.1.9/bin/env
vim dolphinscheduler_env.sh

export JAVA_HOME=${JAVA_HOME:-/opt/software/openjdk8}

# Database related configuration, set database type, username and password
export DATABASE=${DATABASE:-mysql}
export SPRING_PROFILES_ACTIVE=${DATABASE}
export SPRING_DATASOURCE_URL=jdbc:mysql://ip:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false
export SPRING_DATASOURCE_USERNAME=root
export SPRING_DATASOURCE_PASSWORD=

# DolphinScheduler server related configuration
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}

# Registry center configuration, determines the type and link of the registry center
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-ip:2181}

# Tasks related configurations, need to change the configuration if you use the related tasks.
export SEATUNNEL_HOME=${SEATUNNEL_HOME:-/opt/software/seatunnel-2.3.4}

export PATH=$JAVA_HOME/bin:$SEATUNNEL_HOME/bin:$PATH

配置install_env.sh, 根据自己集群资源配置

1
2
3
4
5
6
7
8
9
10
11
12
cd /opt/software/dolphinscheduler-3.1.9/bin/env
vim install_env.sh

ips=${ips:-""} 
sshPort=${sshPort:-"22"} 
masters=${masters:-""} 
workers=${workers:-""} 
alertServer=${alertServer:-""} 
apiServers=${apiServers:-""} 
installPath=${installPath:-"/opt/software/dolphinscheduler-3.1.9/ds"} 
deployUser=${deployUser:-"root"} 
zkRoot=${zkRoot:-"/dolphinscheduler"}

连接mysql, 创建元数据库

1
2
3
4
5
6
CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE 
utf8_general_ci; 
 
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'root'@'%'; 
 
FLUSH PRIVILEGES;

执行 upgrade-schema.sh

1
2
3
4
5
cd /opt/software/dolphinscheduler-3.1.9 
 
chmod 777 tools/bin/upgrade-schema.sh 
 
bash tools/bin/upgrade-schema.sh

创建dolphinscheduler 安装目录, 这是由于我在install_env.sh中的配置, 要根据自己的情况来

1
2
cd /opt/software/dolphinscheduler-3.1.9/
mkdir ds

执行install.sh

1
2
cd /opt/software/dolphinscheduler-3.1.9/bin/
./install.sh

使用默认账密admin/dolphinscheduler123登陆
avatar

检查配置是否生效

1
echo $SEATUNNEL_HOME

重新进行同步任务, 成功.