CentOs7.x安装部署SeaTunnelWeb遇到的坑
文章目录
- 1. 环境
- 2. SeaTunnel安装部署
- 2.1下载安装包
- 2.2 设置环境变量
- 2.3 安装连接器插件
- 2.4 拷贝jar包到lib下
- 2.5 启动命令
- 2.6 执行官方client提交任务demo
- 3. SeaTunnel-Web安装部署
- 3.1 下载安装包
- 3.2 初始化数据库脚本或修改配置application.yml中的数据库连接信息
- 3.2.1 不使用它的初始化脚本自动创建
- 3.2.2 使用脚本初始化数据库
- 3.3 修改端口与数据源
- 3.4 两个文件拷贝配置
- 3.5 拷贝jar到lib下
- 3.6 启动
- 3.7 访问首页
- 3.8 执行myslq-jdbc–>mysl-jdbc的单表数据同步
- 4. 总结
1. 环境
采用的是虚拟机上安装CentOs7.x，安装java15环境，安装mysql8.0.28，这两步骤省略，之前的文章里面有讲，属于基础环境配置，也比较简单，说明我用的是一台CentOs7.x的虚拟机实例，都安装在一台上面，需要开放8081、3306和5801的防火墙，不然会网络访问不到。
2. SeaTunnel安装部署
2.1下载安装包
export version="2.3.3"wget "https://archive.apache.org/dist/seatunnel/${version}/apache-seatunnel-${version}-bin.tar.gz"tar -xzvf "apache-seatunnel-${version}-bin.tar.gz"2.2 设置环境变量
vi /etc/profile.d/seatunnel.sh# 新增如下变量export SEATUNNEL_HOME=/root/apache-seatunnel-2.3.3 #这里设置的是seatunnel的解压目录即可export PATH=$PATH:$SEATUNNEL_HOME/bin然后执行source /etc/profile
2.3 安装连接器插件
进入/root/apache-seatunnel-2.3.3路径下执行安装连接器插件
sh bin/install-plugin.sh 2.3.3可以修改/root/apache-seatunnel-2.3.3/connectors下的plugin-mapping.properties文件，根据需要加入连接器插件，把不需要的连接器插件名称删除留下需要的即可，然后执行上面的安装命令，由于我没有修改这个文件，默认安装的是全部的连接器插件，所以需要一点时间，下载速度取决于网速。
2.4 拷贝jar包到lib下
2.5 启动命令
进入/root/apache-seatunnel-2.3.3路径下执行启动命令：
sh bin/seatunnel-cluster.sh -d-DJvmOption="-Xms1G -Xmx1G"或者nohup sh bin/seatunnel-cluster.sh 2>&1 &jps查看seatunnel的进程如下：
查看logs下的日志均正常输出没有报错，说明启动成功，然后执行下面的官方demo没有报错，说明seatunnel启动是正常的。
2.6 执行官方client提交任务demo
进入/root/apache-seatunnel-2.3.3路径下执行启动命令：
$SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template该命令来自于官网，执行结果如下：
[root@es1 apache-seatunnel-2.3.3]# $SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.templateWARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator信息: Loading configuration '/root/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator信息: Using configuration file at /root/apache-seatunnel-2.3.3/config/seatunnel.yaml12月 19, 2023 12:11:20 上午 org.apache.seatunnel.engine.common.config.SeaTunnelConfig信息: seatunnel.home is /root/apache-seatunnel-2.3.312月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator信息: Loading configuration '/root/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator信息: Using configuration file at /root/apache-seatunnel-2.3.3/config/hazelcast.yaml12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator信息: Loading configuration '/root/apache-seatunnel-2.3.3/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'12月 19, 2023 12:11:20 上午 com.hazelcast.internal.config.AbstractConfigLocator信息: Using configuration file at /root/apache-seatunnel-2.3.3/config/hazelcast-client.yaml2023-12-19 00:11:21,149 INFOcom.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true2023-12-19 00:11:21,233 INFOcom.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING2023-12-19 00:11:21,234 INFOcom.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTEDWARNING: An illegal reflective access operation has occurredWARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/root/apache-seatunnel-2.3.3/starter/seatunnel-starter.jar) to field sun.nio.ch.SelectorImpl.selectedKeysWARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizerWARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING: All illegal access operations will be denied in a future release2023-12-19 00:11:21,294 INFOcom.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel2023-12-19 00:11:21,298 INFOcom.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Trying to connect to [localhost]:58012023-12-19 00:11:21,352 INFOcom.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED2023-12-19 00:11:21,352 INFOcom.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Authenticated with server [localhost]:5801:772efc0a-4c18-4a4b-baa7-b82b9ae4a395, server version: 5.1, local address: /127.0.0.1:360952023-12-19 00:11:21,356 INFOcom.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.2023-12-19 00:11:21,384 INFOcom.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel] [5.1] Members [1] {Member [localhost]:5801 - 772efc0a-4c18-4a4b-baa7-b82b9ae4a395}2023-12-19 00:11:21,421 INFOcom.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.2023-12-19 00:11:21,706 INFOorg.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[]2023-12-19 00:11:21,733 INFOorg.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: /root/apache-seatunnel-2.3.3/config/v2.batch.config.template2023-12-19 00:11:21,799 INFOorg.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64]2023-12-19 00:11:21,848 INFOorg.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {"env" : {"execution.parallelism" : 2,"job.mode" : "BATCH","checkpoint.interval" : 10000},"source" : [{"schema" : {"fields" : {"name" : "string","age" : "int"}},"row.num" : 16,"parallelism" : 2,"result_table_name" : "fake","plugin_name" : "FakeSource"}],"sink" : [{"plugin_name" : "Console"}]}2023-12-19 00:11:21,885 INFOorg.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'2023-12-19 00:11:21,886 INFOorg.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'2023-12-19 00:11:21,895 INFOorg.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /root/apache-seatunnel-2.3.3/connectors/seatunnel2023-12-19 00:11:21,911 INFOorg.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar2023-12-19 00:11:21,912 INFOorg.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Console at: file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar2023-12-19 00:11:21,915 INFOorg.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations.2023-12-19 00:11:21,915 INFOorg.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'2023-12-19 00:11:21,915 INFOorg.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'2023-12-19 00:11:21,916 WARNorg.apache.seatunnel.engine.core.parse.ConfigParserUtil - This configuration is not recommended. A source/transform(FakeSource) is configured with 'result_table_name' option value of 'fake', but subsequent transform/sink(Console) is not configured with 'source_table_name' option.2023-12-19 00:11:21,919 INFOorg.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sources.2023-12-19 00:11:21,919 INFOorg.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'2023-12-19 00:11:21,953 INFOorg.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /root/apache-seatunnel-2.3.3/connectors/seatunnel2023-12-19 00:11:21,970 INFOorg.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar2023-12-19 00:11:21,974 INFOorg.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='FakeSource'} from classpath2023-12-19 00:11:22,003 INFOorg.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all transforms.2023-12-19 00:11:22,003 INFOorg.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sinks.2023-12-19 00:11:22,004 INFOorg.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'2023-12-19 00:11:22,011 INFOorg.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'2023-12-19 00:11:22,090 INFOorg.apache.seatunnel.engine.client.job.ClientJobProxy - Start submit job, job id: 789162834679300097, with plugin jar [file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar, file:/root/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar]2023-12-19 00:11:22,893 INFOorg.apache.seatunnel.engine.client.job.ClientJobProxy - Submit job finished, job id: 789162834679300097, job name: SeaTunnel2023-12-19 00:11:22,956 WARNorg.apache.seatunnel.engine.client.job.JobMetricsRunner - Failed to get job metrics summary, it maybe first-run2023-12-19 00:11:24,370 INFOorg.apache.seatunnel.engine.client.job.ClientJobProxy - Job (789162834679300097) end with state FINISHED2023-12-19 00:11:24,416 INFOorg.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - *********************************************** Job Statistic Information***********************************************Start Time: 2023-12-19 00:11:21End Time: 2023-12-19 00:11:24Total Time(s) : 2Total Read Count:32Total Write Count :32Total Failed Count: 0***********************************************2023-12-19 00:11:24,416 INFOcom.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN2023-12-19 00:11:24,422 INFOcom.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [localhost]:5801:772efc0a-4c18-4a4b-baa7-b82b9ae4a395, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:36095->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-12-19 00:11:24.411, lastWriteTime=2023-12-19 00:11:24.371, closedTime=2023-12-19 00:11:24.420, connected server version=5.1}2023-12-19 00:11:24,422 INFOcom.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED2023-12-19 00:11:24,431 INFOcom.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN2023-12-19 00:11:24,433 INFOorg.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......2023-12-19 00:11:24,433 INFOorg.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed metrics executor service ......2023-12-19 00:11:24,438 INFOorg.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal3. SeaTunnel-Web安装部署
3.1 下载安装包
安装包下在地址如下：
https://seatunnel.apache.org/download解压：
tar -zxvf apache-seatunnel-web-bin-${project.version}.tar.gz解压目录如下：
3.2 初始化数据库脚本或修改配置application.yml中的数据库连接信息
3.2.1 不使用它的初始化脚本自动创建
事先手动执行好脚本，然后在修改application.yml文件中的数据库连接信息
3.2.2 使用脚本初始化数据库
首先，设置变量如下：
export HOSTNAME="localhost"export PORT="3306"export USERNAME="root"export PASSWORD="123456"然后执行：
sh apache-seatunnel-web-bin-2.3.3/script/init_sql.sh如果环境变量有冲突需要改下名字和init_sql.sh中的环境变量的名字，可以加一个前缀可以加上前缀`STWEB_ ，然后把init_sql.sh中上面的那几个环境变量也加上这个前缀即可，然后执行初始化命令
3.3 修改端口与数据源
vim conf/application.yml修改端口号和数据源信息
3.4 两个文件拷贝配置
需要将apache-seatunnel-2.3.3/config/hazelcast-client和apache-seatunnel-2.3.3/connectors/plugin-mapping.properties文件拷贝到apache-seatunnel-web-bin-2.3.3的conf下。
3.5 拷贝jar到lib下
3.6 启动
sh bin/seatunnel-backend-daemon.sh startjps查看java进程如下：
这里有个很坑的地方就是，执行命令如果是进入bin路径执行，访问首页的时候就会有问题，访问首页就会报404的错
sh seatunnel-backend-daemon.sh start访问首页报错如下：
3.7 访问首页
访问ip:8081/ui，此端口为conf/application.yml中配置的端口
http://192.168.1.4:8081/
登录不上有可能是没有启动mysql数据库
service mysqld start #使用service启动mysql服务service mysqld status#查看MYSQL运行状态service mysqld stop#使用service停止mysql服务service mysqld restart #使用service重启mysql服务systemctl enable mysqld.service #设置MySQL服务开机自动启动systemctl is-enabled mysqld.service #确认MySQL服务已经被设置为自动启动
3.8 执行myslq-jdbc–>mysl-jdbc的单表数据同步
执行是成功的，但是在我的CentOs7.x的虚拟机上我是没有安装hadoop3.1.3的环境，执行看日志是没有报错的，官方说是这个hadoop的环境不是必须的也是有道理的，但是之前的文章在本地调试的时候没有hadoop环境会报错，所以建议需要本地编译构建的小伙伴还是安装一个hadoop，以免执行任务报错。
4. 总结
本次分享就到此结束了，踩了一些坑，还是请教了官方的大佬才知道的是哪里有问题，希望我的分享对你有所帮助，请一键三连，么么么哒！