说完了Besu客户端,Tessera隐私交易,是时候也说一下“三剑客”中的最后一位EthSigner。关于EthSigner在《【区块链】HyperLedger Besu Vault密钥服务》一文中稍微有提及过,它主要用作私有链(或联盟链)的区块签署使用的。

之前都没有跟大家详细说一下区块链组件之间是怎么交互的。为了方便说明,我引用官网上用一个隐私交易的交易过程进行说明,如下图:

从Dapp触发一个隐私交易请求(调用eea_sendTransaction接口)给EthSigner,EthSigner会根据Dapp传送的密钥信息找到链上用户签署密钥。接下来EthSigner会通过调用eea_sendRawTransaction接口将交易数据提交Besu上链,Besu会先判断这个交易是普通交易还是隐私交易,若是隐私交易的情况下会转发给Tessera进行隐私交易存证(对双方交易内容包括账号,数据进行隐私处理)后再回传给Besu进行上链。与此同时EthSigner生成的密钥会对上链区块进行数字封存,至此完成了隐私交易的数据上链。至于节点之间同步的我就不再多说了,看过前几篇文章的同学应该都清楚了。

在这里再补充一点信息,如下图:

如上图所示,EthSigner并不是必须的区块链组件。若Dapp足够优秀可以直接取代EthSigner,只不过目前“三剑客”组合仍然是官方推荐的实施方案而已。

说了这么多,下面我们就来看一下关于EthSigner的部署吧。由于我已经废弃了Vault方案,因此这里将直接采用本地密钥存储的方案实行。本地签署密钥需要用到nodejs生成,因此要先安装npm和cnpm(这是因为签署密钥需要通过web3.js生成)。

# 安装npmapt-get install npm# 安装cnpmnpm config set registry https://registry.npm.taobao.orgnpm install -g cnpm --registry=https://registry.npm.taobao.org

在安装cnpm后执行cnpm install web3的时候有机会会报错:

root@node203:~# cnpm install web3internal/modules/cjs/loader.js:638throw err;^Error: Cannot find module 'fs/promises'at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)at Function.Module._load (internal/modules/cjs/loader.js:562:25)at Module.require (internal/modules/cjs/loader.js:692:17)at require (internal/modules/cjs/helpers.js:25:18)at Object.<anonymous> (/usr/local/lib/node_modules/cnpm/node_modules/npminstall/bin/install.js:10:12)at Module._compile (internal/modules/cjs/loader.js:778:30)at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)at Module.load (internal/modules/cjs/loader.js:653:32)at tryModuleLoad (internal/modules/cjs/loader.js:593:12)at Function.Module._load (internal/modules/cjs/loader.js:585:3)

这是因为安装的cnpm版本过高引起的,这个时候需要手动降级。先删除原来的版本

root@node203:~# npm uninstall -g cnpmremoved 550 packages in 2.009s

然后重新安装一个低版本即可(我这里安装7.1.0版本)

root@node203:~# npm install cnpm@7.1.0 -gnpm WARN deprecated uuid@3.4.0: Please upgradeto version 7 or higher.Older versions may use Math.random() in certain circumstances, which is known to be problematic.See https://v8.dev/blog/math-random for details.npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142npm WARN deprecated har-validator@5.1.5: this library is no longer supported/usr/local/bin/cnpm -> /usr/local/lib/node_modules/cnpm/bin/cnpmnpm WARN notsup Unsupported engine for open@8.4.0: wanted: {"node":">=12"} (current: {"node":"10.19.0","npm":"6.14.4"})npm WARN notsup Not compatible with your version of node/npm: open@8.4.0+ cnpm@7.1.0added 845 packages from 995 contributors in 17.895s

之后通过cnpm install web3即可完成安装

root@node203:/home/yzh/Documents/blockchain/ethsigner/shell/nodejs# cnpm install web3✔ Installed 1 packages✔ Linked 340 latest versions[1/8] scripts.postinstall web3@1.7.4 › web3-shh@1.7.4 run "echo \"WARNING: the web3-shh api will be deprecated in the next version\"", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_web3-shh@1.7.4@web3-shh"WARNING: the web3-shh api will be deprecated in the next version[1/8] scripts.postinstall web3@1.7.4 › web3-shh@1.7.4 finished in 7ms[2/8] scripts.install web3@1.7.4 › web3-core@1.7.4 › web3-core-requestmanager@1.7.4 › web3-providers-ws@1.7.4 › websocket@1.0.34 › bufferutil@^4.0.1 run "node-gyp-build", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_bufferutil@4.0.6@bufferutil"[2/8] scripts.install web3@1.7.4 › web3-core@1.7.4 › web3-core-requestmanager@1.7.4 › web3-providers-ws@1.7.4 › websocket@1.0.34 › bufferutil@^4.0.1 finished in 744ms[3/8] scripts.install web3@1.7.4 › web3-core@1.7.4 › web3-core-requestmanager@1.7.4 › web3-providers-ws@1.7.4 › websocket@1.0.34 › utf-8-validate@^5.0.2 run "node-gyp-build", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_utf-8-validate@5.0.9@utf-8-validate"[3/8] scripts.install web3@1.7.4 › web3-core@1.7.4 › web3-core-requestmanager@1.7.4 › web3-providers-ws@1.7.4 › websocket@1.0.34 › utf-8-validate@^5.0.2 finished in 741ms[4/8] scripts.postinstall web3@1.7.4 › web3-bzz@1.7.4 run "echo \"WARNING: the web3-bzz api will be deprecated in the next version\"", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_web3-bzz@1.7.4@web3-bzz"WARNING: the web3-bzz api will be deprecated in the next version[4/8] scripts.postinstall web3@1.7.4 › web3-bzz@1.7.4 finished in 5ms[5/8] scripts.install web3@1.7.4 › web3-utils@1.7.4 › ethereumjs-util@7.1.5 › ethereum-cryptography@0.1.3 › keccak@^3.0.0 run "node-gyp-build || exit 0", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_keccak@3.0.2@keccak"[5/8] scripts.install web3@1.7.4 › web3-utils@1.7.4 › ethereumjs-util@7.1.5 › ethereum-cryptography@0.1.3 › keccak@^3.0.0 finished in 750ms[6/8] scripts.install web3@1.7.4 › web3-utils@1.7.4 › ethereumjs-util@7.1.5 › ethereum-cryptography@0.1.3 › secp256k1@^4.0.1 run "node-gyp-build || exit 0", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_secp256k1@4.0.3@secp256k1"[6/8] scripts.install web3@1.7.4 › web3-utils@1.7.4 › ethereumjs-util@7.1.5 › ethereum-cryptography@0.1.3 › secp256k1@^4.0.1 finished in 740ms[7/8] scripts.postinstall web3@1.7.4 › web3-core@1.7.4 › web3-core-requestmanager@1.7.4 › web3-providers-ws@1.7.4 › websocket@1.0.34 › es5-ext@^0.10.50 run " node -e \"try{require('./_postinstall')}catch(e){}\" || exit 0", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_es5-ext@0.10.61@es5-ext"[7/8] scripts.postinstall web3@1.7.4 › web3-core@1.7.4 › web3-core-requestmanager@1.7.4 › web3-providers-ws@1.7.4 › websocket@1.0.34 › es5-ext@^0.10.50 finished in 376ms[8/8] scripts.postinstall web3@latest run "echo \"WARNING: the web3-shh and web3-bzz api will be deprecated in the next version\"", root: "/home/yzh/Documents/blockchain/ethsigner/shell/nodejs/node_modules/_web3@1.7.4@web3"WARNING: the web3-shh and web3-bzz api will be deprecated in the next version[8/8] scripts.postinstall web3@latest finished in 5ms✔ Run 8 scriptsanti semver web3@1.7.4 › web3-core@1.7.4 › @types/bn.js@5.1.0 › @types/node@* delcares @types/node@*(resolved as 18.0.3) but using ancestor(web3-core)'s dependency @types/node@^12.12.6(resolved as 12.20.55)deprecate web3@1.7.4 › web3-bzz@1.7.4 › swarm-js@0.1.40 › mkdirp-promise@^5.0.1 This package is broken and no longer maintained. 'mkdirp' itself supports promises now, please switch to that.deprecate web3@1.7.4 › web3-eth@1.7.4 › web3-eth-accounts@1.7.4 › uuid@3.3.2 Please upgradeto version 7 or higher.Older versions may use Math.random() in certain circumstances, which is known to be problematic.See https://v8.dev/blog/math-random for details.deprecate web3@1.7.4 › web3-eth@1.7.4 › web3-eth-ens@1.7.4 › content-hash@2.5.2 › multicodec@^0.5.5 This module has been superseded by the multiformats moduledeprecate web3@1.7.4 › web3-bzz@1.7.4 › swarm-js@0.1.40 › eth-lib@0.1.29 › servify@0.1.12 › request@^2.79.0 request has been deprecated, see https://github.com/request/request/issues/3142deprecate web3@1.7.4 › web3-eth@1.7.4 › web3-eth-ens@1.7.4 › content-hash@2.5.2 › multihashes@0.4.21 › multibase@^0.7.0 This module has been superseded by the multiformats moduledeprecate web3@1.7.4 › web3-bzz@1.7.4 › swarm-js@0.1.40 › eth-lib@0.1.29 › servify@0.1.12 › request@2.88.2 › har-validator@~5.1.3 this library is no longer supporteddeprecate web3@1.7.4 › web3-bzz@1.7.4 › swarm-js@0.1.40 › eth-lib@0.1.29 › servify@0.1.12 › request@2.88.2 › uuid@^3.3.2 Please upgradeto version 7 or higher.Older versions may use Math.random() in certain circumstances, which is known to be problematic.See https://v8.dev/blog/math-random for details.deprecate web3@1.7.4 › web3-eth@1.7.4 › web3-eth-ens@1.7.4 › content-hash@2.5.2 › cids@^0.7.1 This module has been superseded by the multiformats moduledeprecate web3@1.7.4 › web3-eth@1.7.4 › web3-eth-ens@1.7.4 › content-hash@2.5.2 › cids@0.7.5 › multicodec@^1.0.0 This module has been superseded by the multiformats moduledeprecate web3@1.7.4 › web3-eth@1.7.4 › web3-eth-ens@1.7.4 › content-hash@2.5.2 › cids@0.7.5 › multibase@~0.6.0 This module has been superseded by the multiformats module✔ All packages installed (362 packages installed from npm registry, used 6s(network 3s), speed 0B/s, json 0(0B), tarball 0B, manifests cache hit 340, etag hit 0 / miss 0)

再之后就就要安装jq插件。安装jq插件的目的是解析http请求返回使用的,譬如:启动区块链服务后我需要通过curl访问restful接口并解析返回json字符串,这个时候通过jq插件就能够轻松解析出结果,就不需要一层一层去反解返回的json字符串了(因为自己是开发出身,所以不经意会以开发的思路解决问题,如果有更好的解决方式jq插件其实可以不安装)。

root@node203:/home/yzh/Documents/blockchain/ethsigner/shell# apt-get install jqReading package lists... DoneBuilding dependency tree Reading state information... DoneThe following additional packages will be installed:libjq1 libonig5The following NEW packages will be installed:jq libjq1 libonig50 upgraded, 3 newly installed, 0 to remove and 153 not upgraded.Need to get 313 kB of archives.After this operation, 1,062 kB of additional disk space will be used.Do you want to continue" />[Y/n] yGet:1 http://mirrors.ustc.edu.cn/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB]Get:2 http://mirrors.ustc.edu.cn/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB]Get:3 http://mirrors.ustc.edu.cn/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB]Fetched 313 kB in 0s (812 kB/s)Selecting previously unselected package libonig5:amd64.(Reading database ... 328030 files and directories currently installed.)Preparing to unpack .../libonig5_6.9.4-1_amd64.deb ...Unpacking libonig5:amd64 (6.9.4-1) ...Selecting previously unselected package libjq1:amd64.Preparing to unpack .../libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ...Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ...Selecting previously unselected package jq.Preparing to unpack .../jq_1.6-1ubuntu0.20.04.1_amd64.deb ...Unpacking jq (1.6-1ubuntu0.20.04.1) ...Setting up libonig5:amd64 (6.9.4-1) ...Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ...Setting up jq (1.6-1ubuntu0.20.04.1) ...Processing triggers for man-db (2.9.1-1) ...Processing triggers for libc-bin (2.31-0ubuntu9.7) ...

最后就要准备好一个密钥创建脚本(create_keyfile.js),如下:

var args = process.argv.splice(2);const Web3 = require('web3');//Web3初始化(指向JSON-RPC节点)const web3 = new Web3(new Web3.providers.HttpProvider('http://127.0.0.1:8545'));var V3KeyStore = web3.eth.accounts.encrypt(args[0], args[1]);console.log(JSON.stringify(V3KeyStore));process.exit();

至此全部前期准备都已经做好,接下来可以通过脚本创建用户了。事不宜迟我们可以先写个脚本(ethsigner_run.sh):

# 节点安装路径deploy_path=/home/yzh/Documents/blockchain/docker/ethsigner# nodejs生成用户密钥文件路径createkey_path=/home/yzh/Documents/blockchain/config/ethsigner/nodejs/create_keyfile.js# ethsigner docker镜像名称docker_image=consensys/ethsigner:21.10.0# 签署用户(虚拟账号)生成个数signer_counter=2000# 区块链id(可以在创世块配置文件中获取)chain_id=12345# 自定义校验节点名称custom_name=node202_signer# 校验节点静态ipethsigner_ip=192.18.0.212# 签署节点ipsigner_node_ip=192.18.0.202# 签署用户(虚拟账号)存档路径has_users=/home/yzh/Documents/blockchain/config/ethsigner/users# 分多少个线程并发thread_num=100if [ ! -d $deploy_path ]; then# 1. 判断是否存在部署目录和是否已经存在用户了if [ ! -d $has_users ]; then# 临时文件名称tmpfifo="tmpfifo"mkfifo ${tmpfifo}exec 6${tmpfifo}rm -f ${tmpfifo}for ((i = 1; i <= ${thread_num}; i++)); do{echo}done >&6# 2. 若不存在则创建部署目录和用户目录mkdir -p $deploy_path && mkdir -p $deploy_path/signer_addressmkdir -p $has_users && chmod 777 -R $has_users# 3. 将密码信息转换成数组并通过循环读取每一个密码信息for ((i = 1; i <= $signer_counter; i++)); do{read -u6{echo "Currently $i accounts have been generated"# 4. 生成签署账号64位的私钥hex_address=$(openssl rand -hex 32)# 5. 然后通过执行createkey.js生成密钥信息keyfile=$(node $createkey_path "0x"$hex_address $i"_BCA")# 6. 通过解析返回的json获取到签署节点的加密密文address_val=$(echo $keyfile | jq '.address')address_val=$(echo $address_val | sed 's/\"//g')# 7. 根据上面收集的内容生成keyFile、passwordFile和配置用的toml文件keyFilePath=$deploy_path/$address_val/keyFilepasswordFilePath=$deploy_path/$address_val/passwordFiletomlFilePath=$deploy_path/signer_address/$address_val".toml"# 8. 将信息按照要求写入到指定的文件里面,这里需要注意的配置文件中关于路径的写法一定要按照镜像中机器的写法不然是没有找到对应路径的mkdir -p $deploy_path/$address_valtouch $keyFilePath &&touch $passwordFilePath && touch $tomlFilePathchmod 777 -R $deploy_path# 9. 将密码写入到passwordFileecho -e $i"_BCA" >>$passwordFilePath# 10. 将生成的key内容写入keyFileecho -e "$keyfile" >>$keyFilePath# 11. 将权限配置文件写入toml配置echo -e "[metadata]" >>$tomlFilePathecho -e "createdAt = $(date "+%Y-%m-%d %H:%M:%S")" >>$tomlFilePathecho -e "description = \"$hex_address account configuration\" \n" >>$tomlFilePathecho -e "[signing] \ntype = \"file-based-signer\"" >>$tomlFilePathecho -e "key-file = \"/var/lib/ethsigner/$address_val/keyFile\"" >>$tomlFilePathecho -e "password-file = \"/var/lib/ethsigner/$address_val/passwordFile\"" >>$tomlFilePathecho "" >&6} &}donewaitexec 6>&-echo "All accounts are generated"# 12. 将配置信息进行备份find $deploy_path -name "*" -exec cp -rf {} $has_users \;elsemkdir -p $deploy_pathfind $has_users -name "*" -exec cp -rf {} $deploy_path \;fi# 13. 使用docker来启动eth_signerdocker run --name $custom_name --network=besu_swarm --ip $ethsigner_ip -p 8945:8545 --mount type=bind,source=$deploy_path,target=/var/lib/ethsigner -d $docker_image --chain-id=$chain_id --downstream-http-host=$signer_node_ip --downstream-http-port=8545 --http-listen-host=0.0.0.0 multikey-signer --directory=/var/lib/ethsigner/signer_addressfi

以上是通过shell模拟多线程执行的脚本用来创建多个区块链签署密钥,脚本上带有注释应该能够看得明白的,但是脚本中关于配置文件的读取其实需要按照一定的规则进行放置,这里由于涉及到不可描述的原因这里就没有提供,并且为了使分享内容不牵扯到知识产权的纠纷问题,脚本内容我也是有做调整。所以脚本不能直接使用,这里只是给各位一个实现思路。

直接执行脚本即可生成本地密钥,如下:

root@node203:/home/yzh/Documents/blockchain/config/ethsigner/shell# ./ethsigner_run.sh Currently 1 accounts have been generatedCurrently 2 accounts have been generatedCurrently 3 accounts have been generatedCurrently 4 accounts have been generatedCurrently 6 accounts have been generatedCurrently 5 accounts have been generatedCurrently 7 accounts have been generatedCurrently 8 accounts have been generatedCurrently 9 accounts have been generatedCurrently 10 accounts have been generatedCurrently 11 accounts have been generatedCurrently 13 accounts have been generatedCurrently 15 accounts have been generatedCurrently 16 accounts have been generatedCurrently 21 accounts have been generatedCurrently 17 accounts have been generatedCurrently 18 accounts have been generatedCurrently 14 accounts have been generatedCurrently 19 accounts have been generatedCurrently 25 accounts have been generatedCurrently 23 accounts have been generatedCurrently 28 accounts have been generatedCurrently 35 accounts have been generatedCurrently 12 accounts have been generatedCurrently 22 accounts have been generatedCurrently 30 accounts have been generatedCurrently 29 accounts have been generated…… 

由于生成过程采用了快速分片模拟多线程的情况,因此看到的输出不是顺序输出

Currently 1995 accounts have been generatedCurrently 1994 accounts have been generatedCurrently 1996 accounts have been generatedCurrently 1997 accounts have been generatedCurrently 1998 accounts have been generatedCurrently 1999 accounts have been generatedCurrently 2000 accounts have been generatedAll accounts are generated5d17812000e480798ef96264a88df76681d92c3f9db8f0baef81035929a1b589

看到这样输出后证明脚本已经执行完成,这个时候可以通过docker logs -f查看服务的输出情况

root@node203:/home/yzh/Documents/blockchain/config/ethsigner/shell# docker logs -f 5d17812000e4Setting logging level to INFO2022-07-07 10:05:08.875+00:00 | main | INFO| SignerSubCommand | Version = ethsigner/v21.10.0/linux-x86_64/-na-openjdk64bitservervm-java-112022-07-07 10:05:09.438+00:00 | main | INFO| Runner | Server is up, and listening on 8545

看到这样的输出证明启动已经成功。

由于签署密钥还要提供给应用程序(Dapp)使用用于与系统账号绑定,我这边是选择将密钥同步到Redis中由应用程序直接跟Redis对接获取密钥信息,于是我又写了一个shell脚本用于将密钥同步到Redis,如下:

scan_path=/home/yzh/Documents/blockchain/config/ethsigner/users# redis微服务ip地址和端口redis_mic_ip=192.168.100.138redis_mic_port=6379# 当前节点编码node_code=YZHdeclare -i count=0# 1. 初始化setup变量为后面拼接字符串做准备setup=""# 2. 遍历指定的文件目录获取到所有后缀为toml文件for file in $(ls $scan_path)do if [ "${file##*.}" = "toml" ]; thenparam="${file%%.*}"# 3. 开始拼接字符串setup="$setup $param"# 4. 由于redis对于传入参数的长度有限制,因此需要分批进行这里面将按照设定进行计数count=$((${count} + 1))fi# 5. 每当循环了100次的时候就保存一次到redis中if (( $count ==100 )); thendocker run --rm --name redis-cli -it goodsmileduck/redis-cli redis-cli -h $redis_mic_ip -p $redis_mic_port RPUSH ETH_ACCOUNTS"_"$node_code $setupcount=0setup=""fidone

这个脚本其实说白了就是将磁盘中的文件读取然后通过redis-cli镜像写入到目标redis里面而已。效果如下:

root@node203:/home/yzh/Documents/blockchain/config/ethsigner/shell# ./setup_account_to_redis.sh (integer) 100(integer) 200(integer) 300(integer) 400(integer) 500(integer) 600(integer) 700(integer) 800(integer) 900(integer) 1000(integer) 1100(integer) 1200(integer) 1300(integer) 1400(integer) 1500(integer) 1600(integer) 1700(integer) 1800(integer) 1900(integer) 2000

执行脚本后就能够在redis客户端中看到数据存储的情况,如下图:

至此,Besu区块链的基础组件部署就演示完毕了,接下来就需要看看实际效果了。