使用 OpenCompass 评测 InternLM2-Chat-7B 模型使用 LMDeploy 0.2.0 部署后在 C-Eval 数据集上的性能

步骤1：下载internLM2-Chat-7B 模型,并进行挂载

以下命令将internlm2-7b模型挂载到当前目录下：

ln -s /share/model_repos/internlm2-7b/ ./

步骤2：编译安装LMdeploy0.2.0

pip install 'lmdeploy[all]==v0.2.0'

步骤3：使用LMdeploy 将模型internLM2-Chat-7B 进行转换

lmdeploy convert internlm2-chat-7b /root/model/Shanghai_AI_Laboratory/internlm2-chat-7b

运行日志：

(internlm-demo) root@intern-studio:~/deploy# lmdeploy convert internlm2-chat-7b /root/model/Shanghai_AI_Laboratory/internlm2-chat-7bcreate workspace in directory workspacecopy triton model templates from "/root/.conda/envs/internlm-demo/lib/python3.10/site-packages/lmdeploy/serve/turbomind/triton_models" to "workspace/triton_models"copy service_docker_up.sh from "/root/.conda/envs/internlm-demo/lib/python3.10/site-packages/lmdeploy/serve/turbomind/service_docker_up.sh" to "workspace"model_name internlm2-chat-7bmodel_format Noneinferred_model_formatinternlm2model_path /root/model/Shanghai_AI_Laboratory/internlm2-chat-7btokenizer_path /root/model/Shanghai_AI_Laboratory/internlm2-chat-7b/tokenizer.modeloutput_formatfp1601/29 17:36:32 - lmdeploy - WARNING - Can not find tokenizer.json. It may take long time to initialize the tokenizer.*** splitting layers.0.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1 *** splitting layers.0.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1 *** splitting layers.0.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.0.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.0.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1 *** splitting layers.1.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1 *** splitting layers.1.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1 *** splitting layers.1.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.1.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.1.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1 *** splitting layers.2.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1 *** splitting layers.2.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1 *** splitting layers.2.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.2.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.2.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1 *** splitting layers.3.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1 *** splitting layers.3.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1 *** splitting layers.3.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.3.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.3.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1 *** splitting layers.4.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1 *** splitting layers.4.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1 *** splitting layers.4.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.4.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.4.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1 *** splitting layers.5.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1 *** splitting layers.5.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1 *** splitting layers.5.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.5.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.5.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1 *** splitting layers.6.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1 *** splitting layers.6.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1 *** splitting layers.6.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.6.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.6.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1 *** splitting layers.7.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1 *** splitting layers.7.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1 *** splitting layers.7.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.7.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.7.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1 *** splitting layers.8.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1 *** splitting layers.8.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1 *** splitting layers.8.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.8.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.8.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1 *** splitting layers.9.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1 *** splitting layers.9.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1 *** splitting layers.9.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.9.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1*** splitting layers.9.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1 *** splitting layers.10.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.10.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.10.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.10.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.10.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.11.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.11.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.11.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.11.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.11.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.12.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.12.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.12.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.12.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.12.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.13.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.13.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.13.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.13.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.13.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.14.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.14.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.14.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.14.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.14.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.15.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.15.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.15.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.15.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.15.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.16.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.16.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.16.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.16.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.16.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.17.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.17.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.17.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.17.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.17.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.18.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.18.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.18.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.18.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.18.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.19.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.19.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.19.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.19.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.19.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.20.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.20.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.20.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.20.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.20.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.21.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.21.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.21.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.21.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.21.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.22.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.22.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.22.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.22.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.22.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.23.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.23.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.23.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.23.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.23.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.24.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.24.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.24.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.24.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.24.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.25.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.25.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.25.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.25.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.25.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.26.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.26.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.26.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.26.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.26.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.27.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.27.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.27.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.27.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.27.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.28.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.28.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.28.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.28.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.28.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.29.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.29.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.29.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.29.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.29.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.30.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.30.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.30.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.30.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.30.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1*** splitting layers.31.attention.w_qkv.weight, shape=torch.Size([4096, 6144]), split_dim=-1, tp=1*** splitting layers.31.attention.wo.weight, shape=torch.Size([4096, 4096]), split_dim=0, tp=1*** splitting layers.31.feed_forward.w1.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.31.feed_forward.w3.weight, shape=torch.Size([4096, 14336]), split_dim=-1, tp=1 *** splitting layers.31.feed_forward.w2.weight, shape=torch.Size([14336, 4096]), split_dim=0, tp=1Convert to turbomind format: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:27<00:00,1.18it/s

步骤4：模型结果测评

首先新建config文件，其中参数”/root/deploy/workspace/“表示LMdeploy转换后的模型地址。

from mmengine.config import read_basefrom opencompass.models.turbomind import TurboMindModelwith read_base(): # choose a list of datasetsfrom .datasets.ceval.ceval_gen_5f30c7 import ceval_datasets# and output the results in a choosen format from .summarizers.medium import summarizerdatasets = sum((v for k, v in locals().items() if k.endswith('_datasets')), [])internlm_meta_template = dict(round=[ dict(role='HUMAN', begin=':', end='\n'), dict(role='BOT', begin=':', end='\n', generate=True),], eos_token_id=103028)# config for internlm-chat-7binternlm2_chat_7b = dict( type=TurboMindModel, abbr='internlm2-chat-7b-turbomind', path='/root/deploy/workspace/', engine_config=dict(session_len=512, max_batch_size=2, rope_scaling_factor=1.0), gen_config=dict(top_k=1, top_p=0.8, temperature=1.0, max_new_tokens=100), max_out_len=100, max_seq_len=512, batch_size=2, concurrency=1, meta_template=internlm_meta_template, run_cfg=dict(num_gpus=1, num_procs=1),)models = [internlm2_chat_7b]

在opencompass 目录下运行：

python run.py configs/eval_turbomind.py

同样可以添加–debug ,输出日志信息。

python run.py configs/eval_turbomind.py --debug

过程日志如下：