NLP-预训练模型-中文-封神榜系列：燃灯/Randeng-T5-784M（中文版的mT5-large）【处理各种从源文本转换到目标文本类型的任务，例如机器翻译，文本摘要等】

青旅半醒 2023-10-14 09:37 10阅读 0赞

### 简介 Brief Introduction ###

善于处理NLT任务，中文版的mT5-large。

Good at handling NLT tasks, Chinese mT5-large.

### 模型分类 Model Taxonomy ###

<table> 
 <thead> 
  <tr> 
   <th>需求 Demand</th> 
   <th>任务 Task</th> 
   <th>系列 Series</th> 
   <th>模型 Model</th> 
   <th>参数 Parameter</th> 
   <th>额外 Extra</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>通用 General</td> 
   <td>自然语言转换 NLT</td> 
   <td>燃灯 Randeng</td> 
   <td>mT5</td> 
   <td>784M</td> 
   <td>中文-Chinese</td> 
  </tr> 
 </tbody> 
</table>

### 模型信息 Model Information ###

我们基于mT5-large，训练了它的中文版。为了加速训练，我们仅使用T5分词器(sentence piece)中的中英文对应的词表，并且使用了语料库自适应预训练(Corpus-Adaptive Pre-Training, CAPT)技术在悟道语料库(180G版本)继续预训练。预训练目标为破坏span。具体地，我们在预训练阶段中使用了[封神框架][Link 1]大概花费了16张A100约96小时。

Based on mT5-large, we implement its Chinese version. In order to accelerate training, we only retrain the vocabulary and embedding corresponding to Chinese and English in T5tokenizer (sentence piece), and Corpus-Adaptive Pre-Training (CAPT) on the WuDao Corpora (180 GB version). The pretraining objective is span corruption. Specifically, we use the [fengshen framework][Link 1] in the pre-training phase which cost about 96 hours with 16 A100 GPUs.

### 使用 Usage ###

from transformers import T5ForConditionalGeneration, AutoTokenizer
    import torch
    tokenizer=AutoTokenizer.from_pretrained('IDEA-CCNL/Randeng-T5-784M', use_fast=false)
    model=T5ForConditionalGeneration.from_pretrained('IDEA-CCNL/Randeng-T5-784M')

--------------------

[IDEA-CCNL/Randeng-T5-784M · Hugging Face][IDEA-CCNL_Randeng-T5-784M _ Hugging Face]

[Link 1]: https://github.com/IDEA-CCNL/Fengshenbang-LM/tree/main/fengshen
[IDEA-CCNL_Randeng-T5-784M _ Hugging Face]: https://huggingface.co/IDEA-CCNL/Randeng-T5-784M