GLM (General Language Model) It is a general-purpose language model pre-trained with autoregressive filling-in-the-blank targets launched by Tsinghua University, which can be fine-tuned for various natural language understanding and generation tasks.

GLM improves on gap-fill pre-training by adding 2D positional encoding and allowing prediction spans in arbitrary order, resulting in better performance than BERT and T5 on NLU tasks. At the same time, GLM can be pre-trained for different types of tasks by changing the number and length of blanks. GLM outperforms BERT, T5, and GPT given the same model size and data on a wide range of tasks across NLU, conditional, and unconditional generation, and achieves 1.25x BERT Larger parameters from a single pretrained model The best performance of , indicating its generalizability to different downstream tasks.

For a detailed description of GLM, please refer to the paper GLM: General Language Model Pretraining with Autoregressive Blank Infilling (ACL 2022)

ChatGLM-6B It is optimized for Chinese QA and dialogue on the basis of the GLM framework.

pre-trained model

Available from OneDrive or Tsinghua-Cloud Download the pretrained model used in the paper.

Unzip the downloaded file into a local folder and set in the corresponding scriptCHECKPOINT_PATHis the folder path.

result

SuperGLUE

Validation set, single model, single task fine-tuning

Seq2Seq

CNN/Daily Mail (test set, no extra data used)

XSum (test set, no extra data used)

Language Modeling

test set, zero sample