RWKV is a language model that combines RNN and Transformer. It is suitable for long texts, runs faster, has better fitting performance, occupies less video memory, and takes less time for training.

The overall structure of RWKV still adopts the idea of ​​Transformer Block, and its overall structure is shown in the figure:

Compared with the structure of the original Transformer Block, RWKV replaces self-attention with Position Encoding and TimeMix, and replaces FFN with ChannelMix. The rest is consistent with Transformer.