BPE Tokenizer

[wannaphong]

Is it possible use bpe tokenizer instead rwkv_vocab_v20230424 in the next model?

I tried rwkv model in Thai language. It look good but it is very slow because Thai is character level for rwkv_vocab_v20230424.

I think if the next model use bpe tokenizer like qwen2, It can improve model and the speed.

[BlinkDL]

or we can simply improve thai tokens in rwkv_vocab_v20230424 :)

please try to improve it and let me know

[ofou]

Is there any way to use a custom tokenizer by the way? (like UTF-8 ids)

I'd be cool to get rid of the tokenizer itself, obviously at the expense of losing some computing efficiency for more interpretability, while covering the whole Unicode.