TagsArticle Source
Beyond Transformers - Intro to RWKV Architecture & The World To …
Abstract
Beyond Transformers - Intro to RWKV Architecture & The World Tokenizer - Eugene Cheah & Harrison Vanderbyl, Recursal AI
Whats comes next after transformers? Introducing RWKV, a linear transformer with 10 to a 100x lower inference cost. And the worlds greenest AI model._ With an architecture lightweight enough to even run on CPUs, or any modern mobile devices. Completely open source, apache2, under the linux foundation. We will be covering the following major topics
- A short introduction to RWKV
- How we plan to scale and build a better model for everyone in the world, with this open source model_x000D_
- How its architecture work, scale and perform
- The RWKV world tokenizer
- How existing AI models tokenizers limits its use cases in non-english use cases, including european languages and character languages (ie. chinese)
- How and why we built the open source world tokenizer
- And how it can benefit all AI models (not just RWKV)