GigaToken: ~1000x faster Language model tokenization

 (github.com)

616 points | by syrusakbary 3 days ago ago

134 comments