Sure it's not as good and not an actual functional component you'd likely use in say, a production app. But it proves that there are other ways to approach these problem besides scaling up. I have hope that even tasks we currently only know how to use Transformers or convnets for will end up having 'workarounds' or at least other algorit…
Sure it's not as good and not an actual functional component you'd likely use in say, a production app. But it proves that there are other ways to approach these problem besides scaling up. I have hope that even tasks we currently only know how to use Transformers or convnets for will end up having 'workarounds' or at least other algorithms. This particular algorithm doesn't illustrate a step on the way to that, but it does speak to the fact that there are plenty of other ways out there waiting for us still to find them and we shouldn't be myopic about the SOTA.
The tokenization as in this OP is largely what is used for language translation that is OK, and dandy, say an app that as quick as you can clip a page of an old language text, in any language gets translated, and then to be able to instantly translate into any language offline with no 'nickels going to the gpt-4 meter'; ( or your data being fed to google or palantir open-ai cia spy centers )
RLE run-length-encoding is all linear & deterministic, so it should scale fine; There are 1,000's of algo's, so a shit detector should be AI trained to determine the best algo for particular data, for instance some RLE can compress a photo by 99%, while not useful for text;
...
In 5-10 years space will hardly matter, but today with prevalent 64gb smart-phones, its like deja-vu programming on a 4k ram memory all over again 1970's, where Algo's rule;
For now try to be the first kid on the block with killer AI app's on mobile phones that allow the user to capture and use huge amounts of data making their experience seem infinite.
Vector Tokenization ( database ) is more useful searching paragraphs in similar material involving 1,000's of documents;
In the coming WAR next year to put LLM models on mobile-phones they first to market will get rich;
The issue is to cut over-head for the weights and the prompts and token lists in&out;
If you can say cut the current token-list say 90% then you can be the first on your block with a lean&mean mobile app that does real AI;
People are gong to want 8K, 16K, 48K prompts, and they will expect to get back same-same 48K of data or more, so the first kids on the block that figure out how to compress all this crap by 90% win.
Sure it's not as good and not an actual functional component you'd likely use in say, a production app. But it proves that there are other ways to approach these problem besides scaling up. I have hope that even tasks we currently only know how to use Transformers or convnets for will end up having 'workarounds' or at least other algorithms. This particular algorithm doesn't illustrate a step on the way to that, but it does speak to the fact that there are plenty of other ways out there waiting for us still to find them and we shouldn't be myopic about the SOTA.
The tokenization as in this OP is largely what is used for language translation that is OK, and dandy, say an app that as quick as you can clip a page of an old language text, in any language gets translated, and then to be able to instantly translate into any language offline with no 'nickels going to the gpt-4 meter'; ( or your data being fed to google or palantir open-ai cia spy centers )
RLE run-length-encoding is all linear & deterministic, so it should scale fine; There are 1,000's of algo's, so a shit detector should be AI trained to determine the best algo for particular data, for instance some RLE can compress a photo by 99%, while not useful for text;
...
In 5-10 years space will hardly matter, but today with prevalent 64gb smart-phones, its like deja-vu programming on a 4k ram memory all over again 1970's, where Algo's rule;
For now try to be the first kid on the block with killer AI app's on mobile phones that allow the user to capture and use huge amounts of data making their experience seem infinite.
Vector Tokenization ( database ) is more useful searching paragraphs in similar material involving 1,000's of documents;
In the coming WAR next year to put LLM models on mobile-phones they first to market will get rich;
The issue is to cut over-head for the weights and the prompts and token lists in&out;
If you can say cut the current token-list say 90% then you can be the first on your block with a lean&mean mobile app that does real AI;
People are gong to want 8K, 16K, 48K prompts, and they will expect to get back same-same 48K of data or more, so the first kids on the block that figure out how to compress all this crap by 90% win.