Import tiktoken

Anvil Q&A
10

I had some time to play with it tonight, and here’s how to access the tiktoken Javascript library from the client:

        self.tiktoken = anvil.js.import_from("https://esm.sh/js-tiktoken@1.0.14")
        self.encoding = self.tiktoken.getEncoding("gpt2")
        print(self.encoding.encode("Hello World"))

esm.sh is a site that’s designed to package these module based libraries for use like this (thanks to Stu Cork for his excellent post about libraries like this: Importing javascript plugin libraries - #9).

Once you have the import done, you can use print(dir(self.tiktoken)) to see what’s available on the module. getEncoding was one of those that dir showed, and that matched up with the Javascript sample code, so it looks like the import worked. And the encoding prints out something.

Presumably you have a better idea than I do about what to do with it from there.

4 Likes