Data [de]compression common to Client and Server/Uplink

Anvil Q&A
1

Two of my apps send plain-text reports from Server to Client. It would reduce resource usage if there was a common compression format.

I originally thought to use zipfile compression, but the last time I tried, it wasn’t available on the Client.

To determine the functions available under Skulpt, where would I look?

2

Looks like it should be there:

(and one of the contributors looks strangely familiar).

2 Likes
3

I’ll create a little experiment (app) and check it out.

Thanks, @owen.campbell !

4

That looks more like the Python zip function, not zipfile compression: 2. Built-in Functions — Python 3.3.7 documentation

5

Drat.

NotImplementedImportError: zipfile is not yet implemented in Skulpt.

Edit: I’ll try some of the other standard compression modules.

ImportError: No module named lzma
ImportError: No module named bz2
NotImplementedImportError: gzip is not yet implemented in Skulpt
ImportError: No module named zlib

I suppose JavaScript might have some built-ins…

6

This looked cool when I found it it is all in one file, but JS scares me :zombie:

7

Yeah… I studied JS years ago… the language itself isn’t scary (mostly). And Anvil provides a nice wrapper for JS code. For a commercial app, the code and its source need to be vetted, though.

For built-in, standardized compression, the proposed standard streaming-compression API is apparently not supported by Firefox. I haven’t found other compression routines built into JavaScript. That leaves add-in JS libraries, as an alternative. It looks like several JS [de]compression libraries have been written. But it takes time to test them and prove compatibility with similar routines, written in/for Python, running server-side. Wish I had the time for that right now.

I appreciate everyone’s efforts. Thankfully, this is not a pressing issue.

8

I wonder if the anvil can easily provide you with browser information from the user agent, and you could at least make it work for everything that isn’t Firefox?

(Also, JS scares me because I know I would like it if I started using it, and I don’t want to start :see_no_evil: )

9

We use Firefox rather heavily, so a solution that excludes Firefox isn’t very attractive here.

10

Ah yes. So it is. My bad.

11

Had to transfer 318k and with zlib it went down to 17k:

On the server side:

import zlib

@anvil.server.callable
def test():
 text = 'test' 
 compressed = zlib.compress(bytes(str(text),'utf-8'))
 res = BlobMedia('text/plain', compressed)
 return res

On the client side, you need to add the Pako zlib in the native library or standard_page.html

<script src="https://cdn.jsdelivr.net/pako/1.0.3/pako.min.js"></script>

and the unzip function in standard_page.html

function unzip(data) {
    result = pako.inflate(data, { to: 'string' });
    return result;
  }

then you can use it in your client functions like this:

compressed = anvil.server.call('test')
res = self.call_js('unzip',compressed.get_bytes())
5 Likes
12

Wonderful, @andyfe ! I’ll be looking into it. References:

http://nodeca.github.io/pako/
http://www.zlib.net

13

The docs are for v2.0.4. Adjust the tag accordingly?

14

Sorry, what do you mean by that?

15

The version number in

does not match
image

so presumably the HTML script tags should instead read

<script src="https://cdn.jsdelivr.net/pako/2.0.4/pako.min.js"></script>
16

You’re right. I could not find it on jsdelivr, but it’s on cloudflare:

<script src="https://cdnjs.cloudflare.com/ajax/libs/pako/2.0.4/pako.min.js"></script>
17

Thanks for opening my eyes. I may be looking for jszip soon.

If I’m going to go third-party JavaScript, I may as well go for one that handles the compressor we’re already using, server-side. (Zip format has the advantage of packaging multiple files, so we can include both finished reports, and their supporting data, in a single, unified package.)

18

Sorry for reviving an old thread but would this approach also work when returning a dict/json to the client? I’m little stuck trying to get it to work with a dict

19

This should work…

On the server side:

import zlib
import json

@anvil.server.callable
def test():
 data = {"hello": "world"}
 compressed = zlib.compress(bytes(json.dumps(data),'utf-8'))
 res = BlobMedia('text/plain', compressed)
 return res

and on the client

import json

compressed = anvil.server.call('test')
data_str = self.call_js('unzip',compressed.get_bytes())
data = json.loads(data_str)
1 Like
20

I’m glad to see this topic revived, as I left it without a definitive answer.

Following up on @andyfe’s new work, I added a few missing imports, and other syntax fixes:

As you can see, the Client gets

NameError: Could not find global JS function 'unzip'.

Something’s missing. Does this Client code require some third-party javascript library, like pako or jszip?