Serialising dictionary with numeric keys

Feature Requests
1

Hi all - I believe you’re already hard at work enhancing the serialisation of objects between client and server and wanted to check if this case was already being looked at please:

anvil.server.SerializationError: Cannot serialize return value from function. Cannot serialize dictionaries with keys that aren’t strings

I got that error when trying to return a dict similar to {1: "one"}. I could obviously convert int to str on the server then str to int on the client, but a ‘fix’ would be vastly preferable to a workaround, especially if this dovetails with your existing solution for passing objects/classes generally. It’s a simple, pure Python dictionary so I hope you agree Anvil should be able to handle it…

Thanks.

1 Like
2

This error (and others) is the sort of behavior that leads me to suspect a straightforward JSON implementation underneath it all.

The “natural” representation of a dict is as a JSON Object. But in Data types and syntax, it is shown that a JSON Object’s keys are always strings.

When a dict contains a date (or other non-JSON type), we get similar errors.

The code to detect such cases would have to be executed on every call, even in that majority of cases where it is unnecessary. This suggests a two-level scheme, whereby existing calls are left alone, and run at top speed; but an extended version of @anvil.server.callable lets us identify those few cases where more care (and time) is necessary.

3

Yep that all makes sense @p.colbert , and I think the proposed catch-all solution is to offer a decorator e.g. @anvil.server.serializable_type which would cater for the two-level scheme as you describe it. I guess I’m just asking/requesting that whatever solution is offered through such a feature also covers the case where dict keys are integers.

4

Bumping this topic now that the super new Portable Classes are available for use.

As you’ll see from my test code it seems like Anvil still can’t serialise a dictionary with numeric keys (“SimpleDict” in the attached example) whether it’s subclassed and wrapped as a Portable Class, or even if the dict is actually just an attribute of a totally new class (“NumberDict” in the example):

https://anvil.works/build#clone:WZFJQDAKT7WYPMZE=2SVPAUHZC5PIMDACA2KJVRAA

Server Module (copied for convenience)

import anvil.server
from .PortableClassesTest import SimpleDict, NumberDict, JsonDict, print_attributes

# Testing the ability of Anvil Portable Classes to return a dict with numeric keys

@anvil.server.callable
def test_simpledict():
    sd = SimpleDict()
    print_attributes(sd, "SERVER")
    return sd

@anvil.server.callable
def test_numberdict():
    nd = NumberDict()
    print_attributes(nd, "SERVER")
    return nd
  
@anvil.server.callable
def test_jsondict():
    jd = JsonDict()
    print_attributes(jd, "SERVER")
    return jd

Client Module

from ._anvil_designer import PortableClassesTestTemplate
from anvil import *
import anvil.server
import json

def print_attributes(test_dict, client_or_server):
    print(f"This is the {client_or_server} speaking...")
    print("Type:", type(test_dict))
    for attribute in ("__dict__", "dict", "json"):
      if hasattr(test_dict, attribute):
        print(f".{attribute}:", getattr(test_dict, attribute))
    print(test_dict)
    print()
    
@anvil.server.portable_class
class SimpleDict(dict):
  def __init__(self, data={1: "one"}):
    self.__dict__.update(data)
  def __repr__(self):
    return str(vars(self))

@anvil.server.portable_class
class NumberDict:
  def __init__(self, data={2: "two"}):
    self.dict = data

@anvil.server.portable_class
class JsonDict:
  def __init__(self, data={3: "three"}):
    self.json = json.dumps(data)    
  @property
  def dict(self):
    return {int(k):v for k,v in json.loads(self.json).items()}
  def __repr__(self):
    return str(self.dict)

class PortableClassesTest(PortableClassesTestTemplate):
  def __init__(self, **properties):
    # Set Form properties and Data Bindings.
    self.init_components(**properties)
    for test in ("test_simpledict", "test_numberdict", "test_jsondict"):
      try:
        result = anvil.server.call(test)
        print_attributes(result, "CLIENT")
      except Exception as E:
        print(E,"\n")

I’ve come up with a workaround (called “JsonDict” above), but I still can’t help thinking: if the whizzy new Portable Classes can handle so much complexity and sophistication already, wouldn’t it be more elegant to handle this requirement at source as part of the Portable Class feature as well?

If there is already a way of serialising dictionaries with numeric keys and I’m just missing something obivous please let me know! (If the answer is soemthing about “Custom Serialisation”, I haven’t got my head around that yet… and it would need to be simpler than my own workaround to obviate the benefit of this Feature Request anyway…) Otherwise please consider this request as a way of resolving a Python/Json “wrinkle” at source and making the Portable Classes truly serialise any class you throw at them?

1 Like
5

Hi @AWSOM,

Portable classes allow you to transport things that Anvil can’t serialise natively, so your JsonDict implementation is precisely the way to go.

Feature Request noted! :slight_smile:

1 Like
6

A much simpler approach than creating a new JsonDict class occurred to me last night:

Forget about json, just convert your dictionary (with pesky numeric keys) to a list of lists or a list of tuples!

Client code (concise)

data_dict = {1: "one", 2: "two"}
new_data_dict = {k:v for k,v in anvil.server.callable("test_list", list(data_dict.items()))}

Server code (concise)

@anvil.server.callable
def test_list(data_list):
    data_dict =  {k:v for k,v in data_list}
    return list(data_dict.items())

Simples.

@bridget: This feature request now becomes almost trivial to implement… Add it as a fallback method instead of raising anvil.server.SerializationError: Cannot serialize return value from function. Cannot serialize dictionaries with keys that aren’t strings? Or in the meantime perhaps worthy of a footnote in the Valid arguments and return values?

Client code (verbose for readability and checking output)

def dict_to_list(self):
    """Sends a list, receives a list and converts back to a dict"""
    data_dict = {1: "one", 2: "two"}
    print("Client Data Dict (input):", data_dict)
    data_list = list(data_dict.items())
    print("Client Data List (intermediate):", data_list)
    new_data_dict = {k:v for k,v in anvil.server.callable("test_list", data_list)}
    print("Client Data Dict (output):", data_dict)

Server code (verbose for readability and checking output)

@anvil.server.callable
def test_list(data_list):
    """Receives a list, converts to a dict, returns a list"""
    print("Server Data List (input):", data_list)
    data_dict =  {k:v for k,v in data_list}
    print("Server Data Dict (intermediate):", data_dict)
    new_data_list = list(data_dict.items())
    print("Server Data List (output):", data_list)
    return data_list

Output

> Client Data Dict (input): {1: 'one', 2: 'two'}
> Client Data List (intermediate): [(1, 'one'), (2, 'two')]
> Server Data List (input): [[1, 'one'], [2, 'two']]
> Server Data Dict (intermediate): {1: 'one', 2: 'two'}
> Server Data List (output): [[1, 'one'], [2, 'two']]
> Client Data Dict (output): {1: 'one', 2: 'two'}

No big deal since both are iterable, but I notice that in serialisation Anvil also changes a list of tuples to a list of lists (perhaps because Anvil itself serialises using json?).

7

For anyone stumbling on this thread… I just learnt that json has built-in keywords default= and object_hook= for handling non-serialisable objects from the following tutorial:

8

note that this will only work on the server - json keywords are not currently supported on the client.

9

Aha! Thanks for the warning @stucork.

1 Like
10

Hi everyone - quick addition on getting your dictionary into the Client code when your keys are integers.

I had a list of dict called ‘list_dict_var’

It looked something like:
[{0: ‘Aaron’, 1: ‘Johnson’, 2: ‘a.johnson@example.com’}, {0: ‘A’, 1: ‘Blackhawk’, 2: ‘a.b@example.com’}]

The keys are integers and the values are strings. When I use:

jsonString = json.dumps(list_dict_var)

I get this output which can be sent back to the Client code:

[{“0”: “Aaron”, “1”: “Johnson”, “2”: “a.Johnson@example.com”}, {“0”: “A.J”, “1”: “Johnson”, “2”: “a.j.Johnson@example.com”}]

I wanted to get back to single quotes in my list so I could display the data in a Data Grid. To do this, I used

paylaod = json.loads(jsonString)

This returns:

[{‘0’: ‘Aaron’, ‘1’: 'Johnson, ‘2’: ‘a.j@example.com’}, {‘0’: ‘A.J’, ‘1’: ‘Johnson’, ‘2’: ‘a.j.Johnson@example.com’}]

As you can see, the integers are now strings.

In summary use
a = json.dumps(variable_for_your_list_of_dict)
b = json.loads(a)
b is your list of dictionaries where the keys are now strings.

Hope that helps!

1 Like