The order of keys in a dictionary returned from a server function is changing mysteriously

Anvil Q&A
1

What I’m trying to do:

Preserve the order of keys in a list of dictionaries when returned to the client from a server function.

The keys are getting jumbled somehow after being returned to the main form in certain reproducible cases, despite being put in the desired order server side.

I have not been able to find any reason to suspect that my code is the cause, and as far as I understand them, dictionary key order is preserved by default in Python 3.x. I wonder if you’d be able to help me get to the bottom of the mystery.

What I’ve tried and what’s not working:

My app allows the user to specify which metric columns (keys) they want in their csv (built from the list of dictionaries). The columns need to be displayed in the correct order for my users to be able to read them easily, so I want the keys in my list of dictionaries to be in a certain order.

Everything works beautifully most of the time (thank you Anvil!), except in a few certain cases, in which it consistently doesn’t work. Here is the flow of data, with the problem area italicised:

User inputs what they want client side → sent to the server → server gets what is needed via API and put into a list of dictionaries in desired order → [ list of dictionaries returned to client [keys become jumbled] → list of dictionaries sent to csv function ] → converted to csv media object server side → csv media object is returned to client to download, but the column order is unhelpful to the user due to the key jumbling.

When the user specifies certain metric columns to be included (returned in the list of dicts and converted to csv), the order of the keys in the list of dictionaries after it has been returned to client (viewed via print statement immediately after being returned) is jumbled despite being returned in the desired order by the server function - verified by print just before return.

One theory is that the keys become jumbled when a dictionary which has more than a certain number of keys is sent from server to client? In this case it’s about 12.

Code

Client side:

resultsDictList = anvil.server.call(‘main_program’, user_input_list, market, scope, results_per_seed, mode, get_cpc)

Server side:

resultsDictList when printed immediately before returning, has the keys in the desired order

Client side:

print(resultsDictList)

^^ The order of keys changed

I even wrote a server function especially to reorder the keys. It successfully reordered them server side, but when it sent them to the client, they were jumbled again! So it really looks like just being in the client is causing the keys in this list of dictionaries to become jumbled.

2

It’s most likely an issue with the serialization from server to client. I did an experiment with an OrderedDict, and while it returned it didn’t maintain the order either.

Your best bet is to include ordering information in the dictionary and sort on the client side.

1 Like
3

Dictionaries are ordered on both the client and the server but it looks like passing the dictionary from client to server does not guarantee order.

Since portable_classes you can come up with your own serialization and keep the order.

import anvil.server

@anvil.server.portable_class
class odict(dict):
    def __serialize__(self, global_data):
        return list(self.items())
    def __deserialize__(self, data, global_data):
        self.__init__(data)

Here’s proof of concept

https://anvil.works/build#clone:XN3TIGKFYZRMZKFI=FSCEFHEZQ37H2YJRPCZMQBRZ

1 Like
4

@stucork Thank you very much for testing and confirming what is going on.

I had a go at using portable_class but went with another method in the end because I couldn’t work out how to apply it to my app.

In the proof of concept you provided, it looks like the dictionary which the portable class works on is created client side, sent to server, and back to client - am I right here? As far as I can tell, it looks like the dictionary originally needs to be created client side for the whole thing to work? In my app however, the dictionary is created server side. I wasn’t able to work out how to apply the portable_class to it in my case - I may very well be missing something here. I hope I am, as it would be great to be able to use an elegant type of solution like this rather than what I eventually went with. Sending an ordered list of dictionaries created server side to client / vice-versa is something I really want to be able to do easily in a way like the one you proposed, rather than writing a lot of extra lines of code like I ended up doing.

Would you expect the portable_class solution to work under these circumstances / does it already but I didn’t see it?

The solution I went with

I ended up writing client side code which creates a list of the expected column names which will be in the list of dictionaries generated server side based on the checkboxes the user selected, then gets that list of dictionaries (as before) by calling the main server side function, then passes the list of expected column names from the client to the make_csv function server side, and within the make_csv function, build a new list of dictionaries using the list of expected column names passed from the client, before creating the media object and sending it back to client for download.

Client side:

resultsDictList = anvil.server.call('main_program', user_input_list, market, scope, results_per_seed, mode, get_cpc)

 ## Create dict ordering list
columnList = ['Seed', 'Keywords']
...
...
for m in range(len(priorityOrderedMarketList)):
  columnList.append(str(priorityOrderedMarketList[m]) + ' Search Volume')
  if get_cpc == True:
    columnList.append(str(priorityOrderedMarketList[m]) + ' Average CPC')
    columnList.append(str(priorityOrderedMarketList[m]) + ' Competition')
if mode == 'data_only':
  columnList = columnList[1:] 
  correctColumnOrderList = columnList

## CSV download      
self.message_label.text = 'Creating download'
csv_media = anvil.server.call('make_csv', resultsDictList, market, scope, user_input_list[0], mode, correctColumnOrderList)

Server side:

@anvil.server.callable
def make_csv(resultsDictList, market, scope, firstKeyword, mode, correctColumnOrderList):
    import csv
    import random
    print('resultsDict list received from client = ' + str(resultsDictList[0]))
    print('\ncorrectColumnOrderList when received by make_csv = ' + str(correctColumnOrderList))
    
orderedDictList = []
    for d in range(len(resultsDictList)):
      if d == 0:
        print('Dictionary working on = ' + str(resultsDictList[d]))
      sortedTuples = sorted(resultsDictList[d].items(), key=lambda pair: correctColumnOrderList.index(pair[0]))
      sortedDict = dict(sortedTuples)
      orderedDictList.append(sortedDict)
    print('\nresultsDictList converted to orderedDictList = ' + str(orderedDictList[0]))
    ## Creating a new ordered list of dictionaries out of the jumbled resultsDictList sent by the client, using correctColumnOrderList received from client
    keys = orderedDictList[0].keys()
    csvId = random.randint(0,100)
...
...
    with open(csvFilename, 'w', newline='')  as output_file:
      dict_writer = csv.DictWriter(output_file, keys)
      dict_writer.writeheader()
      dict_writer.writerows(orderedDictList)
    csv_media = anvil.media.from_file(csvFilename, 'csv')
    return csv_media

@jsheffstall and @stucork both of your answers helped me to think about the problem in a new way and get to a working solution so thank you both

5

Not necessarily - You can also import client modules to the server. So on the server you could do

from .Module1 import odict

And use the portable class in server code as you need.

Another idea I was thinking was - to take the burden of creating/converting dicts to portable class - you could create a decorators to your server functions to convert returned dictionaries to odicts. (Being careful about nested return values here if you have them).

Similarly you could wrap the anvil.server.call function on the client to clean the args in a similar way.

If those seem useful and you want some help implementing them then let me know.

2 Likes
6

I did not test it, but I wonder if:

from collections import OrderedDict

… if using OrderedDict on both sides of transferring the object would survive the serialization issue?

This is what I used to use before python 3.5 when parsing large xml documents.

Although maybe I don’t really understand what is happening.

7

Wouldn’t work I’m afraid.

OrderedDict is a subclass of dict so unless it’s a registered portable_class with its own __serialize__ method it will get treated as a dict for the purpose of serialization to and from the server.

1 Like
8

Recently, I’ve been seeing that dictionary order appears to be preserved when returned from the server, through serialization.

Can we rely on this? Is dictionary order now preserved through serialization/deserialization?

9

Afraid not @danbolinson

Nothing has changed on this front internally.

If I do the following:

# Server

@anvil.server.callable
def gen_dict():
    import string
    d = dict.fromkeys(string.ascii_lowercase)
    print(d.keys())
    return d

And then use the client console to call this function I get the following output


>>> d = anvil.server.call("gen_dict")
dict_keys(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'])
# printed from server
>>> d.keys()
dict_keys(['y', 'q', 'r', 'v', 'o', 'n', 'w', 'm', 'e', 's', 'l', 'k', 'z', 'g', 'c', 'j', 'h', 'b', 'd', 'f', 't', 'x', 'p', 'i', 'a', 'u'])
# returned to client
10

The fastest (to code, not execution time) way around this I have found (beyond creating a portable class with it’s own __serialize__ method) is just to create an ordered list data structure keeping the key, value pair in one way or another and then rebuild it on the other side.

so instead of
return data_dict

you use
return [(k,v) for k,v in data_dict.items()]

…and then on the other side you just rebuild it to keep the order by using iterable unpacking:

data_dict = {k:v for k,v in data_dict}

Edit:
A few hours later, I remembered the dict() method actually will just happily consume an iterator of packed key / value pairs, so you can just do: data_dict = dict(data_dict) without any fancy comprehensions on the receiving side.
In-fact you could just call it on everything you sent in to the function and dict() would just ignore regular dictionaries. This would make it function similar to an overload, allowing you to pass either a real dict object or any iterator of key/value pairs.
Example:
image

3 Likes
11

And if you want to get experimental there’s the kompot module in anvil_labs.

https://anvil-labs.readthedocs.io/en/latest/guides/modules/kompot.html

Replace anvil.server.call and anvil.server.callable with kompot.call and kompot.callable to preserve dictionary order.

3 Likes
12

Dear Stu,
I heavily use external sql-tables.
I take great care to create proper Sql-Statements with the correct field order on the server side. They are are turned into dicts just fine by the server side functions. It is very frustrating then when in the result from calling these functions the order is not preserved and one has to reorder the many fields again.
I see the different ways above how this problem is addressed.

Just out of interest: What is the actual reason the dictionaries are tumbled in the process? There seems to be no apparent logic and from my obviously naive standpoint it seems it takes more effort to tumble than to just leave the order. Thanks for enlightening me! Franz

13

A dictionary is a tool to manage key/value pairs, not to preserve the keys order. The order of the keys is whatever makes their management most efficient.

The order of keys in a dictionary was not preserved in python until 3.6 (not 100% sure). It is preserved in later versions only as a side effect of creating new algorithms, not intentionally.

So python <3.6 does change the order of the keys and python >=3.6 doesn’t just because they use different algorithms. The same goes with javascript, postgres and any other parties involved in the lifecycle of a dictionary db-python server-some transmission protocol-javascript-python on client. On top of that, add that all these parties may (do) change the way they work across versions.

So, if you need the additional feature of preserving the order of the keys, you need to explicitly work for it, and very likely pay the price in terms of performance.

14

I should add that the “Python” running in the browser is actually Skulpt, a Python-to-JavaScript transpiler, not off-the-shelf CPython. Its dict emulation likely pre-dates CPython 3.6, and might not promise to preserve key order.

15

Stefano, Phil,
Thanks a lot for the insights and sorry about the question. I am not yet quite familiar with the internal workings of Python (one can at least hope!). The jumbling thus occurs because the keys in a dict are hashed and ordered in a way that allows for quick access. I can live with that!
Thanks - and Merry Christmas!

1 Like