Backend Server-Module function returns a "List of NamedTuples", but Frontend caller receives flattened "List of Lists" instead

Anvil Q&A
#1

Hello Friends:

A Data Structure like the following is created and returned to the Frontend via an anvil.server.call() to a Backend Server Module:

Returned:

d = { 'k1' : [NamedTuple(a='a1', b='b1'),
              NamedTuple(a='a2', b='b2'),
              NamedTuple(a='a3', b='b3')],

      'k2' : [NamedTuple(a='c1', b='d1'),
              NamedTuple(a='c2', b='d2'),
              NamedTuple(a='c3', b='d3')],

      'k3' : [NamedTuple(a='e1', b='f1'),
              NamedTuple(a='e2', b='f2'),
              NamedTuple(a='e3', b='f3')], }

Backend testing behaves properly. For example:

d['k2'][1].b does indeed return d2.

However, the Frontend receives a flattened Data Structure like the following (which isn’t subscriptable by attribute as with NamedTuples) :

Received:

d = { 'k1' : ['a1', 'b1'], ['a2', 'b2'], ['a3', 'b3'],
      'k2' : ['c1', 'd1'], ['c2', 'd2'], ['c3', 'd3'],
      'k3' : ['e1', 'f1'], ['e2', 'f2'], ['e3', 'f3'] }

I suspect it’s serialization / deserialization (Marshalling) related, and am wondering if the community can set me straight with this. :slight_smile:

PS: It’s unrelated to the problem, but I’m doing this to save some round-trips.

Thank you!

#2

I would expect for the named tuples to become either tuples or dictionaries, but not to be flattened. (Your received snippet seems to be wrong.)

When you cross the border between client and server, some serialization and conversion goes on, and some types can change.

For example when you send a dictionary from Postgress → Python 3.7 → Skulpt → javascript, at the end you get a dictionary that behaves pretty close to the original, but it is not the original. For example dictionaries have ordered keys since Python 3.6, but they don’t in Postgress, I don’t now about Skulpt and I don’t know if Skulpt has its own definition of a dictionary or relies on the javascript one. So… I don’t rely on dictionary keys being ordered. I obviously still use and abuse dictionaries, because the 99% of the times they behave as expected.

So you can adjust to this limit and work around your named tuples metamorphosis, or you can play with portable classes and make any kind of class portable.

1 Like
#3

@stefano.menci

Thank you for the explanation and suggestion. Let me try wrapping the same code into a portable class right now (trying that as you read this). I glanced over the documentation just now, and It seems simple enough to do; but hey, I keep falling down the rabbit holes on this project. LoL :laughing:

SIDE NOTE: I was going to try the following until you chimed in: Since the source for this data is a Data-Table, I suppose I could make that Data-Table read-only search() able to a specific Form, and iterate through the search results (but now on the Frontend) to populate the exact same NamedTuple Data Structure shown above. But this seems less secure (IMHO), so I’ll prefer the portable class approach first (way better if I can get it work).

Fingers crossed :crossed_fingers: and I’ll let you know. Thanks again.

1 Like
#4

Opening the doors makes prototypes easier to write, but also introduces vulnerabilities.

So, if you are still playing around the idea, making the data table searchable from the form is a good option.

If you are already writing code meant to stay around for the long run, then it’s better to keep the access to data table limited.

2 Likes
#5

I couldn’t agree more. It’s meant to hang around for a while, so I’ll heed your advice!

#6

@stefano.menci

Sadly, refactoring and wrapping as described in the portable classes document resulted in the same outcome. No worse (no exceptions), but no better either (still a List of Lists is received). (Note: Postgres or Data Tables aren’t problematic here. It’s straight Python backend, SerDes, Python frontend).

Back to the drawing board. Onto finding another Data Structure whose form survives marshalling. Maybe @anvil has a suggestion. Thank you.

#7

You can absolutely rely on dictionaries being ordered on the client (in skulpt).

You can’t rely on dictionary ordering being preserved from server to client and back again.

(Side note anvil_labs.kompot supports ordered dictionaries from server to client)

2 Likes
#8

If you could share a clone link or some code here for your portable class then the forum can take a look. It should (and can be made to) work.

1 Like
#9

Hi. Let me cook-up a basic App (with just the relevant parts) and share a clone link. I have never shared a clone link before (sorry I’m new to the platform). Let me work on that and share. Thank you.

#10

No need to email support. Plenty of knowledgeable people here.

If you’re working in the new ide click the triple dots on the publish button at the top and you should find a clone link option. (Or go to the sidebar-> settings-> collaboration and scroll down).

Yeah always best to create a minimal example. Any data in datatables will be cloned so never clone an app with sensitive data.

Clone links are often best. Pasting code snippets can also be sufficient, depending on the nature of the question.

1 Like
#11

@stucork @stefano.menci @anvil and Friends:

Here is a clone link of an App – SerDes Trials – that implements my original post using a portable class. You can just click Run (… Forms.Main is set as the startup Form). Thank you for looking.

https://anvil.works/build#clone:DDNGOSLJQTEFE2IA=4UHBYCYKEYKSHZGCWR7SI6WU

1 Like
#12

First off it’s worth noting that tuples get serialized as lists.
And (for better or worse) subclasses of builtins get serialized in the same way as their builtin parent.

The current attempt in the clone just kicks the can down the road for serializing because you haven’t told Anvil how to serialize the namedtuple class.

Serializing a tuple correctly is slightly different to the documented examples since it’s an immutable data structure.
(You can read about this in the python docs - you need to customize __new_deserialized__ rather than __deserialize__)

# defined in a client module
from collections import namedtuple

_Row = namedtuple('Row', ['a', 'b'])

@anvil.server.portable_class
class Row(_Row):
    def __serialize__(self, global_data):
        return list(self)
    
    @classmethod
    def __new_deserialized__(cls, data, global_data):
        return cls(*data)

You could also turn this into more of a factory function

# defined in a client module
def portable_tuple(_NT):
    class NT(_NT):
        def __serialize__(self, global_data):
            return list(self)
    
        @classmethod
        def __new_deserialized__(cls, data, global_data):
            return cls(*data)
    
    NT.__module__ = _NT.__module__
    NT.__name__ = _NT.__name__
    NT.__qualname__ = _NT.__qualname__

    return anvil.server.portable_class(NT)

And use that factory like

# defined in a client module
Row = portable_tuple(namedtuple('Row', ["a", "b"]))

Hope that helps! There’s quite a lot to get your head around there.

4 Likes
#13

@stucork Thank you for the blow by blow explanation. I’ll study it in the morning (… I’m typing this via a phone because it’s getting late here in Eastern time). I’ll tinker and report back. Thanks for typing that all out!

1 Like
#14

@stucork As you warned, I had to study your solution a bit, but it worked beautifully once implemented. (Thank you). The question, clonable App, and your solution are nicely abstracted, so I’ll keep the clonable App around in case others run into this.

One follow-up question, which applies to any portable class. The below code-snippet resides in a Server Module, where we notice calls to the portable class inside a loop. Would each iteration incur round-trip traffic? Thanks.

# =======================
# In a Server Module
# =======================
from .utils_frontend.anvil_portable_classes import Portable_NamedTuple
from collections import defaultdict

     [ ... snip ... ]

aDict = defaultdict(list)  # Will store In-Mem representation of a Data-Table.
for Row in app_tables.aDataTable.search():
    d = {'k1': Row['k1'], 'k2': Row['k2'], 'k3': Row['k3']}
    aDict[Row['k1']].append(Portable_NamedTuple(**d))

PS: Just as an aside, aDataTable above is a relatively small reference table that never grows or changes; so once the “warm-up” impact is known – of building up aDict and returning it to the Client at app-initialization time, that impact never changes and is amortized (and even saves query-traffic and time) over the life of the App. That’s why all the contortions here. :smiley:

#15

Calling search on the client triggers a round trip that returns the first batch (100?) of rows to the client. Once the rows are on the client, there are no round trips triggered (unless you are not using the new accelerated tables and you are using linked tables or simple object or etc.).

Portable classes are serialized/deserialized only when you use them in a call that explicitly crosses the server/client/uplink border.

That’s nice of you, and it will become unmanageable in the long term :slight_smile:
My clone links in the forum have something like “Clone link in forum” inside the app name. Ugly!

Then I started to clone those apps to my personal free account and put the clone link from there, so I don’t pollute my business account with apps that only exist for those clone links.

3 Likes
#16

That’s a good idea, to use my free account for saving Apps cloned-shared with the community.

During the calls to Portable_NamedTuple(…), I don’t know if theres some Anvil-specific magic behind the import statement to make things run only on the Server-Side. Fortunately, I checked and the loading performance is fine (~3 or 4 seconds to load - including pulling in ~1600 small, static records that never change).

#17

OMG, that was a really mind blowing for me. I was looking for how to make a portable class out of a namedtuple and found this jewel.

1 Like