Some tips for improving performance are collected here:
My understanding is that a round trip from client to server and back just inherently takes some time. Aside from that, it may help to understand more about what happens when you run a server call: see the “Move slow server imports inside the callable function” section of the Wiki post linked above. When you time the server code, are you including the time it takes for the imports?
[p.s. I moved your question to a separate post, since the cause is likely different from the one discussed there.]