PDF Rendering not able to work out: server took to long

Anvil Q&A
#1

I want to generate a report as PDF - but doesn’t work because of only 30sec server

Client-side:
I do the calculations before and narrowed time down to 2sec. Then I call the server-function.
Still it is not able to do it. Does PDFRenderer take more than 28sec for about 5-6 pdf.pages?

Server-side:

@anvil.server.callable
def create_pdf_assessment(formtouse, game_event,optimizer,part, data_pack):
  import anvil.pdf
  from anvil.pdf import PDFRenderer
  pdf = PDFRenderer(filename="Assessment-Report.pdf", page_size='A4', scale=0.8).render_form(formtouse, game_event, optimizer, part, data_pack)

Question:
what can I do to speed up rendering?
Is PyPDF faster? Or something else?

Create Any Arbitrarily Large PDF w/ clone link (paid tier features)
#2

Background task can help with anything taking more than 30 seconds.
Does you your pdf need a lot of data from the Table?

#3

This is generally a good advice, but for pdf generation may not be enough.

The pdf is generated by spinning a headless Chrome and rendering a form in it. If I remember correctly, after the pdf has been generated or after 30 seconds the headless Chrome is killed because it’s a memory hog. A background task can last longer than 30 seconds, but the headless Chrome can’t, not even in a background task. Here is a similar discussion.

I use a background task to collect, calculate and serialize all I need, then pass it to the pdf renderer. The first part can take a minute or two, then the rendering of a pdf with up to 20 pages works just fine.

I used to get timeout errors once in a while, but I haven’t seen them in months. I think because of some improvements on the Anvil side that sped up the rendering.

1 Like
#4

Other than the general advice already given, you can show your actual form that’s being used as the basis for the PDF. The code in that form is executed as part of the PDF generation, and can contribute to any slowdown.

#5

Hi,
when I run the form it take 2sec. Have a look:

from ._anvil_designer import A_report_pdfTemplate
from anvil import *
import anvil.tables as tables
import anvil.tables.query as q
from anvil.tables import app_tables
import anvil.server
from .. import _globals
import datetime
from time import localtime, strftime
from Translations import Translations


class A_report_pdf(A_report_pdfTemplate):
  def __init__(self, assessment, optimizer=0, part=0, data_pack=[], **properties):
    # Set Form properties and Data Bindings.
    self.init_components(**properties)
    print('a-report 0', datetime.datetime.now())
    _globals.language = Translations.LOCALE.lower()
    self.label_1.text = "Assessment Report"
    self.item = assessment
    _globals.simcod = self.item['simcod']
    Translations.register_translation(self.label_47, 'text')
    self.label_2.text = strftime("%Y-%m-%d ", localtime())
    data_times = data_pack[0]
    IPS = data_pack[1]
    rots = data_pack[2]
    gelbs = data_pack[3]
    greens = data_pack[4]
    anzahl_gleiche_farbe = data_pack[5]
    keine_extreme = data_pack[6]
    duration = data_pack[7]
    self.plot_all_kpi_polar.config['displayModeBar'] = False
    #get kpi-chain data
    if self.item['kpi_chain']:
      self.repeating_panel_chain.visible = True
      kpi_chain_text = _globals.kpi_chain(self.item, 'assessment', None)
      self.repeating_panel_chain.items = kpi_chain_text
    fig_kpi, fig_str, fig_pro, fig_cul = None, None, None, None
    self.plot_all_kpi_polar.figure = fig_kpi
    self.plot_strategy_polar.figure = fig_str
    self.plot_proz_polar.figure = fig_pro
    self.plot_cult_polar.figure = fig_cul
    print('a-report 1', datetime.datetime.now(),)
    if IPS == 0:
      ips_text = "very low" if self.item['language']=='en' else 'sehr niedrig'
      ips_color = "theme:Rot"
    elif IPS <=8:
      ips_text = "low" if self.item['language']=='en' else 'niedrig'
      ips_color = "theme:Secondary 700"
    elif IPS <= 27:
      ips_text = "average" if self.item['language']=='en' else 'durchschnittlich'
      ips_color = "theme:Gelb"
    elif IPS <=64:
      ips_text = "high" if self.item['language']=='en' else 'hoch'
      ips_color = "theme:Grün"
    elif IPS <= 125:
      ips_text = "very high" if self.item['language']=='en' else 'sehr hoch'
      ips_color = "theme:Grün"
    else:
      ips_text = "excellent" if self.item['language']=='en' else 'exzellent'

    self.button_ips.text = IPS
    self.button_ips.background = '#140c76'
    if self.item['simcod']=='SUSI':
      self.label_ips_is.text = "The Sustainability Power Score of the organisation is "
      self.label_4.text = 'Sustainability Power Score'
    else:
      self.label_ips_is.text = "The Innovation Power Score of the organisation is "
      self.label_4.text = 'Innovation Power Score'
    self.label_ips_x.text = ips_text
    self.label_5.text = "Strategy is about the organisation's ambition for the future and how this is translated into action."
    Translations.register_translation(self.label_ips_is, 'text')
    Translations.register_translation(self.label_ips_x, 'text')
    Translations.register_translation(self.label_5, 'text')
    Translations.register_translation(self.label_6, 'text')
    #Strategie
    print('a-report 2', datetime.datetime.now())
    self.plot_strategy_polar.config['displayModeBar'] = False
    data_times_strat = data_times[0:6]
    data_times_strat.sort(key=lambda item: -item.get('avg'))
    self.repeating_panel_strat.items = data_times_strat
    #Prozess
    self.label_9.text = "Process speaks of efficiency, effectiveness and speed with which the potential for transformation is exploited."
    Translations.register_translation(self.label_8, 'text')
    Translations.register_translation(self.label_9, 'text')
    self.plot_proz_polar.config['displayModeBar'] = False
    print('a-report 3', datetime.datetime.now())
    data_times_proz = data_times[6:12]
    data_times_proz.sort(key=lambda item: -item.get('avg'))
    self.repeating_panel_proz.items = data_times_proz
    #Kultur
    self.label_11.text = "Culture is the necessary human ability, will and permission for transformation to take place."
    Translations.register_translation(self.label_10, 'text')
    Translations.register_translation(self.label_11, 'text')
    self.plot_cult_polar.config['displayModeBar'] = False
    print('a-report 4', datetime.datetime.now())
    data_times_cult = data_times[12:18]
    data_times_cult.sort(key=lambda item: -item.get('avg'))
    self.repeating_panel_cult.items = data_times_cult
    #highest
    print('a-report 4h', datetime.datetime.now())
    data_times_high = data_times
    data_times_high.sort(key=lambda item: -item.get('avg'))
    Translations.register_translation(self.label_12, 'text')
    Translations.register_translation(self.label_13, 'text')
    self.repeating_panel_highest.items = data_times_high[0:4]
    #lowesthighest
    print('a-report 4l', datetime.datetime.now())
    data_times_low = data_times
    data_times_low.sort(key=lambda item: item.get('avg'))
    Translations.register_translation(self.label_14, 'text')
    Translations.register_translation(self.label_15, 'text')
    self.repeating_panel_lowest.items = data_times_low[0:4]
    #größte Übereinstimmung
    print('a-report 4s+', datetime.datetime.now())
    data_times_einheit = data_times
    data_times_einheit.sort(key=lambda item: item.get('std'))
    Translations.register_translation(self.label_16, 'text')
    Translations.register_translation(self.label_17, 'text')
    self.repeating_panel_einheit.items = data_times_einheit[0:4]
    #niedrigste Übereinstimmung
    print('a-report 4s-', datetime.datetime.now())
    data_times_spalt = data_times
    data_times_spalt.sort(key=lambda item: -item.get('std'))
    Translations.register_translation(self.label_18, 'text')
    Translations.register_translation(self.label_19, 'text')
    self.repeating_panel_spalt.items = data_times_spalt[0:4]
    #skipped
    print('a-report 4sk', datetime.datetime.now())
    data_times_skipped = data_times
    data_times_skipped.sort(key=lambda item: -item.get('skipped'))
    Translations.register_translation(self.label_20, 'text')
    Translations.register_translation(self.label_21, 'text')
    self.repeating_panel_skipped.items = data_times_skipped[0:4]

    #Dashboard mit Ampelfarben
    print('a-report 5', datetime.datetime.now())
    board_name = "_/theme/board_"+self.item['simcod'].lower()+"_"+self.item['language'].lower()+".png"
    self.image_board1.source = board_name   # "_/theme/board_susti.jpg"
    self.image_board1.visible = True
    self.xy_panel.background = 'theme:White'
    x_pos = [ 65, 65, 81,111,156,207,390,440,485,515,530,530,437,391,328,269,207,157,298]
    y_pos = [306,251,190,137, 91, 61, 61, 91,137,190,250,306,464,492,509,509,492,464,276]

    data_times.sort(key=lambda item: item.get('nr'))
    #speichere werte der avg in assessment kpi_list für game später
    kpi_list = []
    for element in data_times:
      kpi_list.append([element['identifier'], element['avg']])
    assessment_dict = {'kpi_list': kpi_list}
    if app_tables.assessment.has_row(self.item):
      assessment_dict['updated'] = datetime.datetime.now()
      self.item.update(**assessment_dict)
    else:
      # Raise an exception if the article doesn't exist in the Data Table
      raise Exception("Assessment does not exist")
    #zeige Dashboard
    print('a-report 6a', datetime.datetime.now())
    j, stra_green, proc_green, cult_green = 0,0,0,0
    for x, y, d in zip(x_pos, y_pos, data_times):
      if d['avg'] < 0:
        color = 'theme:Rot'
      elif d['avg'] >= 4:
        color = 'theme:Grün'
        if j<6:
          stra_green +=1
        if j>5 and j<12:
          proc_green +=1
        if j>11:
          cult_green +=1
      else:
        color = 'theme:Gelb'
      if d['std']>2.39:
        color = 'theme:Gray 800'
      j+=1
      self.xy_panel.add_component( Link(icon='fa:circle', font_size=40, tooltip=d['titel'], foreground=color, visible=True), x=x, y=y)
    mittel_wert = (stra_green + proc_green + cult_green) /3
    l_ausgewogen = ((stra_green-mittel_wert)**2 + (proc_green-mittel_wert)**2 + (cult_green-mittel_wert)**2)**0.5

    print('a-report 6b', datetime.datetime.now())

    Translations.register_translation(self.label_22, 'text')
    Translations.register_translation(self.label_23, 'text')
    Translations.register_translation(self.label_24, 'text')
    Translations.register_translation(self.label_25, 'text')
    Translations.register_translation(self.label_26, 'text')
    Translations.register_translation(self.label_27, 'text')
    Translations.register_translation(self.label_28, 'text')

    _globals.data_titel = self.item['assessment_title']
    self.repeating_panel_ist.items = data_times
    print('a-report 6c', datetime.datetime.now())
    self.drop_down_units.visible = True
    self.repeating_panel_wahl.visible = False
    #print('a-report 6d', datetime.datetime.now())  #beide folgende  dauern lang
    #self.show_compare()
    print('a-report 6e', datetime.datetime.now())
    #self.create_plot(data_times, self.item['assessment_title'], data_times, self.item['assessment_title'])
    print('a-report 7', datetime.datetime.now())
    self.label_ips_von_216.text = str(IPS) + " / 216"
    self.label_30.text = self.label_ips_is.text # = "The Innovation Power Score of the organisation is "
    self.label_ips_bewertung.text = self.label_ips_x.text # = ips_text
    self.link_sum_ips.foreground = ips_color
    self.label_31.text = "Agreement"
    if _globals.assessment_avg_all < 1:
      text_avg = "very high"
      allavg_color = 'theme:Grün'
    elif _globals.assessment_avg_all < 1.5:
      text_avg = "high"
      allavg_color = 'theme:Grün'
    elif _globals.assessment_avg_all < 2.2:
      text_avg = "medium"
      allavg_color = 'theme:Gelb'
    elif _globals.assessment_avg_all < 2.8:
      text_avg = "low"
      allavg_color = 'theme:Secondary 700'
    else:
      text_avg = "very low"
      allavg_color = 'theme:Rot'
    self.link_1.foreground = allavg_color
    self.label_32.text = "{:.2f}".format(_globals.assessment_avg_all)
    self.label_33.text = "The assessments between the participants is "
    self.label_34.text = text_avg
    Translations.register_translation(self.label_ips_bewertung, 'text')
    Translations.register_translation(self.label_30, 'text')
    Translations.register_translation(self.label_31, 'text')
    Translations.register_translation(self.label_34, 'text')
    Translations.register_translation(self.label_33, 'text')

    self.label_37.text = "The balance between dimensions is "
    #l_ausgewogen = _globals.ausgewogen
    if l_ausgewogen < 1:
      text_ausgewogen = "high"
      ausgewogen_color = 'theme:Grün'
    elif l_ausgewogen < 2:
      text_ausgewogen = "medium"
      ausgewogen_color = 'theme:Gelb'
    else:
      text_ausgewogen = "low"
      ausgewogen_color = 'theme:Rot'
    self.link_2.foreground = ausgewogen_color
    self.label_36.text = "{:.2f}".format(l_ausgewogen)
    self.label_38.text = text_ausgewogen
    Translations.register_translation(self.label_37, 'text')
    Translations.register_translation(self.label_38, 'text')
    Translations.register_translation(self.label_39, 'text')
    print('a-report 8', datetime.datetime.now())
    self.repeating_panel_qu1.items = app_tables.a_user_assessment_data.search(q.all_of(assessment=self.item, qu1=q.none_of('')))
    self.label_qu1.text = len(self.repeating_panel_qu1.items)
    self.repeating_panel_qu2.items = app_tables.a_user_assessment_data.search(q.all_of(assessment=self.item, qu2=q.none_of('')))
    self.label_qu2.text = len(self.repeating_panel_qu2.items)
    self.repeating_panel_comments.items = app_tables.a_user_assessment_data.search(q.all_of(assessment=self.item, comment=q.none_of('')))
    self.label_40.text = len(self.repeating_panel_comments.items)
    Translations.register_translation(self.label_42, 'text')
    Translations.register_translation(self.label_43, 'text')
    Translations.register_translation(self.label_44, 'text')
    Translations.register_translation(self.label_45, 'text')
    Translations.register_translation(self.label_46, 'text')

    self.label_41same_answer.text = anzahl_gleiche_farbe
    self.label_43avoid_extrems.text = keine_extreme
    self.label_45_duration.text = str("{:.0f}".format(duration[0]/60))+" min. (Max: " + str("{:.0f}".format(duration[2]/60)) + " / Min: " + str("{:.0f}".format(duration[1]/60))
#6

I see that there are some plots in this.
Are you passing a traces to the plots or are you building the plots when the form is rendered?

One trap I have fallen in to is getting the form I am trying to render to PDF to build the plots. This gave a lot of time outs. You can get around it though by passing the complete plots to the form or better yet pass an image of the plot instead of the interactive plotly plot (its going to get PDF’d any way so why not use an image).

#7

Thanks and what a brilliant idea. How do you save the plots (in DB?)

#8

Once you have the image you can save it straight to a data tables media column.

As for getting the image from the plot I have not tried this in Anvil but here is the plotly docs for exporting an image.
https://plotly.com/python/static-image-export/

You might have to use the uplink for this or choose another plotting package.

Realistically though the first step would be to comment out the plot code in your pdf form and confirm that is the reason for the time out.

1 Like
#9

Unfortunately it is not working.
First I have comment out the plot code.
Form construction is down to 1.4 sec.
Should leave enough time for pdf.renderer - but is not working.
Any idea how to increase time on server side? (I try and send an email to anvil - hope they can increase it)

#10

Try to print timestamps when you launch the renderer, when the form loads, etc., then look at the logs and see where it’s spending the time during pdf generation vs preview.

1 Like
#11

Here are the timestamps:

image
image722×663 62.7 KB

start at 17:30:32
Server entry 17:30:35 3sec
pdf.renderer 17:30:44 9sec - thats what takes so long - I have no idea to influence that?!?
report end 17:30:45 1sec
something has to happen afterwards (sum sec so far = 13sec)

#12

Is this including the plots?

I have not yet found a way to speed up the PDF Renderer… this has been problematic for me in the past too. The way I got around it was by making simpler PDFs.

The other (much more painful) way you could do it - depending on the actual layout of the form - is to either construct everything in a Canvas component or construct everything in a word doc on the server and then PDF that. Have you got an example pdf?

I am guessing the something afterwards is probably return the pdf to the client maybe?

#13

It is without plots.
Creating everything in a Canvas for 12 pages is no way (I guess it depends only on a few seconds server-time.)
So the only way seems to be having simpler PDF - what a shame, having a cool tool to work with and than you can not show

@Anvil-Team: maybe you can give more server-time (60sec) or increase it just for PDF-Rendering.

#14

12 pages - yikes! yeah that is not going to work.

Increasing the server timeout will definitely help.

What about creating a page at a time and merging them together?
The merge code is dead simple - I have an app that I built just for that because I got sick of paying for Ad0be.

1 Like
#15

I think this is what we suggested in the past for someone else, I don’t know if they went that path?

#16

I think we may be chasing our tail optimising plots here, because I think @Aaron is saying it’s still a problem with all the plot code commented out!

To confirm, @Aaron: You’ve tried commenting out the plots, and now the form load time is down to 1.5 seconds, and something is still timing out? That suggests that our problem here isn’t just the plots! I suggest leaving the plots switched off (so we know they’re not the problem), then debugging whatever is causing these timeouts. Then you can switch them back on.

First, let’s confirm that it really is the PDF that’s timing out. I’m guessing that AssessmentView, line 147 is the anvil.server.call(...) that calls the server function that renders the PDF, correct? It looks like that server function is timing out before the PDF render times out. I think it probably is the PDF that’s timing out, but let’s make sure - can you do the render in a background task? That way, we can see the PDF timeout error with our own eyes.

Next, let’s take a look at all the things that might happen between your a-report 9 line and the render completing:

  • Where is the a-report 9 print statement, anyway? I don’t see it in your code sample above.

  • Is any code running in the show event handler?

  • What else is on your form, apart from the plots? Are there other components that might be taking a while to load? Can you try trimming those out and see if you can identify a culprit? (Don’t worry, you can use version control to save a version and roll back once you’ve identified the problem!)

1 Like
#17

Thank you Meredydd,

love to have the plots in the report - hope you can solve it.

Some Infos to your questions:

I made several tries and without any of the plots it worked - although time total was above 20sec.
I put in one plot and that works sometimes, more often not - so that should be around 30 sec.
Info:
in the black picture above it shows timestamp of anvil.server.call() from AssessmentView below the red error-message, although that happends before the pdf.renderer

there is no code in show event handler.
a-report 9 line: it is the last time stamp

#18

another info:
I tried background task as well, but resulting in time out as well when background task calls form to render

#19

Aaron - I have been working a relatively complex .pdf generator for some time now - I worked through all of the issues you’re reporting. I use a background task to pull and synthesize ALL of my data. I also use plotly graph objects in the background task to create my plots…

I load all of this into a single dictionary, and I pass the dictionary to my ‘base’ form. Based on values in the dictionary, conditional logic determines which imported forms (pages) are to be included in the report. I only pass the required dictionary key/value pairs to each form.

It works, just know that what you’re doing is delicate no matter what platform you’d be using.

1 Like
#20

Thank you for your points. I hope Meredydd and his team will solve it use it conveniently. Would be a too big thing to generate that work-around.
@meredydd : any progress so far?