Third Party Log Analysis / ELK - Where to put Config Files?

Anvil Q&A
1

In a previous project I used Rollbar (+ PaperTrail) to aggregate error messages and consolidate messages across a diverse ecosystem.

I am looking to explore doing something similar with my anvil app. While perusing the installed packages I stumbled upon

logzio-python-handler - 2.0.13

And based a quick 30 seconds at logz.io website, their free / community tier will basically address my issue.

It seems the basic integration is configure a the Logz Log Handler, into the standard Python log confirmation.

What is the best place to store such a config ?

I’ve seen a few threads that discourage putting plain text files into the assets folder. Not sure if storing this config file in the database makes sense either (something about accessing the database to create an instance of a logger feels wrong)…

I don’t really know that much about the Anvil runtime, “it just works” for me, so I haven’t tried to understand under the hood, maybe there is a standard place in a YAML file or something I am unaware of.

FWIW, I was basically trying to replicate the getting started information on their website which is based off a plain text config file.

Anyway, any thoughts on best place to configure the python logger would be appreciated.

2

FWIW, I’d like a “Log Service”, similar to Email Service, etc, where you can configure logging options, etc, and know that reasonable Factory and / or singleton patterns are being followed, and can leverage some “Inversion of Control” (aka Dependency Injection) patterns, so that every module does not create and configure an instance of a logger.

My current guess, is that every module / python file, will have to create an instance of the logger, or I’ll have to write my own Log Factory custom Module. But this seems like it could be a common enough pattern for a standard mechanism to be provided by Anvil. (I realize I am ignorant on some of these issues, and therefore my current assumptions could be wrong.)

anyway, I was thinking a standard log configuration / partnership might be useful, similar to the “Stripe” configurations, and there are many log file aggregators (as there are many payment gateways), so having a reasonable default may be useful to future users.

3

This is off the top of my head …

… but you should be able to store a text file in the data tables service as a media blob, then when your app starts, read it back and store it in the “/tmp/myconf.conf” and point the library to it -

logging.config.fileConfig('/tmp/myconf.conf')

The /tmp dir is not guaranteed to survive sessions, but if you only need it briefly then that might work.

Alternatively I think there’s a way you can use bytesIO to read a variable as a file? Bit unsure on that one tbh.

edit - something like this -

my_var = mediablob
my_file = io.BytesIO(my_var)

logging.config.fileConfig(my_file)
4

Picking back up on this, if I want to ship my logs from anvil somewhere for easier consumption and such is that even possible? I would love to have what is in the app logs going somewhere, then I can do alerting on that. The only thing really coming to mind might be something like sentry but that is for more python exception logging and stuff.

Thanks

5

@serpenttracker

Howdy, I am shipping my logs to LogzIO

Only logging from Server Module’s get shipped. The configuration is stored in a dictionary in a shared module.

Then in my server modules I include lines like this

from ServerShared import LOGGING  
logging.config.dictConfig(LOGGING)
root = logging.getLogger()
logger = root

You’ll need to be mindful of flushing the buffer here is a gist I wrote on the topic FYI, I put in a request for the LogzIO team to build an annotation into their code base and it seems they did so in version 2.0.15, but I haven’t started using it.

The gist also shows the dictionary that is stored in the LOGGING variable from the shared server.

I picked LogzIO, because I saw logzio-python-handler in the installed packages list. It seemed to be shortest path to getting something up and running. (I.e. fewer requests to the Anvil team to install python packages, etc)

One Tip, if you use the multiple Anvil apps for DEV / PROD flows, make sure you put the Anvil App Id in your formatter e.g.

        'logzioFormat': {
            'format': '{"additional_field": "value", "ANVIL_APP_ID": "%s"}' % str(anvil.app.id)
        }

Then you can filter by App on LogzIO side for development or production version of the app.

Hope this helps.

Cheers,
Tyler

1 Like
6

You’ll need to be mindful of flushing the buffer

You might be able to set this environment variable to make sure there’s no annoying output buffer:

PYTHONUNBUFFERED=true

In python that would be

import os
os.environ['PYTHONUNBUFFERED'] = True

Haven’t tried it but worth a go - do it once and you can forget about it. Anvil’s Server Modules are python processes running in a carefully-security-engineered *nix environment, so you can do this sort of thing!

3 Likes
7

Look who’s back!
:slight_smile:

2 Likes