Http_endpoint [POST] Issues Publishing to Data Table

Anvil Q&A
1

Hello -

Hope all is well. I am new to the Anvil Community and needing help in understanding how best to set up an http_endpoint [“POST”] which logs data to a Data Table.

I am pulling data from a streaming internet infrastructure data source that allows sending to a POST endpoint via a webhook. To be clear, the data source is not “streaming” in the sense that there is always data being pushed. Sometimes I will get one alert every 30 seconds, another day I might only get one alert the entire day.

Right now I am attempting to listen for new alerts via the below endpoint and auto-update my Data Table. I have watched 5-6 alerts in my terminal so i know data are posting. Yet looking at my Data Table I see nothing. I have confirmed that my Data Table has the same columns as shown below and allows the server module read/write access.

Within my streaming data source I provided the public URL for the endpoint to my streaming data source, but I’m not seeing new alerts posting to my Data Table.

@anvil.server.http_endpoint('/new_alert',methods=['POST'])
def internet_stream(**response):
  
  return app_tables.bb_internet.add_row(asn=response.get('asn',None),hostnames=response.get('hostnames', None), 
                            org=response.get('org', None), port=response.get('port', None), 
                            transport=response.get('transport',None), isp=response.get('isp', None), 
                            referring_website=response.get('bb_internet', None).get('options', None).get('hostname', None),
                            module_name=response.get('bb_internet',None).get('module', None), vulns=response.get('opts',None).get('vulns',None),
                            city=response.get('location',None).get('city',None),
                            longitude=response.get('location',None).get('longitude',None), 
                            latitude=response.get('location',None).get('latitude',None), country_code=response.get('location',None).get('country_code',None),
                            country_name=response.get('location',None).get('country_name',None), area_code=response.get('location',None).get('area_code',None),
                            os=response.get('os', None), ip_str=response.get('ip_str',None), timestamp=response.get('timestamp',None), 
                            domains=response.get('domains',None))

In addition to the above code I have also attempted to set up a background task that routinely checks the endpoint, set up a timer in the client-side code to check the endpoint every .5 seconds, etc. In both of those attempts I received a server timeout error.

Appreciate in advance your help with this.

Thanks,

Michael

2

Hello @mrossi and welcome to the forum!

A few things here.

  • have you tried posting to the end point with some fake data using postman to check it works?
  • have you checked your app logs? if it’s failing there will probably be some helpful information there.

other notes

  • Anvil just updated so that you can’t return a row object to an http endpoint. I think you’ll have to convert it to a dict by doing return dict(app_tables.bb_internent.add_row(...))

a couple of code ideas

  • my_dict.get('foo') returns None if it can’t find foo so my_dict.get('foo', None) is redundant

  • response.get('location',None).get('longitude',None) will fail if location is not a key in response… you could instead do: response.get('location',{}).get('longitude')

1 Like
3

Thanks @stucork I’m jumping into the app now to test out those suggestions. I’ll experiment with Postman and reply back with results. Appreciate the feedback.

1 Like
4

I think your response is an empty dictionary.

You should do something like this:

@anvil.server.http_endpoint('/new_alert',methods=['POST'])
def internet_stream(**response):
  bb = anvil.server.request.body.get_bytes()
  data = json.loads(bb)
  return app_tables.bb_internet.add_row(asn=data.get('asn'), ...

Also, try to add some prints so you can check the app log and see what’s in there:

@anvil.server.http_endpoint('/new_alert',methods=['POST'])
def internet_stream(**response):
  print(response)
  bb = anvil.server.request.body.get_bytes()
  data = json.loads(bb)
  print(data)
  return app_tables.bb_internet.add_row(asn=data.get('asn'), ...

The body_json should contain the json payload, but sometimes it doesn’t work, perhaps because my requests don’t have the correct header, so I use the body.

EDIT

A little cleanup:

@anvil.server.http_endpoint('/new_alert', methods=['POST'])
def internet_stream():
  bb = anvil.server.request.body.get_bytes()
  data = json.loads(bb)
  return app_tables.bb_internet.add_row(
      asn=data.get('asn'),
      hostnames=response.get('hostnames'),
      [...]
  )
2 Likes
5

image
image2016×302 18.7 KB

Jackpot! That did it. Thank you @stucork and @stefano.menci. I used Postman and checked error logs and implemented the exact script in the body as @stefano.menci suggested. Worked great. Fake data using Postman shown above.

I greatly appreciate the help on this. Lots of “user error” to fix moving forward! :joy: