Server log analysis limitations

Server logs are heavily under designed currently with lot of limitations on functionalities. And since there are so many limitations, at the least they should give us an ability to download the logs in excel/text so that we can do our own manual analysis of various kinds.

A couple of ideas in ideaboard to cater to these:

Please upvote if you feel connected with these ideas.

Here’s more context on issues that I faced and my thoughts on challenges.

A few limitations that I see:

  • Can’t search by regex or patterns
  • Can’t do any AND / OR in searches
  • It is not clear where all does the “contains” search in. Does it search in data type names? Does it search in session id etc.
  • Can’t search with negative filters e.g. The log doesn’t contain certain text or log is not from certain user etc.
  • Can’t save search criteria in case we want to repeat same searches
  • Can’t see IPs
  • There is no easy way to check one particular session completely
  • There are limited types of events that we can filter. e.g. I can’t see just page loads or just user logins
  • Can’t do any aggregation, filtering, sorting e.g. If I want to see how often people logged in or how often did a particular workflow was called or how many workflow units got consumed by certain set of users etc.
  • “Zooming into a workflow” is not very helpful most often as it shows limited information and then one can’t easily “go back” to where one was at in the logs.
  • Log searches are slow and buggy. Often I’ll just repeat the same search multiple times and either I would get wrong results, or different results or blank screens
  • And of course logs being available for limited duration is a big bummer. A lot of issues are raised by people bit later and it becomes impossible to debug those

I recently had an unexpected capacity spike in our app which caused app to be unusable for many. Now I tried to debug it with logs, capacity charts etc. but just hit the dead-ends with not much showing in logs, pie charts ending in “Other” or “null”

I am confident that I won’t have more than 100-200 users on the app at that time and it was business as usual without any special day which would cause high volume etc, so I was very curious to analyse, and just hit dead-end.

Support team did dig some data on my asking multiple times and shared some info on top ids etc but beyond that they just had their usual “submit this idea to ideaboard, we are sorry” etc.

3 Likes

Not sure why the links above are getting converted to bubble homepage links instead of ideaboard links.

Here are the links again:
https://bubble.io/ideaboard?idea=1683734165029x870499538873876500
https://bubble.io/ideaboard?idea=1668035227231x924689389135069200