Re: logging in high performance systems.

On 11/24/2011 06:33 PM, Theo Schlossnagle wrote:
> I see the next steps being:
>  1) agreeing that a problem exists (I know one does, but I suppose
> consensus is req'd)

+1, high volume logging is also a huge problem here at Skype. Some of
the issues that immediately come to mind:

- extracting usage statistics from logs
- extracting error messages from logs
- pushing the logs to remote log server (and no, syslog is not actually
  usable for that ;)

>  2) agreeing that "hooks" are the right approach, if not propose a
> different approach. (fwiw, it's incredible common)

+1 for hook based approach. While the whole logging system could
possibly use an overhaul, this is not likely going to happen for 9.2.
Meanwhile hooks provide the flexibility to implement custom log
collectors for those who really need it.

>  3) reworking the implementation to fit in the project; I assume the
> implementation I proposed will, at best, vaguely resemble anything
> that gets integrated. It was just a PoC.
>

I'd vote for a hook to be added to EmitErrorReport. That way all logged
messages pass through the hook, which could then decide whether to
actually write the message to server log or to discard it. As an added
bonus, the message is  broken down into pieces so there is no need to
parse the messages for severity, context, etc.

Patch attached.

regards,
Martin