Re: [GENERAL] apache logs to pgsql

I did even dirtier job: the cgi parser the log every time people request a
report! -- for the time being, it is used as a prototype and also used
by a few people. so, not a problem.

batch processing is bad when people expect instant real time report.
seems that to parse the log, it depends a lot on the url (get string).
so, we need mod_perl to allow the customization -- that is the apache
side.

seems to me it is totally a apache question, not a pg question. the
parser has to be written in perl with regular exression for easy
development and maintenaince.

On Thu, 20 Jan 2000, Eric van der Vlist wrote:

> Hi,
>
> I am currently using a less elegant -but efficient- solution which is to
> use the standard Apache log module and to post process the log files
> into postgreSQL tables.
>
> The advantage of this method is that you can perform this post
> processing on another server, removing all the strain from the web
> server.
>
> If anyone is interested, I can contribute the scripts I have developed.
> They are tailored to my specific needs but could easily be adapted.
>
> Hope this helps.
>
> Eric
>
> Marc Tardif wrote:
> >
> > In the contrib directory, there is an example on how to redirect apache
> > logs to a postgresql database. It says to create a table and change two
> > lines in the apache configuration file to use psql for copying the
> > logs to the database.
> >
> > My question is: using an sql db is mostly compelling for very active
> > sites but, if this is the case, is it sensible to use psql?
> >
> > First, I find that using the psql command could have too much overhead.
> > Seeing the executable is 46696 bytes and dynamically linked, I'm not
> > really sure what is the strain on the system for bringing up psql.
> >
> > Second, I have tried to see the alternative of writing an apache module
> > or, more sensibly, changing the current module responsible for logging.
> > After reading mod_log_config.c, I'm not really sure how I can integrate
> > libpq and avoid having to connect and disconnect repeatedly. If I can't
> > somehow sustain a connection, I imagine the module would be just as
> > efficient as using psql.
> >
> > Lastly, I'm confused and overwhelmed by the size of the task at hand. I'd
> > appreciate any suggestions or recommendations or thoughts on whether
> > writing a module is even worthwhile at all.
> >
> > Thanks,
> > Marc
> >
> > ************
>
> --
> ------------------------------------------------------------------------
> Eric van der Vlist                                              Dyomedea
>
> http://www.dyomedea.com                          http://www.ducotede.com
> ------------------------------------------------------------------------
>
> ************
>