Re: Caching Python modules

On 17/08/11 14:09, PostgreSQL - Hans-Jürgen Schönig wrote:
> CREATE OR REPLACE FUNCTION textprocess.add_to_corpus(lang text, t text) RETURNS float4 AS $$
> 
>         from SecondCorpus import SecondCorpus
>         from SecondDocument import SecondDocument
> 
> i am doing some intense text mining here.
> the problem is: is it possible to cache those imported modules from function to function call.
> GD works nicely for variables but can this actually be done with imported modules as well?
> the import takes around 95% of the total time so it is definitely something which should go away somehow.
> i have checked the docs but i am not more clever now.

After a module is imported in a backend, it stays in the interpreter's
sys.modules dictionary and importing it again will not cause the module
Python code to be executed.

As long as you are using the same backend you should be able to call
add_to_corpus repeatedly and the import statements should take a long
time only the first time you call them.

This simple test demonstrates it:

$ cat /tmp/slow.py
import time
time.sleep(5)

$ PYTHONPATH=/tmp/ bin/postgres -p 5433 -D data/
LOG:  database system was shut down at 2011-08-17 14:16:18 CEST
LOG:  database system is ready to accept connections

$ bin/psql -p 5433 postgres
Timing is on.
psql (9.2devel)
Type "help" for help.

postgres=# select slow();slow
------

(1 row)

Time: 5032.835 ms
postgres=# select slow();slow
------

(1 row)

Time: 1.051 ms

Cheers,
Jan