Re: how to pass data (tuples) to worker processes?

On Sat, Aug 3, 2013 at 6:31 AM, Andrew Tipton <andrew@kiwidrew.com> wrote:
> Robert:  any chance you could share a few more details on the enhancements
> you're planning for bgworkers?  I seem to recall reading that communicating
> with the dynamic bgworkers after they had been launched was next on your
> agenda...

Yeah, it is.  I'm working on a patch to allow additional shared memory
segments to be created on the fly.  The idea I'm working with is that
a backend that plans to launch a worker will first create a dynamic
shared memory segment, then pass the ID of that segment to the worker
via bgw_main_arg.  The worker will map the segment, and then the two
processes can use that to communicate.  My thought is to create a
queue abstraction that sits on top of the dynamic shared memory
infrastructure, so that you can set aside a portion of your dynamic
shared memory segment to use as a ring buffer and send messages back
and forth with using some kind of API along these lines:

extern void dsm_queue_send(dsm_queue *, char *data, uint64 len);
extern uint64 dsm_queue_receive(dsm_queue *, char **dataptr);

It would also be possible to implement message sending and receiving
using pipes, but I'm leaning away from that because it would require
even more OS-dependent code than I'm already having to write, and
writing OS-dependent shim layers is one of the world's less-rewarding
coding tasks; and also because I think it will be easier to achieve
zero-copy semantics using shared memory.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company