Left lateral join with for update and skip locked

Hello,

I have a rather big query which should match id's from various tables
together. To be able to use multiprocessing I'm currently using
postgresql 9.5-dev, because of the SKIP LOCKED feature.

SELECT
   start,
   destination,
   ST_Distance(start_geom, end_geom) AS distance_meter
   FROM (
     SELECT id as "start", geom as start_geom
     FROM de_sim_points_start
     WHERE NOT used AND rs = '057700032032'
     ORDER BY RANDOM()
     LIMIT 200
     FOR UPDATE SKIP LOCKED
   ) AS s
LEFT JOIN LATERAL (
     SELECT id as destination, geom as end_geom
     FROM de_sim_points_end
     WHERE NOT used AND rs IN (
       SELECT sk.rs
       FROM de_commuter_kreise ck
         INNER JOIN de_shp_kreise sk
           ON sk.rs = ck.rs AND ST_DWithin((SELECT ST_Union(geom) FROM
de_shp WHERE rs ='057700032032'), sk.geom, 5000)
       UNION
       SELECT cg.rs
       FROM de_commuter_gemeinden cg
         INNER JOIN de_shp_gemeinden sg
           ON sg.rs = cg.rs AND ST_DWithin((SELECT ST_Union(geom) FROM
de_shp WHERE rs = '057700032032'), sg.geom, 5000)
     )
        AND NOT ST_DWithin(geom, start_geom, 2000) AND ST_DWithin(geom,
start_geom, 5000)
     FOR UPDATE SKIP LOCKED
     LIMIT 1
) AS e ON TRUE

What I think the query is doing:
1. It SELECTs the startpoints
2. Then for each row it selects in the lateral join a corresponding endpoint
2.1 skipping already locked endpoints and find the next not lock one
2.2 While it selects this endpoint it is lock due to the FOR UPDATE
3. Result is presented

BUT what happens is, that I get the same endpoint a couple of times. It
seems to me that the lateral join does not evaluate the SKIP LOCKED
right, since this endpoint occurs multiple times.
Is this a bug or a feature?