|| quote_ident(colname) || ' IN (SELECT ' || quote_ident(colname) || ' FROM ' || quote_ident(sel_tblname)
This deletes rows from the table depending upon finding the same value in colname in the subquery.
' where $1 in (select $1 from ' || quote_ident(sel_tblname) ||
This deletes rows from the table depending only upon finding at least one row in the subquery - the limit is basically pointless. Since all rows return the value provided in $1, and $1 = $1.
So these indeed produce different results, as they should. The "error" one is behaving exactly as it should. If it doesn't do what you want - which is likely as it is a very unusual query - don't use it.