They DO hang back farther when they're not actively carrying pieces to you. They need to be close enough sometimes that you can talk to them. This was hard to balance, but admittedly could be better.
I've seen more than one player adopt the trick of trapping them in a box.
So if I have one helper out while the rest are trapped, he grabs every 4th piece and won't even touch any other piece.
That's true - they don't really know if you're intending them to stay out or be part of building. They don't really even know they're trapped permanently - for all they know it's just a temporary thing, and you're going to break down the wall and let them help, any second now...