Issue applies only on the different selection tool:
- Place Fill Origin
- Set Position of Line build helper
It's still OK that the left mouse is used to confirm the selection, the real problem is that remove actions still get through when using this build selection.
It already prevents build actions from happening, and it should also prevents remove actions.