configure: Avoid adding flags twice