Release their names to the congregation as potential leaders. If someone has an objection, they must first go to the potential elder (Matthew 18). If they feel unsafe, they should take someone with them, possibly an elder. (If they come directly to an elder, then they are violating Matthew 18.)
Either the issue is resolved, or if true the potential elder can withdraw their name.