Post-hoc explainability methods (e.g., Grad-CAM, Integrated Gradients, LIME, SHAP) are widely adopted in weakly supervised semantic segmentation (WSSS) as pseudo-annotators or seed generators. This paper argues that these visual explanations, while convenient, often produce unreliable and misaligned cues for segmentation. Drawing on experiments with Pascal VOC 2012, CUB-200-2011, and USIS10K, we highlight how saliency-based explanations are frequently inconsistent across methods, focus on incomplete or spurious regions, and show poor correlation with actual object masks or downstream segmentation performance. We demonstrate qualitatively and quantitatively how individual explainers can fail– e.g., highlighting only small discriminative parts of an object or even background artifacts— and how different explainers often disagree with each other on the same image. Using a foundation segmentation model like Segment Anything Model as a proxy for high-quality segmentation, we reveal significant mismatches between explanation maps and object extents. These findings question the trustworthiness of saliency maps as supervision signals for segmentation. We further discuss the gap between interpretability vs. utility: an explanation may faithfully reflect a model’s prediction logic and yet be a poor proxy for the full object region needed in segmentation tasks. We conclude with challenges for the XAI and segmentation communities: Should we blindly trust visual explanation maps for pixel-level supervision? How do we ensure their faithfulness and consistency if repurposed for guiding segmentation models? Can we evaluate explainer quality using downstream segmentation performance? Our position advocates for a principled re-examination of explainer reliability in WSSS, aiming to foster methods that bridge the gap between human-interpretable explanations and effective segmentation cues.