Study of robustness and chain-of-thought consistency in RL fine-tuning of VLMs
On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs
AI BriefingReinforcement learning (RL) fine-tuning is extended to vision-language models (VLMs). The study finds that simple text perturbations—misleading captions or incorrect chains of thought (CoT)—significantly reduce model robustness and confidence, with open-source models showing a greater decline. Closed-source models exhibit similar failure patterns, but have stronger robustness and reasoning consistency. Further analysis reveals a trade-off between accuracy and faithfulness: fine-tuning improves benchmark accuracy while eroding CoT reliability and robustness to contextual changes; adversarial training can improve robustness but cannot prevent faithfulness drift. Introducing faithfulness-aware rewards restores alignment between answers and reasoning, but training tends to collapse into shortcut strategies when combined with robustness enhancements. These findings highlight the need to jointly consider correctness, robustness, and faithfulness in visual reasoning.
Source: Apple Machine Learning Research · machinelearning.apple.com