Vision-Language Model-Guided Post-Training for Explainable End-to-End Autonomous Driving
Keywords:
vision-language models; end-to-end autonomous driving; post-training; explainability; safety assurance; governance; deployment; socio-technical systemsAbstract
Vision-language models have become a significant force in artificial intelligence because they connect visual perception with natural language reasoning, enabling systems to describe, question, and justify what they observe. This paper examines how vision-language model-guided post-training can be used to improve explainability in end-to-end autonomous driving. End-to-end driving systems learn to map sensor data directly to control outputs, but their opacity creates serious problems for safety assurance, regulatory oversight, and public trust. Post-training with vision-language models offers a way to refine driving policies after initial pretraining by aligning them with language-based explanations, human preferences, and semantic constraints. The paper argues that this approach should not be treated as a narrow model improvement technique. Instead, it must be understood as a socio-technical architecture involving data pipelines, simulation infrastructure, edge and cloud computation, safety monitors, validation regimes, governance structures, and sustainability considerations. The analysis covers the foundations of end-to-end driving, the architecture of vision-language model-guided post-training, explainability as a systems property, robustness and validation, fairness and policy, deployment and infrastructure, and cross-domain comparisons with aviation, medicine, and industrial automation. It also identifies limitations, including hallucination risk, distribution shift, computational cost, unclear liability, and the absence of standardized evaluation for explanation quality. The central conclusion is that vision-language model-guided post-training can make end-to-end autonomous driving more interpretable and governable, but only if explainability, safety, fairness, and sustainability are treated as first-class requirements throughout the system lifecycle rather than as retrospective additions.
References
1. Codevilla, F., Müller, M., López, A., Koltun, V., & Dosovitskiy, A. (2018). End-to-end driving via conditional imitation learning. In Proceedings of the IEEE International Conference on Robotics and Automation (pp. 4693–4700). IEEE.
2. Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., Zhang, X., Zhao, J., & Zieba, K. (2016). End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316.
3. Kim, J., & Canny, J. (2017). Interpretable learning for self-driving cars by visualizing causal attention. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2942–2950). IEEE.
4. Xu, H., Gao, Y., Yu, F., & Darrell, T. (2017). End-to-end learning of driving models from large-scale video datasets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2174–2182). IEEE.
5. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017). CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning (pp. 1–16). PMLR.
6. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (pp. 8748–8763). PMLR.
7. Alayrac, J. B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., ... & Simonyan, K. (2022). Flamingo: A visual language model for few-shot learning. In Advances in Neural Information Processing Systems (Vol. 35, pp. 23716–23736). Curran Associates.
8. Li, J., Li, D., Xiong, C., & Hoi, S. (2022). BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning (pp. 12888–12900). PMLR.
9. Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A. J., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigcerver, J., Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thapliyal, A. V., Bradbury, J., ... & Soricut, R. (2023). PaLI: A jointly-scaled multilingual language-image model. In Proceedings of the International Conference on Learning Representations. OpenReview.
10. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (Vol. 35, pp. 27730–27744). Curran Associates.
11. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (Vol. 30, pp. 4299–4307). Curran Associates.
12. Winner, L. (1980). Do artifacts have politics? Daedalus, 109(1), 121–136.
13. Kendall, A., Hawke, J., Janz, D., Mazur, P., Reda, D., Allen, J. M., Lam, V. D., Bewley, A., & Shah, A. (2019). Learning to drive in a day. In Proceedings of the IEEE International Conference on Robotics and Automation (pp. 8248–8254). IEEE.
14. Amini, A., Rosman, G., Karaman, S., & Rus, D. (2019). Variational end-to-end navigation and localization. In Proceedings of the IEEE International Conference on Robotics and Automation (pp. 8958–8964). IEEE.
15. Sadat, A., Casas, S., Ren, M., Wu, X., Dhawan, P., & Urtasun, R. (2020). Perceive, predict, and plan: Safe motion planning through interpretable semantic representations. In Proceedings of the European Conference on Computer Vision (pp. 414–430). Springer.
16. Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020). nuScenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 11621–11631). IEEE.
17. Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., Vasudevan, V., Han, W., Ngiam, J., Zhao, H., Timofeev, A., Ettinger, S., Krivokon, M., Gao, A., Joshi, A., ... & Anguelov, D. (2020). Scalability in perception for autonomous driving: Waymo Open Dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2446–2454). IEEE.
18. Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013). Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, 32(11), 1231–1237.
19. Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206–215.
20. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete problems in AI safety. arXiv preprint arXiv:1606.06565.
21. Mittelstadt, B., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016). The ethics of algorithms: Mapping the debate. Big Data & Society, 3(2), 1–21.
22. Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), 389–399.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Intelligent Engineering Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.