Balancing Trust and Deliberation in Human–AI Decision Support: The Effects of Explainable AI and Cognitive Forcing Functions
DOI:
https://doi.org/10.54489/ijcim.v6i1.617Keywords:
Artificial Intelligence, Human-AI Collaboration, Explainable AI, Trust Calibration, Cognitive Load, Healthcare Decision-MakingAbstract
Reliance on AI systems for decision support is expanding into domains where mistakes carry real consequences, which makes it important that users can weigh AI suggestions against their own judgment rather than deferring to them by default. Prior work has found that human-AI teams sometimes underperform AI alone, a pattern usually attributed to automation bias: people continue to follow AI recommendations even when those recommendations are wrong. Most existing systems address this through text-based explanations (XAI), intended to make the AI's reasoning legible to users. Because everyday decision-making leans heavily on fast, intuitive, bias-prone processing, users may skim past these explanations or fail to engage with them in any depth. One proposed remedy is the cognitive forcing function (CFF), a design intervention meant to interrupt automatic responding and prompt more deliberate evaluation. We examine how six such decision-support mechanisms affect engagement, trust, and collaborative task performance in a diabetes meal-planning scenario. In a controlled study with 108 participants (18 per condition), we tested two explanation types, text and visual, alongside four CFFs: AI confidence levels, human feedback, AI-generated questions, and performance visualization. AI confidence levels, text explanations, and performance visualization each significantly improved decision accuracy, and reasoning cues such as text and confidence levels also improved trust. Human feedback and AI-driven questions prompted deeper reflection but appear to have come at a cost to performance, plausibly because of the added cognitive effort and scrutiny involved, which in turn dampened trust. Visual explanations, despite their simplicity, did not meaningfully move trust. These results argue for a contextual, balanced pairing of CFF and XAI design that accounts for interactivity, decision frequency, and task complexity.
References
Ahmed, A., & Ahmad, E. (2023). The future of New Zealand tertiary education: Generation alpha. New Zealand Quality. Research and Innovation.
Al-Sa'di, A. (2022). User centred design methods in animal centred design: A systematic review. In 2022 International Conference on Theoretical and Applied Computer Science and Engineering (ICTASCE). DOI: https://doi.org/10.1109/ICTACSE50438.2022.10009842
Al-Sa'di, A., Ahmad, E., Yamjal, P., & McPhee, C. C. A. (2023b). Strategies to enhance educators' soft skills: The role of managers. In 2023 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE) (pp. 1–6). IEEE. DOI: https://doi.org/10.1109/TALE56641.2023.10398289
Al-Sa'di, A., & Al-Samarraie, H. (2022). A Delphi evaluation of user interface design guidelines: The case of Arabic. Advances in Human-Computer Interaction, 2022, Article 5492230. DOI: https://doi.org/10.1155/2022/5492230
Al-Sa'di, A., & Alsamarraie, H. (2023). Design thinking mindset and self-awareness of user experience practitioners: The case of New Zealand. SSRN. https://doi.org/10.2139/ssrn.4361224 DOI: https://doi.org/10.2139/ssrn.4361224
Al-Sa'di, A., & Khrypko, Y. (2024). User experience for online course strategies: EduFlex framework. In 2024 9th International Conference on Business and Industrial Research (ICBIR) (pp. 1587–1592). IEEE. DOI: https://doi.org/10.1109/ICBIR61386.2024.10875962
Al-Sa'di, A., & Litayem, N. (2023). User frustration: Shaping the experience of automotive and computing. In 2023 IEEE 3rd International Conference on Computer Systems (ICCS) (pp. 56–61). IEEE. DOI: https://doi.org/10.1109/ICCS59700.2023.10335584
Al-Sa'di, A., & Miller, D. (2023). Exploring the impact of artificial intelligence language model ChatGPT on the user experience. International Journal on Technology, Innovation, and Management (IJTIM), 3(1), 1–8. DOI: https://doi.org/10.54489/ijtim.v3i1.195
Al-Sa'di, A., & Miller, D. (2026). Supporting novice creativity in design education through human-centred explainable AI. Theoretical and Applied Ergonomics, 2(2), 4. DOI: https://doi.org/10.3390/tae2020004
Al-Sa'di, A., Yamjal, P., Ahmad, E., Panjabi, R., Allott McPhee, C., & Guler, O. (2023a). Assessing educators' soft skills: Developing a self-assessment instrument. Administrative Sciences, 13(9), 208. DOI: https://doi.org/10.3390/admsci13090208
Al-Sadi, A., Moses, T., & Altawabeyeh, S. (2022). iAWHINA: Towards designing an offline disaster mobile application. In Proceedings of the ISCRAM Asia Pacific Conference 2022.
Al-Sa'Di, A., Puri, S. K., & Panjabi, R. (2025a). Trust, negotiation, and skill acquisition in social commerce: Evidence from Facebook Marketplace. In 2025 6th International Conference on Data Analytics for Business and Industry (ICDABI) (pp. 1–5). IEEE. DOI: https://doi.org/10.1109/ICDABI67967.2025.11547409
Al-Sa'Di, A., Puri, S. K., & Panjabi, R. (2025b). Tracing the emotional pathways of Facebook Marketplace. In 2025 6th International Conference on Data Analytics for Business and Industry (ICDABI) (pp. 1–4). IEEE. DOI: https://doi.org/10.1109/ICDABI67967.2025.11547487
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023). Explainable artificial intelligence (XAI): What we know and what is left to attain trustworthy artificial intelligence. Information Fusion, 99, Article 101805. DOI: https://doi.org/10.1016/j.inffus.2023.101805
Arnold, M., Goldschmitt, M., & Rigotti, T. (2023). Dealing with information overload: A comprehensive review. Frontiers in Psychology, 14, Article 1122200. DOI: https://doi.org/10.3389/fpsyg.2023.1122200
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–16). ACM. DOI: https://doi.org/10.1145/3411764.3445717
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1–21. DOI: https://doi.org/10.1145/3449287
Cao, S., Liu, A., & Huang, C.-M. (2024). Designing for appropriate reliance: The roles of AI uncertainty presentation, initial user decision, and user demographics in AI-assisted decision-making (arXiv:2401.05612). arXiv. https://doi.org/10.48550/arXiv.2401.05612 DOI: https://doi.org/10.1145/3637318
Cohen, J. (2013). Statistical power analysis for the behavioral sciences. Routledge. DOI: https://doi.org/10.4324/9780203771587
Colberg, S. R., Sigal, R. J., Yardley, J. E., Riddell, M. C., Dunstan, D. W., Dempsey, P. C., Horton, E. S., Castorino, K., & Tate, D. F. (2016). Physical activity/exercise and diabetes: A position statement of the American Diabetes Association. Diabetes Care, 39(11), 2065–2079. DOI: https://doi.org/10.2337/dc16-1728
Ellahham, S. (2020). Artificial intelligence: The future for diabetes care. The American Journal of Medicine, 133(8), 895–900. DOI: https://doi.org/10.1016/j.amjmed.2020.03.033
Evert, A. B., Dennison, M., Gardner, C. D., Garvey, W. T., Lau, K. H. K., MacLeod, J., Mitri, J., Pereira, R. F., Rawlings, K., & Robinson, S. (2019). Nutrition therapy for adults with diabetes or prediabetes: A consensus report. Diabetes Care, 42(5), 731–754. DOI: https://doi.org/10.2337/dci19-0014
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127. DOI: https://doi.org/10.1136/amiajnl-2011-000089
Kahneman, D. (2011). Thinking, fast and slow. Farrar, Straus and Giroux.
Kruskal, W. H., & Wallis, W. A. (1952). Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260), 583–621. DOI: https://doi.org/10.1080/01621459.1952.10483441
Lambe, K. A., O'Reilly, G., Kelly, B. D., & Curristan, S. (2016). Dual-process cognitive interventions to enhance diagnostic reasoning: A systematic review. BMJ Quality & Safety, 25(10), 808–820. DOI: https://doi.org/10.1136/bmjqs-2015-004417
Litayem, N., & Al-Sa'di, A. (2023). Exploring the programming model, security vulnerabilities, and usability of ESP8266 and ESP32 platforms for IoT development. In 2023 IEEE 3rd International Conference on Computer Systems (ICCS) (pp. 150–157). IEEE. DOI: https://doi.org/10.1109/ICCS59700.2023.10335558
Logg, J. M., Minson, J. A., & Moore, D. A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90–103. DOI: https://doi.org/10.1016/j.obhdp.2018.12.005
Ma, S., Lei, Y., Wang, X., Zheng, C., Shi, C., Yin, M., & Ma, X. (2023). Who should I trust: AI or myself? Leveraging human and AI correctness likelihood to promote appropriate trust in AI-assisted decision-making. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (pp. 1–19). ACM. DOI: https://doi.org/10.1145/3544548.3581058
McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153–157. DOI: https://doi.org/10.1007/BF02295996
Panjabi, R., Ahmed, A., & Ahmad, E. (2024). Advantage of using AI in education to enhance learning: Review and analysis. In 2024 2nd DMIHER International Conference on Artificial Intelligence in Healthcare, Education and Industry (IDICAIEI) (pp. 1–6). IEEE. DOI: https://doi.org/10.1109/IDICAIEI61867.2024.10842853
Sarsam, S. M., Al-Samarraie, H., Bahar, N., Shibghatullah, A. S., Eldenfria, A., & Al-Sa'Di, A. (2021). Detecting real-time correlated simultaneous events in microblogs: The case of men's Olympic football. In International Conference on Human-Computer Interaction (pp. 368–377). Springer International Publishing. DOI: https://doi.org/10.1007/978-3-030-77277-2_28
Shannon, C. E. (1948). A mathematical theory of communication. The Bell System Technical Journal, 27(3), 379–423. DOI: https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52(3–4), 591–611. DOI: https://doi.org/10.1093/biomet/52.3-4.591
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023). Explanations can reduce overreliance on AI systems during decision-making. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1), 1–38. DOI: https://doi.org/10.1145/3579605
Wilcoxon, F. (1992). Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution (pp. 196–202). Springer. DOI: https://doi.org/10.1007/978-1-4612-4380-9_16
Yamjal, P., & Ahmed, A. (2022). Strategies for retention and completion in vocational education: Faculty perspectives. Journal of Management and Business Education, 5(4), 247–265. DOI: https://doi.org/10.35564/jmbe.2022.0015
Yang, F., Huang, Z., Scholtz, J., & Arendt, D. L. (2020). How do visual explanations foster end users' appropriate trust in machine learning? In Proceedings of the 25th International Conference on Intelligent User Interfaces (pp. 189–201). ACM. DOI: https://doi.org/10.1145/3377325.3377480
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020). Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (pp. 295–305). ACM. DOI: https://doi.org/10.1145/3351095.3372852
Zuchowski, L. C., Zuchowski, M. L., & Nagel, E. (2024). A trust based framework for the envelopment of medical AI. npj Digital Medicine, 7(1), 230. DOI: https://doi.org/10.1038/s41746-024-01224-3
