| Creators: |
Cenk, Gökhan and Engel, Tobias and Kressel, Jonathan and Andersson, Jonas |
| Title: |
From Hype to Evidence: Evaluating LLM Reliability in Supply Chain Management |
| Item Type: |
Conference or Workshop Item |
| Event Title: |
(Proceedings of the) 32nd American Conference on Information Systems (AMCIS) |
| Event Location: |
Reno, NV, USA |
| Event Dates: |
August, 20-22, 2026 |
| Projects: |
TTZ Leipheim |
| Date: |
20 August 2026 |
| Divisions: |
Informationsmanagement |
| Abstract (ENG): |
Large Language Models (LLMs) promise to transform supply chain management (SCM) through improved forecasting and automated decision support (Aggarwal & Davè, 2018). However, generic benchmarks (Maslej et al., 2025) reveal little about domain-specific performance, leaving open whether supply chain managers can trust LLM-derived proposals or whether the field is building on unverified assumptions.
We argue the field needs a dedicated, domain-specific benchmarking approach that accounts for the operational realities of supply chain tasks. Our preliminary work, running repeated forecasting trials with agentic LLM orchestration on a self-hosted infrastructure using CrewAI and Retrieval Augmented Generation (RAG), reveals that LLM-generated forecasts do not outperform traditional algorithmic approaches (Lewis et al., 2020). This confirms that the gap between LLM potential and domain-specific performance exists and demands systematic, rigorous investigation.
We invite the IS community to shape a research agenda for rigorous LLM evaluation in SCM, focusing on dimensions such as accuracy, consistency, contextual fit, and cost efficiency. Configuration choices, including temperature settings, prompt design, and model architecture, play a significant role in
operational outcomes and deserve attention. From a socio-technical perspective (Bostrom & Heinen, 1977), this includes examining how firms, especially small and medium-sized enterprises (SMEs), can build
evaluation capabilities needed for responsible AI adoption in line with data sovereignty requirements. |
| Forthcoming: |
Yes |
| Main areas or research: |
Mobility & Logistics |
| Language: |
English |
| Citation: |
Cenk, Gökhan and Engel, Tobias and Kressel, Jonathan and Andersson, Jonas
(2026)
From Hype to Evidence: Evaluating LLM Reliability in Supply Chain Management.
In: (Proceedings of the) 32nd American Conference on Information Systems (AMCIS), August, 20-22, 2026, Reno, NV, USA.
|
 |
View Item in edit mode (academic staff only) |
