Artificial Intelligence

Maximising the value of LLM Chatbots in UK Defence

Robin Riley

Robin Riley

— 2 min read

In a series of blog posts I'll be talking through my academic research into AI in large organisations - undertaken as part of my AI MSc, which I completed in 2025.

In this first post I'll simply summarise my research and the main conclusions. Later, I'll go into more detail and describe how the approach I developed can be generalised to other large organisations.

Overview of the research

In general, Large Language Model (LLM) Chatbots offer the potential for radical improvements in workplace productivity. However deploying such chatbots at scale to your workforce carries cost and potentially risk (for example, of inaccuracy or risks to information security).

Therefore, large organisations have a need to deploy LLM Chatbots in ways that
maximise the value to their users, while minimising risk and cost.

In essence my research examined the value derived from using LLM Chatbots reported by the UK Ministry of Defence workforce, through quantitative research and some AI experimentation, and by drawing upon models for value and behaviour from the field of digital transformation, AI and psychology.

My investigation sought to explore:

  1. The factors which most influence value reported by users
  2. The impact on user value of various targeted interventions; and
  3. the feasibility of deploying a Machine Learning model to predict user value.

The key results

The factors found to be most strongly positively correlated with user value were their attitudes to AI, their AI usage and their level of experience in using AI. A user’s ‘AI Persona’ was also an indicator of likely value, indicating that a Persona-based approach to deployments may be useful.

Four parallel interventions (addressing security, training, access and accuracy respectively) were designed and implemented to increase the value reported by users - typically by addressing some belief that users held about AI - and their effect observed. In all four cases, the user population with an intervention reported higher value than those without; for example, being shown a security briefing helped to address negative attitudes around the security of LLM chatbots.

This suggests that targeted interventions - tailored to a specific workforce - can increase the value of LLM Chatbots for users.

Finally, the responses from the user surveys (around 600 people) were used to develop a practical Machine Learning model for predicting user value among the sampled population.

Experiments with this data showed that a Decision Tree model, based on a simplified version of the user survey with only a few questions, could be useful in identifying which staff will gain most value from LLM Chatbots, and so support a cost-effective deployment.


High-level conclusions

Overall, my research showed that the value of LLM chatbots in large organisations can be understood and then increased through a tailored approach to measurement and interventions. My findings were used to make real-world recommendations about how to increase the value of LLM Chatbot deployments in the UK MOD.

My research also provides a generalisable, high-level framework for maximising the value of generative AI deployments in large organisations.

I'll be publishing more details in coming blog posts.

Read more