A friendly chatbot that answers you with emojis inspires confidence when discussing everyday life. But when faced with a serious medical decision, that same casual tone becomes counterproductive. A study published in Computers in Human Behavior reveals that the trust placed in an AI depends on a precise alchemy between its personality, its moral reasoning, and the stakes of the situation.
What you will learn
- Why a warm chatbot does not inspire trust in high-stakes moral situations
- How an AI’s ethical reasoning — utilitarian or based on strict rules — shapes users’ perception
- What these results imply concretely for the design of AI in health and safety
AI Facing Moral Dilemmas: A New Reality
AI chatbots are no longer limited to drafting emails or summarizing documents. They advise on health, steer professional decisions, sometimes in situations where choices have real consequences for human lives.
In these contexts, users evaluate not only the accuracy of an answer. They also judge whether the machine is capable of reasoning morally — and whether it deserves to be trusted to do so.
447 Participants, Two Personalities, Two Moral Logics
Researchers at Shanghai Jiao Tong University recruited 447 participants, divided into eight groups, each exposed to a different combination of variables.
The experiment unfolded in two phases. First, each participant spent six minutes chatting with a chatbot about their personal stress. Depending on the group, the AI adopted either a warm and encouraging tone, punctuated with emojis, or an analytic and professional style, without frills.
Next, the chatbot faced a moral dilemma inspired by the classic trolley problem: choose between two outcomes, one involving minor injuries, the other potentially fatal consequences. The AI responded either in a utilitarian manner — prioritizing the greater good — or in a deontological manner — adhering to a strict rule such as “do not harm” no matter what.
Consistency Takes Precedence Over Warmth
The central finding of the study dispels the notion that a more friendly chatbot would automatically be more trustworthy. What matters is the coherence between the personality displayed and the moral logic employed.
A warm tone paired with a deontological approach — fixed rules applied without calculation — was well received in low-risk situations. Users perceived it as a stable and reassuring form of benevolence.
But as the stakes rose, the friendly tone became a negative signal. In the face of a potentially life-threatening decision, participants demanded rigorous, explicit reasoning, carried by a professional style that signals competence and responsibility. The conviviality, in this context, sounded like misplaced lightness.
What This Means for AI Design
These results have direct implications for developers of AI systems deployed in medicine, safety, or psychological support.
Designing a single, consistent personality for a chatbot — warm or professional, once and for all — is not sufficient. The architecture must enable the AI to adapt its communication style according to the perceived stakes. An assistant capable of switching from an empathetic register to a deliberative one based on the severity of the situation would be structurally more trustworthy than a chatbot with a uniform tone.
Trust in an AI, the researchers conclude, is not a fixed trait that you program once. It is a dynamic perception shaped with every interaction by the coherence between what the machine says, how it says it, and what the situation requires.
