How Does Explicit Knowledge Affect Model-Based Decision Making?

What you need to know

• Initial behavior in a two-step decision task was predominantly model-free across healthy individuals and those with OCD or other disorders
• Model-based control emerged only modestly with experience in a minority of participants, and less in those with OCD
• Providing explicit information about task structure strongly increased model-based control similarly across all groups
• Explicit knowledge, rather than experience alone, was the primary determinant of model-based reinforcement learning

Initial behavior is model-free without instructions

The researchers developed a simplified two-step decision task to examine how model-based and model-free reinforcement learning strategies contribute to behavior when participants must learn the task structure through experience alone, without instructions.

In the first session, healthy participants showed strong evidence of model-free behavior:

• Stay probability (likelihood of repeating a choice) was strongly influenced by whether the previous trial was rewarded, regardless of the transition type
• There was no significant effect of transition type or transition-outcome interaction on stay probability
• This pattern is consistent with simple model-free reinforcement of choices that lead to rewards

Importantly, this model-free behavior was evident from the very start of the first session, rather than emerging over time. This suggests participants were using a basic model-free strategy of repeating rewarded actions from their initial interactions with the task.

Modest increase in model-based control with experience

Comparing behavior between sessions 1 and 3 showed some evidence of participants learning about the task structure and shifting towards more model-based strategies:

• Increased influence of transition type and transition-outcome interaction on stay probability
• Faster reaction times for common vs rare transitions
• Reinforcement learning model fits showed increased value learning rates

However, these shifts were relatively modest:

• Model-free influences still dominated behavior for most participants
• Only about 15% of participants showed strong evidence of model-based control by session 3
• The strength of model-based influence correlated with reward rates, suggesting those who learned the structure performed better

This indicates that while some participants were able to learn aspects of the task structure through experience alone, model-free strategies remained predominant for most.

Impaired learning of model-based control in OCD

Compared to healthy controls, participants with OCD showed:

• No significant increase in the influence of transition type or transition-outcome interaction on choices between sessions 1-3
• Increased influence of reward outcomes alone, suggesting enhanced model-free control
• Less evidence of developing model-based strategies through experience

This aligns with previous research showing deficits in model-based control in OCD. However, the impairment was specific to learning from experience, as OCD participants were able to implement model-based strategies similarly to controls when given explicit instructions (see below).

Explicit knowledge increases model-based control

After session 3, some participants were given explicit information about the task structure before completing session 4. This debriefing had dramatic effects:

• Over 50% of previously model-free participants shifted to model-based strategies
• Large increases in the influence of transition type and transition-outcome interaction on choices
• Increased model-based and decreased model-free influences in reinforcement learning model fits

Importantly, these effects were similar across healthy controls, OCD patients, and those with other mood/anxiety disorders. This indicates that all groups were equally able to implement model-based strategies when given explicit knowledge of the task structure.

Conclusions

• Humans rely primarily on model-free reinforcement learning when faced with novel, complex decision environments
• Model-based strategies emerge slowly and unevenly through experience alone
• Explicit instruction about task structure leads to rapid shifts toward model-based control
• OCD is associated with impaired learning of model-based strategies, but not impaired implementation when given explicit knowledge
• Providing clear information about task structure may be an effective way to promote optimal decision-making strategies across clinical and non-clinical populations