Surprise Minimization Under Tyrosine in a Credit Assignment EEG Study

Learning from feedback is an essential form of adaptive behavior. This process is often complicated when the connection between action and outcome is hidden, posing the credit assignment problem. The neuromodulator dopamine, has been robustly implicated in processing reward prediction errors, making it directly relevant to learning from feedback and the embedded credit assignment problem. We sought to replicate earlier work formalizing surprise-minimization as a computational solution to the credit assignment problem (Wurm et al. 2024). This hierarchical model instantiates multiple reinforcement learning policies each reflecting a different hypothesis about the state of the environment. On the lower level, each policy generates reward prediction errors, while on the higher-level, an arbitration process steers action selection toward the policy that minimizes surprise. Given that dopamine is critical for signalling reward prediction errors, we extended prior work by testing...Show moreLearning from feedback is an essential form of adaptive behavior. This process is often complicated when the connection between action and outcome is hidden, posing the credit assignment problem. The neuromodulator dopamine, has been robustly implicated in processing reward prediction errors, making it directly relevant to learning from feedback and the embedded credit assignment problem. We sought to replicate earlier work formalizing surprise-minimization as a computational solution to the credit assignment problem (Wurm et al. 2024). This hierarchical model instantiates multiple reinforcement learning policies each reflecting a different hypothesis about the state of the environment. On the lower level, each policy generates reward prediction errors, while on the higher-level, an arbitration process steers action selection toward the policy that minimizes surprise. Given that dopamine is critical for signalling reward prediction errors, we extended prior work by testing dopaminergic modulation in a double-blind, placebo-controlled EEG study with a dopamine precursor, the naturally occurring amino-acid l-tyrosine. Twenty-six adults completed a modified multiple-bandits task with hidden action-outcome mappings. Overall win rates did not differ between treatment conditions. Unexpectedly, tyrosine slowed reaction times. The surprise-minimization model outperformed simpler reinforcement learning models. Interestingly tyrosine showed marginal evidence for improved model fit. We investigated neural correlates for model proposed quantities using single-trial regressions. Contrary to our hypothesis, we found no dissociable neural signatures for separate policies, failing to replicate this aspect. Replicating prior work, we found a P300 component at centroparietal electrodes tracking the model quantity of policy related evidence. A novel finding was a frontal correlate for model proposed arbitration weight. Critically, we did not find neural differences between tyrosine and placebo. Overall, the results support surprise-minimization as a behavioral account for credit assignment but provide limited neural support. The marginal finding of an improved model fit under tyrosine raises the possibility of higher-level arbitration between different cognitive strategies possibly modulated by dopamine, rather than individual differences in model parameters. This could have implications for understanding pathologies tied to aberrant decision-making rooted in dysregulated dopaminergic processes, such as seen in schizophrenia or addiction. We also discuss methodological limitations that constrain the interpretive power of the study.Show less

Leiden University Student Repository

Documents

In Collections

Surprise Minimization Under Tyrosine in a Credit Assignment EEG Study