Gureckis & Love (2015)
What is reinforcement learning closely related to? (2 things)
Instrumental conditioning and operant conditioning
What is a prediction error?
It’s the difference between what you expected to happen and what actually happened (difference helps you learn)
What brain areas were linked to exploiting known rewards?
Reward related areas (striatum, prefrontal cortex) were more active when people chose options they already knew were good
What is the study of how animals and machines adapt their behavior in order to maximize their reward in an experience?
Reinforcement Learning (RL)
What was Thorndike’s experiment and its 3 key features?
A cat in a puzzle box is trying to escape to get to food. 3 key features are:
1. exploration (the cats finding its way out)
2. exploitation (using previous actions for quicker escape)
3. the goal is complex
How does temporal difference learning explain learning over time?
It says we update our expectations step by step as events happen, not just at the end, so we can start predicting rewards earlier
What brain areas were linked to exploration?
Regions involved in planning and control were more active when people tried new/uncertain options
What is a theoretical framework for understanding how artificial and natural agents make decisions and offer insights to the overall function/objective of adaptive decision making
Computational Reinforcement Learning (CRL)
What does the next state (a distinct situation the agent might be in) in transition probabilities depend on AND what does it not depend on?
Depends on only the current state and action. Does not depend on the full history of actions that led up to that point
What is the ε-greedy algorithm and what is one downside?
it is an explore/exploit strategy that chooses the option associated with the highest value Q(s,a) but chooses randomly from available alternatives.
DOWNSIDE: it continues to explore with probability ε after agent has gained experience in environment
How are dopamine neurons related to prediction errors?
They fire more when rewards are better than expected, don’t change when rewards are expected, and fire less when rewards are missing (like a prediction error signal)
What does this study show about how the brain makes decisions?
It suggests the brain may use different systems (one focused on reward habits and the other helps us take a step back and try new strategies)
What is the softmax rule?
A guide to exploration by expected value and are determined probabilistically on the basis of actions’ relative expective values
What is the optimal policy?
Returns most reward over long term across all possible states of the environment
Why do the authors say dopamine is not the whole story?
Because some brain signals that look like prediction errors may actually reflect attention or other processes, meaning learning is more complex
Why is the exploration exploitation problem important for decision making?
We constantly have to decide whether to stick with something we know works (exploit) or try something new that might be better (explore).
Helps us adapt to changing situations
Why is Q-learning considered an “off-policy” learning algorithm?
The prediction error for the current choice assumes that the agent will choose the best action on the next state rather than following their own current policy
What is one limitation of reinforcement learning as a theory of the brain?
It explains reward based learning well, but doesn’t fully explain things like planning, memories, or learning relationships between events
How did the researchers study this problem using computational models?
They used reinforcement learning models to track how people learned from rewards and predicted which choices were exploring vs exploiting