1 min readfrom Machine Learning

Delay-corrected Bellman operator + causal attribution for constrained RL contraction proof under unknown stochastic delay [R]

Standard constrained RL assumes consequences are immediate and attributable to the current action. This breaks down whenever violations are delayed and stochastic, which is most real-world settings you end up penalizing whatever action happened to precede the observed violation, not the action that caused it.

Working on CCPL (Causal Consequence-Penalized Learning) to address this:

- A delay-corrected Bellman operator using an adaptive effective discount learned from the consequence-delay distribution. Contraction proof holds under unknown stochastic delay.

- An Interventional Consequence Net (ICN), pretrained on structural-causal-model labels, estimating marginal causal contribution per action for attribution rather than penalizing based on temporal proximity.

Limitations, to be upfront about them:
- The ICN currently requires access to the environment's structural causal model to generate pretraining labels it's not learned end-to-end from observational or interventional data alone. That's a real constraint on applicability outside benchmark settings where the SCM is known or can be reasonably specified.

Open to contributions and collaborators, especially if you work in constrained/safe RL or causal inference feel free to open an issue or reach out directly.

submitted by /u/No_Cauliflower7923
[link] [comments]

Want to read more?

Check out the full article on the original site

View original article

Tagged with

#Reinforcement Learning (RL)
#Constrained RL
#Causal Inference
#Causal Attribution
#Delay-Corrected Bellman Operator
#Stochastic Delay
#Interventional Consequence Net (ICN)
#Structural Causal Model (SCM)
#CCPL
#Causal Consequence-Penalized Learning
#Contraction Proof
#Adaptive Discount
#Temporal Proximity
#Marginal Causal Contribution
#Safe RL
#Interventional Data
#Observational Data
#Environment
#Benchmark Settings
#Consequence-Delay Distribution