MIT 6.231：Dynamic Programming and Stochastic Control（2015 秋）

请帮我把「MIT 6.231：Dynamic Programming and Stochastic Control（2015 秋）」的学习环境一次准备好，让我之后只需要按顺序学习和做练习。
开始前先问我：课程放在哪个目录（等我给出完整路径），确认我的操作系统和已有工具。确认前不要创建文件或安装软件。

课程入口：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/pages/lecture-notes/
我的系统：Windows。如果实际不符，先调整方案。
本次路线：在线学习。只准备这条路线需要的工具；只读路线不安装运行环境、不配置模型密钥。
练习：官方 Assignments、Exams 及 Projects 页提供习题、考试与项目方向；先独立推导动态规划递推，再对照公开解答。 首课是动态规划讲义 PDF 与纸笔练习，不强制安装 Python/Bash。阅读 Lecture 1 后写出状态、决策和阶段代价，并保留个人笔记；外链教材不自动继承 OCW 许可。
课节（官方顺序）：
1. Lecture 1：Introduction to Dynamic Programming Examples of Dynamic Programming Significance of Feedback：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec1/
2. Lecture 2：The Basic Problem Principle of Optimality The General Dynamic Programming Algorithm State Augmentation：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec2/
3. Lecture 3：Deterministic Finite-State Problem Backward Shortest Path Algorithm Forward Shortest Path Algorithm Alternative Shortest Path Algorithms：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec3/
4. Lecture 4：Examples of Stochastic Dynamic Programming Problems Linear-Quadratic Problems Inventory Control：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec4/
5. Lecture 5：Stopping Problems Scheduling Problems Minimax Control：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec5/
6. Lecture 6：Problems with Imperfect State Info Reduction to the Perfect State Info Cas Linear Quadratic Problems Separation of Estimation and Control：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec6/
7. Lecture 7：Imperfect State Information Sufficient Statistics Conditional State Distribution as a Sufficient Statistic Finite-State Analysis：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec7/
8. Lecture 8：Suboptimal Control Cost Approximation Methods: Classification Certainty Equivalent Control Limited Lookahead Policies Performance Bounds Problem Approximation Approach Parametric Cost-To-Go Approximation：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec8/
9. Lecture 9：Rollout Algorithms Cost Improvement Property Discrete Deterministic Problems Approximations to Rollout Algorithms Model Predictive Control (MPS) Discretization of Continuous Time Discretization of Continuous Space Other Suboptimal Approaches：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec9/
10. Lecture 10：Infinite Horizon Problems Stochastic Shortest Path (SSP) Problems Bellman’s Equation Dynamic Programming – Value Iteration Discounted Problems as a Special Case of SSP：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec10/
11. Lecture 11：Review of Stochastic Shortest Path Problems Computation Methods for SSP Computational Methods for Discounted Problems：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec11/
12. Lecture 12：Average Cost Per Stage Problems Connection With Stochastic Shortest Path Problems Bellman’s Equation Value Iteration, Policy Iteration：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec12/
13. Lecture 13：Control of Continuous-Time Markov Chains: Semi-Markov Problems Problem Formulation: Equivalence to Discrete-Time Problems Discounted Problems Average Cost Problems：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec13/
14. Lecture 14：Introduction to Advanced Infinite Horizon Dynamic Programming and Approximation Methods：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec14/
15. Lecture 15：Review of Basic Theory of Discounted Problems Monotonicity of Contraction Properties Contraction Mappings in Dynamic Programming Discounted Problems: Countable State Space with Unbounded Costs Generalized Discounted Dynamic Programming An Introduction to Abstract Dynamic Programming：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec15/
16. Lecture 16：Review of Computational Theory of Discounted Problems Value Iteration (VI) Policy Iteration (PI) Optimistic PI Computational Methods for Generalized Discounted Dynamic Programming Asynchronous Algorithms：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec16/
17. Lecture 17：Undiscounted Problems Stochastic Shortest Path Problems Proper and Improper Policies Analysis and Computational Methods for SSP Pathologies of SSP SSP Under Weak Conditions：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec17/
18. Lecture 18：Undiscounted Total Cost Problems Positive and Negative Cost Problems Deterministic Optimal Cost Problems Adaptive (Linear Quadratic) Dynamic Programming Affine Monotomic and Risk Sensitive Problems：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec18/
19. Lecture 19：Introduction to approximate Dynamic Programming Approximation in Policy Space Approximation in Value Space, Rollout / Simulation-based Single Policy Iteration Approximation in Value Space Using Problem Approximation：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec19/
20. Lecture 20：Discounted Problems Approximate (fitted) VI Approximate PI The Projected Equation Contraction Properties: Error Bounds Matrix Form of the Projected Equation Simulation-based Implementation LSTD, LSPE, and TD Methods：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec20/
21. Lecture 21：Review of Approximate Policy Iteration Projected Equation Methods for Policy Evaluation Simulation-Based Implementation Issues, Multistep Projected Equation Methods Bias-Variance Tradeoff Exploration-Enhanced Implementations, Oscillations：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec21/
22. Lecture 22：Aggregation as an Approximation Methodology Aggregate Problem Simulation-based Aggregation Q-Learning：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec22/
23. Lecture 23：Additional Topics in Advanced Dynamic Programming Stochastic Shortest Path Problems Average Cost Problems Generalizations Basis Function Adaptation Gradient-based Approximation in Policy Space An Overview：https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/resources/mit6_231f15_lec23/

1. 环境
- 使用支持 PDF 的现代浏览器。创建个人笔记目录；本课程阅读不需要安装 Bash、Python 或 GPU。
- 按每课链接打开官方材料；需要学校登录的内容保留链接并标注访问限制，不把无法读取的材料写成已准备。
- 首课是动态规划讲义 PDF 与纸笔练习，不强制安装 Python/Bash。阅读 Lecture 1 后写出状态、决策和阶段代价，并保留个人笔记；外链教材不自动继承 OCW 许可。
- 安装前先列出要装的软件和命令，等我同意；已装好的直接复用。
2. 材料
- 官方课程表与材料：只记录链接 https://ocw.mit.edu/courses/6-231-dynamic-programming-and-stochastic-control-fall-2015/pages/lecture-notes/（仅保存链接；未确认统一下载许可，不批量复制课件或视频。）
3. 目录（最外层放在我确认的位置）
mit-6231-fall2015/
├── README.md        # 课程入口、课节清单、每次怎么开始
├── progress.md      # 每课：读讲义 / 做练习 / 记笔记
├── questions.md     # 待解决的问题
├── check-env.md     # 环境自检，按 README 操作
├── data/            # 练习用到的样例数据
└── lessons/         # 每课一个文件夹：notes.md + exercises/
    ├── 01-lesson-01/
    ├── 02-lesson-02/
    ├── 03-lesson-03/
    ├── 04-lesson-04/
    ├── 05-lesson-05/
    ├── 06-lesson-06/
    ├── 07-lesson-07/
    ├── 08-lesson-08/
    ├── 09-lesson-09/
    ├── 10-lesson-10/
    ├── 11-lesson-11/
    ├── 12-lesson-12/
    ├── 13-lesson-13/
    ├── 14-lesson-14/
    ├── 15-lesson-15/
    ├── 16-lesson-16/
    ├── 17-lesson-17/
    ├── 18-lesson-18/
    ├── 19-lesson-19/
    ├── 20-lesson-20/
    ├── 21-lesson-21/
    ├── 22-lesson-22/
    └── 23-lesson-23/
- 每课文件夹里建 notes.md（顶部写本课官方链接）和 exercises/；课节文件夹名称与上面一致。
- 已有课程目录先检查并复用；不覆盖笔记、作业或 progress.md，不重置已有进度。尚未发布的课节保留入口并注明未发布，不能代写讲义或虚构练习。
4. 自检
- 创建 check-env.md，逐项检查并记录“通过/未通过”：
- 按浏览器或在线 notebook 清单检查，不为自检安装本机终端。
  - 人工检查：打开第一课链接，阅读标题并保存一条自己的学习笔记。  → 官方材料可读，笔记可重新打开；此为人工检查，不作为 shell 命令执行。
- 标注为人工检查、浏览器操作或 notebook 单元的项目，按相应方式操作，不直接粘贴进 shell；脚本只包含当前系统可执行的命令。
- 仅在当前路线需要、对应材料已获准保存、工具已安装时运行该项；需要登录、付费或尚未开放的材料记为“需要我操作/未验证”，不能伪报通过。不执行会调用付费 API、提交作业或修改云服务的检查。
- 运行一次，把结果给我看；未通过的项目说明原因和修复办法。
5. 学习计划
- README.md：课程入口、课节清单（带链接）、每次学习怎么开始（使用“每课提示词”）。
- progress.md：每课三项勾选——读讲义、做练习、记笔记。
6. 最后分三部分汇报：已完成、需要我自己操作的、尚未验证的。

边界：
- 需要管理员权限、会产生费用或需要填写密钥的操作，先告诉我原因，等我同意。密钥由我在本机配置，不发到聊天，不写进笔记或版本库；原项目需要 .env 时确认已被 Git 忽略。
- 不替我写练习或作业答案，也不替我提交。我卡住时只给提示；我明确要求时才给解答。
- 只下载许可允许个人保存的材料；其余只放链接。不把未运行的命令说成成功。
如果你不能直接操作我的文件系统，请在确认目录后给出适用于我的系统、可直接执行的命令和需要保存的文件内容。