Selected links for “sequential decision analytics” and Reinforcement Learning and Stochastic Optimization

Warren B. Powell

I have been accumulating a number of links to pages on “sequential decision analytics” and my new book Reinforcement Learning and Stochastic Optimization.  Also, I have been using “tinyurl” links because they are easier to remember (and I can track traffic), but not everyone can use these links.  This page will list the most interesting resources, including both the “tinyurl” link and the original link.  Please use the “tinyurl” link if you can so I can see which links are attracting the most interest. 

Introduction to the fields of sequential decision problems:

An introduction to the field I am calling “sequential decision analytics”:
Tinyurl link: https://tinyurl.com/sdafield/
Original link: https://castlelab.princeton.edu/sda/

My original “jungle of stochastic optimization” webpage:
Original link: https://castlelab.princeton.edu/jungle/

A discussion that bridges “reinforcement learning” to “sequential decision analytics”:
Tinyurl link: https://tinyurl.com/RLtoSDA/
Original link: https://castlelab.princeton.edu/RLtoSDA/

There is a lot of interest in “reinforcement learning” but a surprising lack of consensus on precisely what this means.  For my discussion of this topic go to
Tinyurl link: https://tinyurl.com/what-is-rl/
Original link: https://castlelab.princeton.edu/what-is-rl/

Books:

The link to the page for Reinforcement Learning and Stochastic Optimization:
Tinyurl link: https://tinyurl.com/RLandSO/
Original link: https://castlelab.princeton.edu/RLSO/

The link to the page for Sequential Decision Analytics and Modeling. This is a companion book for RLSO written for an undergraduate level course (this is a free download, and has a python module for most of the chapters):
Tinyurl link: https://tinyurl.com/sdamodeling/
Original link: https://castlelab.princeton.edu/sdamodeling/

One way to learn this material is to write your own chapters in the same style as the book Sequential Decision Analytics and Modeling.  If you have a student who wants to work in this area, encourage them to write a chapter on a problem of their choosing and add it to:
Tinyurl link: https://tinyurl.com/sdabookpublic/
Original link: https://www.overleaf.com/project/5c8add1d8faacf78eabd5761

The companion book above has a Python module developed for most of the chapters (these are also used in the new RLSO book).  These are available at:
Tinyurl link: https://tinyurl.com/sdagithub/
Original link: https://github.com/wbpowell328/stochastic-optimization

Courses and teaching materials:

Suggested courses based on Reinforcement Learning and Stochastic Optimization:
Tinyurl link: https://tinyurl.com/RLSOcourses/
Original link: https://castlelab.princeton.edu/RLSOcourses/

I am preparing a series of short lectures (using Google docs).  This is a work in progress, but you can access the lectures at:
Tinyurl link: https://tinyurl.com/teachingsda/
Original link: https://docs.google.com/document/d/1bHoHEXGZ3SEtZBeCqdgmU9OGSIkG6NfcOR2nYSd3IfM/edit

If you are thinking of teaching this material, please add your name to the signup list at:
Tinyurl link: https://tinyurl.com/RLSOsignup/
Original link: https://docs.google.com/spreadsheets/d/1mCCLM3E_gNw1I_bRHu3c-HLw-gEqk_xwISqHf1QXY6Q/edit#gid=0

I am preparing a series of short tutorials on topics related to sequential decision problems:
Tinyurl link: https://tinyurl.com/teachingsda/
Original link: https://docs.google.com/document/d/1bHoHEXGZ3SEtZBeCqdgmU9OGSIkG6NfcOR2nYSd3IfM/edit

I taught a course called Optimal Learning to undergraduates at Princeton.  Optimal learning problems are pure learning problems, but arise in a wide range of settings.  There is an online version of the 2nd edition of my book Optimal Learning along with all the lectures at:
Tinyurl link: https://tinyurl.com/optimallearningcourse/
Original link: https://castlelab.princeton.edu/orf-418/

A great way to teach about optimizing over policies is to tune the parameters of a simple PFA policy.  Below is a link to a spreadsheet where students can tune the parameters of a “buy low, sell high” policy for energy storage: 
Tinyurl link: https://tinyurl.com/energystorageoptimization
Original link: https://docs.google.com/document/d/1bHoHEXGZ3SEtZBeCqdgmU9OGSIkG6NfcOR2nYSd3IfM/edit#heading=h.mefhgf3fsks

Short notes about sequential decision analytics

One of the most confused topics in reinforcement learning is state variables (try to find a definition of a state variable in any MDP or RL book).  The webpage “On state variables” discusses this topic:
Tinyurl link: https://tinyurl.com/Onstatevariables/ 
Original link: https://castlelab.princeton.edu/statevariables/

Notes on notation for sequential decision problems:
Tinyurl link: https://tinyurl.com/SDAnotation/
Original link: https://castlelab.princeton.edu/2022/05/06/sdanotation/

Notes on notation for sequential decision problems:
Tinyurl link: https://tinyurl.com/SDAnotation/
Original link: https://castlelab.princeton.edu/2022/05/06/sdanotation/

A discussion of the issue of tunable parameters that arise in any form of policy search:
Tinyurl link: https://tinyurl.com/tunableparameters/
Original link: https://castlelab.princeton.edu/2022/04/30/tunableparameters/

A quick introduction to the four classes of policies that I compiled from a series of posts on LinkedIn:
Tinyurl link: https://tinyurl.com/FourClassesofPolicies/
Original link: https://docs.google.com/document/d/1YqCgvraULSpKXFme6HXdKl2uVlp2Nkl8rc4x530PUGo/edit#heading=h.egpzjlkwh66q

This is a paper aimed at a transportation audience that provides a review of the universal framework, and then focuses on two forms of direct lookahead approximations (DLA) that tend to be useful in transportation and logistics: stochastic lookaheads, and parameterized deterministic lookaheads.
Tinyurl link: https://tinyurl.com/PowellLookaheadPolicies/
Original link: https://ieeexplore.ieee.org/abstract/document/9702124

This is a brief discussion on the topic of “languages” for sequential decision problems:
Tinyurl link: https://tinyurl.com/SDAlanguages/
Original link: https://castlelab.princeton.edu/sdalanguages/

Almost entirely overlooked in the research literature, but widely used in practice (in an ad-hoc way) is the idea of creating policies using simplified (typically deterministic) optimization models.  This idea is outlined on the webpage:
Tinyurl link: https://tinyurl.com/CFAPolicy/
Original link: https://castlelab.princeton.edu/CFA/

Some video introductions:

My best video introduction to sequential decision analytics was prepared for a Distinguished Speaker series in supply chain management:
Tinyurl link: https://tinyurl.com/sdafieldyoutube/
Original link: https://www.youtube.com/watch?v=t2P8Fz6xcKM

A four part tutorial that I gave at the Informs Optimization Society conference, March, 2022.  This provides a more in-depth introduction to the modeling framework and the four classes of policies:
Part I:
Tinyurl link: https://tinyurl.com/SDAPartI/
Original link: https://www.youtube.com/watch?v=67UL0Oob_N4&feature=youtu.be

Part II:
Tinyurl link: https://tinyurl.com/SDAPartII/
Original link: https://www.youtube.com/watch?v=Jcuxof7_gW4

Part III:
Tinyurl link: https://tinyurl.com/SDAPartIII/
Original link: https://www.youtube.com/watch?v=fwDC6u8A6S8

Part IV:
Tinyurl link: https://tinyurl.com/SDAPartIV/
Original link: https://www.youtube.com/watch?v=srnuFBsi2xA