[ML] Ornstein Uhlenbeck Process

程式語言：Python
Package：
numpy
namedtuple

簡介：Ornstein Uhlenbeck Process 常用於 DDPG

Xn+1=Xn+θ(μ−Xn)Δt+σΔWnWtn+1−Wtn=ΔWn∼N(0,Δt)=Δt N(0,1)E(Xt)=X0e−θt+μ(1−e−θt)cov(Xs,Xt)=σ22θ(e−θ|t−s|−e−θ|t+s|)
  • 隨機且連續的過程，但長時間來看會徘徊於 μ
  • 參數意義
    • θ：拉回到 mu 的力道，值越大越會在 μ 旁徘徊，也就是漂移越小
    • μ：中心平均值
    • Δt：變化的時間間隔
    • σ：隨機的強度，值越大隨機範圍越大
  • DDPG 可用的參數，習慣範圍落在 [-1, 1] 之間，再乘上動作最大值

程式碼 

  1. import numpy as np
  2. from collections import namedtuple
  3.  
  4.  
  5. class OrnsteinUhlenbeckNoise:
  6.     """
  7.     A Ornstein Uhlenbeck action noise, this is designed to aproximate brownian motion with friction.
  8.  
  9.     Based on http://math.stackexchange.com/questions/1287634/implementing-ornstein-uhlenbeck-in-matlab
  10.  
  11.     :param dim: (tuple) the dimension of the noise
  12.     :param mu: (float) the mean of the noise
  13.     :param theta: (float) the rate of mean reversion, affect converge
  14.     :param sigma: (float) the scale of the noise, affect random
  15.     :param dt: (float) the timestep for the noise
  16.     """
  17.  
  18.     def __init__(self, dim, mu=0, theta=0.15, sigma=0.2, dt=1.0):
  19.         self.dim = dim
  20.         self.mu = mu
  21.         self.theta = theta
  22.         self.sigma = sigma
  23.         self.dt = dt
  24.  
  25.         self.X = np.ones(self.dim) * self.mu
  26.  
  27.     def reset(self):
  28.         self.X = np.ones(self.dim) * self.mu
  29.  
  30.     def __call__(self):
  31.         drift = self.theta * (self.mu - self.X) * self.dt
  32.         random = self.sigma * self._delta_wiener()
  33.  
  34.         self.X = self.X + drift + random
  35.  
  36.         return self.X
  37.  
  38.     def _delta_wiener(self):
  39.         return np.sqrt(self.dt) * np.random.randn(self.dim)
  40.  
  41.  
  42. if __name__ == "__main__":
  43.     Parameter = namedtuple("Parameter", ["theta", "mu", "dt", "sigma"])
  44.     pars = [
  45.         # theta
  46.         Parameter(1, 0, 1, 1),
  47.         Parameter(1e-2, 0, 1, 1),
  48.         # mu
  49.         Parameter(1, 0, 1, 1),
  50.         Parameter(1, 10, 1, 1),
  51.         # dt
  52.         Parameter(1, 0, 1, 1),
  53.         Parameter(1, 0, 1e-2, 1),
  54.         # sigma
  55.         Parameter(1, 0, 1, 1),
  56.         Parameter(1, 0, 1, 1e-2),
  57.         # DDPG
  58.         Parameter(0.15, 0, 1, 0.2),
  59.         Parameter(0.15, 0, 1e-2, 0.3),
  60.     ]
  61.     t = np.linspace(0, 10000, 10000)
  62.     ys = []
  63.     for par in pars:
  64.         noise = OrnsteinUhlenbeckNoise(
  65.             1, mu=par.mu, dt=par.dt, theta=par.theta, sigma=par.sigma
  66.         )
  67.         ys.append([noise() for _ in t])
  68.  
  69.     import matplotlib.pyplot as plt
  70.  
  71.     fig, ax = plt.subplots(nrows=len(ys), sharex=True)
  72.     fig.suptitle(
  73.         "$X_{n+1} = X_n + \\theta (\mu - X_n)\Delta t + \sigma \Delta W_n$", fontsize=16
  74.     )
  75.     for i, y in enumerate(ys):
  76.         ax[i].set_title(
  77.             f"$\\theta={pars[i].theta:.2f}, \mu={pars[i].mu:.2f}, \Delta t={pars[i].dt:.2f}, \sigma={pars[i].sigma:.2f}$"
  78.         )
  79.         ax[i].plot(t, y)
  80.  
  81.     fig.subplots_adjust(top=0.85, hspace=0.9)
  82.     plt.show()

參考

Implementing Ornstein–Uhlenbeck in Matlab
Wiki Ornstein–Uhlenbeck process

留言

張貼留言