After the warm-up. The cost-blind agent is identical but learns as if trading were free; it is still charged for its trades.
Price moves earned by the position held, funding received or paid, and trading costs.
Last 1,500 bars. The position runs from −1 (fully short) to +1 (fully long).
Run the agent at 9 values of the risk aversion \(\lambda\) on the same data. Higher \(\lambda\) penalises variance, so the agent should take smaller positions. Press Risk-aversion sweep.
Gabriel Borrageiro's work with me
With Nick Firoozye and me, Gabriel built trading agents that learn online, by transferring features from one model to another and updating them every bar, in markets where the data never stop changing. All three papers count realistic costs: spreads, fees and funding.
- The recurrent reinforcement learning crypto agent. G. Borrageiro, N. Firoozye, P. Barucca, IEEE Access 10, 38590–38599 (2022). Journal · arXiv · Code Ocean. An echo state network supplies features to a direct recurrent RL agent trading the BitMEX XBTUSD perpetual swap intraday. Over about five years it returns 350% net of costs, with an annualised information ratio of 1.46; 71% of the return comes from funding.
- Reinforcement learning for systematic FX trading. G. Borrageiro, N. Firoozye, P. Barucca, IEEE Access (2021). Journal · arXiv. Features from a radial basis function network with Gaussian-mixture hidden units feed a recurrent RL agent trading 36 currency pairs, updated by an extended Kalman filter. Over a 7-year test set, trading at the 5 pm close when spreads are widest, it earns a 9.3% compound annual return with an information ratio of 0.52, net of execution and funding costs.
- Online learning with radial basis function networks. G. Borrageiro, N. Firoozye, P. Barucca (2021). arXiv. Transferring learned feature representations and updating them sequentially beats a random walk and strong batch-trained models at forecasting returns over several horizons.
What this page does and does not reproduce
The agent, utility, costs and update rules follow the papers. The market is simulated by default (a slowly drifting trend, noise and a funding rate) because the BitMEX order-book data are not public here; you can load your own prices instead. The numbers on this page are therefore illustrations of the mechanism, not the papers' results; the simulated trends are also easier to learn than real ones.
Features: an echo state network
A fixed random recurrent reservoir of 60 tanh units is driven by the standardised return \(u_t\) and funding rate,
$$h_t=(1-a)\odot h_{t-1}+a\odot\tanh\big(W h_{t-1}+W_{\text{in}}u_t\big),$$with \(W\) sparse and rescaled to spectral radius 0.9 (the echo state property), and leak rates \(a_i\) spread between 0.003 and 0.5 so that some units remember the last few bars and others the last several hundred. Only the agent on top of it learns; the reservoir is the transferred representation.
Policy
The position is a recurrent function of the features and of the previous position,
$$f_t=\tanh\big(\theta^\top x_t\big),\qquad x_t=[1,\,h_t,\,f_{t-1}],\qquad -1\le f_t\le 1 .$$Reward
Per bar, in percent of notional,
$$r_t=f_{t-1}\,\Delta p_t-\delta\,|f_t-f_{t-1}|+\kappa_t f_t ,$$where \(\Delta p_t\) is the return of the bar, \(\delta\) the cost per unit traded and \(\kappa_t\) the funding received per unit long (negative of the funding rate, accrued every bar at an eighth of the 8-hour rate).
Utility
The agent maximises a quadratic risk-adjusted utility of exponentially weighted moments,
$$\upsilon_t=\mu_t-\tfrac{\lambda}{2}\sigma_t^2,\quad \mu_t=\tau\mu_{t-1}+(1-\tau)r_t,\quad \sigma_t^2=\tau\sigma_{t-1}^2+(1-\tau)(r_t-\mu_t)^2 .$$Online learning
The gradient follows the position through time (real-time recurrent learning):
$$\frac{d\upsilon_t}{d\theta}=\frac{\partial\upsilon_t}{\partial r_t}\Big(\frac{\partial r_t}{\partial f_t}\frac{df_t}{d\theta}+\frac{\partial r_t}{\partial f_{t-1}}\frac{df_{t-1}}{d\theta}\Big),\qquad \frac{df_t}{d\theta}=(1-f_t^2)\Big(x_t+\theta_f\frac{df_{t-1}}{d\theta}\Big),$$with \(\partial\upsilon_t/\partial r_t=(1-\tau)\big[1-\lambda\tau(r_t-\mu_t)\big]\), \(\partial r_t/\partial f_t=\kappa_t-\delta\,\mathrm{sgn}(\Delta f_t)\) and \(\partial r_t/\partial f_{t-1}=\Delta p_t+\delta\,\mathrm{sgn}(\Delta f_t)\). Stochastic gradient ascent steps \(\theta\leftarrow\theta+\eta\,\nabla\upsilon_t\). The extended Kalman filter of the FX paper instead keeps a matrix \(P\) and, with \(g=\nabla\upsilon_t\),
$$z=1+g^\top P g/\tau,\quad k=Pg/(z\tau),\quad \theta\leftarrow\theta+k,\quad P\leftarrow P/\tau-kk^\top z .$$Gradients are rescaled by a running estimate of the bar volatility so the same learning rate works across markets.
Baselines and statistics
- Buy and hold and always short (which collects funding when it is positive).
- Momentum: the sign of an exponentially weighted average of past returns (half-life about 50 bars).
- Cost-blind agent: the same agent trained with \(\delta=0\), still charged for its trades.
The information ratio is the mean over the standard deviation of bar returns, annualised with \(\sqrt{8760}\) (hourly bars). Turnover is the average \(|\Delta f_t|\) per bar.
The simulated market
Bar returns are \(\Delta p_t=m_t+s_t\varepsilon_t\) with a persistent drift \(m_t=0.995\,m_{t-1}+\sqrt{1-0.995^2}\;\kappa_m s\,\eta_t\), where \(\kappa_m\) is the trend strength, and a stochastic volatility \(s_t\) around 0.6% per hour (about 55% a year). The funding rate per 8 hours follows an AR(1) around its mean with occasional spikes.