Documents Live, a web authoring and publishing system
If you see this, something is wrong
Table of contents
First published on Wednesday, Jul 8, 2026 and last modified on Wednesday, Jul 8, 2026 by François Chaplais.
HSBC Business School, Peking University, University Town, Nanshan District, Shenzhen, 518055, China Email
HSBC Business School, Peking University, University Town, Nanshan District, Shenzhen, 518055, China Email
recursive utility, deep learning, dynamic programming, neural network, dynamic models, portfolio choice, strategic asset allocation
Abstract
We propose the first deep learning algorithm, the Certainty Equivalent Learning (CEL) algorithm, for solving high-dimensional discrete-time dynamic programming problems with recursive utility. Dynamic programming with recursive utility is numerically challenging because the recursive utility does not have an explicit representation and the Bellman equation contains a certainty equivalent that is difficult to evaluate. The CEL algorithm learns this certainty-equivalent value directly with neural networks and jointly approximates value functions, policy functions, and certainty-equivalent functions. The CEL algorithm is mesh-free and simulation-based, allowing high-dimensional state and control spaces, and does not rely on Euler equations, first-order conditions, or differentiability of the state transition function. The CEL algorithm also works for dynamic programming problems with expected utility as expected utility is a special case of recursive utility. We apply the CEL to discounted linear exponential quadratic Gaussian control, small-noise robust control, Epstein–Zin DSGE, and multivariate strategic asset allocation problems. Compared with closed-form and VFI-based benchmarks, the CEL delivers accurate value and policy approximations, remains effective in high-dimensional problems, achieves accuracy comparable to VFI in the small-noise robust-control case, and produces out-of-sample Bellman errors and Euler or first-order residuals that are in the range from \( \text{1.0e-4}\) to \( \text{1.0e-3}\) for most problems.
JEL classification: C12, C14, C22, C52, C61, C63, C65, C68, C88, E32, E37, G11, G12
1 Introduction
Dynamic programming problems are central to economics and finance, with applications in stochastic growth, real business cycle, asset pricing, etc. A large literature has formulated and developed numerical methods for such problems; see, for example, [40], [11], [41], and [42]. Traditional grid-based numerical methods suffer from the curse of dimensionality, i.e., the number of grid points grows exponentially with the number of state variables. As a result, grid-based numerical methods are limited to low- or moderate-dimensional problems.
A growing literature has developed deep learning methods for high-dimensional dynamic programming problems with expected utility, using neural networks to approximate value functions, policy functions, and first-order optimality conditions. These methods show that deep learning can be effective for high-dimensional dynamic programming problems with expected utility.
However, many economic and financial applications require preference specifications beyond standard expected utility. Recursive preferences, such as Epstein–Zin and risk-sensitive preferences, provide a flexible framework because they separate risk aversion from intertemporal substitution and allow agents to respond to long-run risk, model uncertainty, and stochastic volatility. Expected utility is nested in this recursive formulation as a special case. When the transformation is linear and the intertemporal aggregator is additive, the certainty-equivalent term reduces to the usual conditional expectation, and the Bellman equation collapses to the standard expected-utility Bellman equation.
Recursive utility has therefore been widely used in asset pricing, long-horizon portfolio choice, robust control, climate policy, and other dynamic decision problems under uncertainty.
Despite their theoretical and practical appeal, discrete-time dynamic programming problems with recursive utility are difficult to solve numerically. First, recursive utility generally has no explicit expression in terms of future states and controls. Unlike expected utility, where the value function can be explicitly represented as the sum of a discounted stream of future utilities, recursive utility is defined implicitly as the fixed point of the Bellman equation. Second, the Bellman equation contains a certainty-equivalent term, which is a nonlinear transformation of the conditional expectation of a nonlinear function of future utility. The certainty equivalent is a high-dimensional nonlinear function of the state and control and is difficult to calculate. Third, because of the nonlinear function in the certainty equivalent, the estimation of the certainty equivalent is more delicate than that in expected-utility models. With expected utility, conditional expectation of the future value can be estimated by average of samples of future values without bias. With recursive utility, however, nonlinear function of the sample average of future values is not an unbiased estimate of the certainty equivalent, which is the nonlinear function of conditional expectation. Fourth, the certainty-equivalent value is a function of state and control. During policy improvement, the algorithm must evaluate certainty-equivalent values at candidate controls that are not optimal instead of only at optimal controls. Fifth, certainty-equivalent learning and policy improvement are tightly coupled. The certainty-equivalent target depends on the future value approximation, the policy update depends on the learned certainty-equivalent value, and the updated policy changes the distribution of future simulated states. These feedback loops can amplify approximation errors across iterations and motivate learning the state-control certainty-equivalent value as a separate object. Section 3.1 discusses these challenges in more detail.
These computational difficulties are compounded by a general limitation of gradient-based deep learning methods. Neural-network training typically involves nonconvex optimization, so the computed solution is not guaranteed to be globally optimal. This makes independent accuracy evaluation especially important. For recursive utility, this issue is particularly relevant because the learned solution must jointly satisfy the Bellman equation, the nonlinear certainty-equivalent relation, and the implied policy optimality conditions. Moreover, to the best of our knowledge, existing deep-learning approaches to recursive utility have not systematically validated learned solutions against closed-form benchmark solutions. This motivates our use of closed-form benchmarks, VFI comparisons, and out-of-sample Bellman errors and Euler or first-order residuals in the numerical evaluation.
This paper proposes the Certainty Equivalent Learning (CEL) algorithm, the first deep learning algorithm for solving high-dimensional discrete-time dynamic programming problems with recursive utility. The key idea is to learn the certainty-equivalent value directly, rather than repeatedly evaluating nested nonlinear expectations during each update. CEL jointly approximates value functions, policy functions, and certainty-equivalent functions using neural networks. This provides a simulation-based and mesh-free approach for nonlinear, stochastic, and high-dimensional decision problems with recursive utility.
This paper makes six main methodological contributions. First, we develop CEL as a deep learning framework that treats the certainty-equivalent value as a separate learnable object. Instead of repeatedly evaluating nested nonlinear expectations during each update, CEL learns the state-control certainty-equivalent function that enters the Bellman equation. This directly addresses the main computational difficulty created by recursive preferences, namely that the future value enters the current value through a nonlinear certainty-equivalent operator.
Second, CEL only requires forward simulation of the discrete-time state transition and does not use Euler equations or first-order conditions as training objectives. Unlike Euler-equation or first-order-condition-based methods, which typically require differentiability of the transition law whenever derivatives of future states enter the optimality conditions, CEL does not backpropagate through the state transition function. The law of motion may be nonlinear, non-Gaussian, non-smooth, high-dimensional, or available only as a simulator. Therefore, CEL can be applied to general discrete-time Markov state evolution.
Third, the paper introduces a modular family of neural architectures for recursive utility. The baseline three-network architecture approximates the value function, the policy function, and the certainty-equivalent function. The four-network architecture further decomposes the certainty-equivalent term into a standard conditional expectation component and a nonlinear risk-adjustment component. The two-network architecture removes the explicit value network and represents the current value implicitly through the policy and certainty-equivalent networks. These architectures allow CEL to adapt the number of learned objects to the numerical difficulty of the model.
Fourth, CEL is mesh-free and simulation-based. The algorithm does not rely on tensor-product grids, interpolation over state-control grids, perturbation around a steady state, problem-specific projection bases, Euler equations, or first-order conditions. As a result, CEL is suitable for high-dimensional discrete-time dynamic programming problems with recursive utility, nonlinear policy functions, high-dimensional state and control variables, and complex stochastic dynamics.
Fifth, the paper evaluates the learned solutions using out-of-sample residual diagnostics computed from the model’s Bellman equation and optimality conditions. Following the convention of using residual-based accuracy measure in numerical dynamic programming, especially the Euler equation residuals [1], we assess the final learned value and policy functions on independently simulated test states. Specifically, we report Bellman errors and Euler or first-order residuals obtained by substituting the learned policy, value, and certainty-equivalent functions into the original Bellman equation and optimality conditions. The conditional expectations in these diagnostics are recomputed using new Monte Carlo samples rather than the training samples. These accuracy measures are therefore out-of-sample diagnostics instead of in-sample ones, and they provide separate evidence on value-function consistency and policy optimality.
Sixth, the CEL algorithm can also be applied for dynamic programming problems with expected utility, as expected utility is a special case of recursive utility when the nonlinear function in the certainty equivalent reduces to linear and the time aggregator function is simple summation.
We further provide numerical evidence across several benchmark problems, including Gaussian linear-quadratic control, small-noise robust control, Epstein–Zin DSGE models with stochastic volatility, and multivariate strategic asset allocation. These applications cover different recursive-utility specifications, model uncertainty, stochastic volatility, and high-dimensional state and control spaces. The numerical results show that CEL can learn stable value and policy approximations, accurately handle certainty-equivalent terms, and provide reliable solutions in models where traditional grid-based methods are computationally expensive or infeasible. In particular, the high-dimensional Gaussian benchmark and the multivariate portfolio-choice application show that the method can scale to state and control dimensions that are difficult for conventional grid-based approaches. In the final evaluation, the reported out-of-sample Bellman errors and Euler or first-order residuals further show that the learned solutions remain consistent with the original Bellman equation and optimality conditions outside the training sample.
To the best of our knowledge, [43] is the only paper that uses deep learning to solve a discrete-time dynamic programming problem with recursive utility. They study an integrated assessment model with Epstein–Zin utility, which has a 12-dimensional state with a 3-dimensional shock and a 3-dimensional control. Their approach builds on deep equilibrium networks [2], using neural networks to approximate policy functions, Lagrange multipliers, and the value function and solving the model by minimizing residuals from a model-specific system of equations.
CEL differs from this approach in several important respects. First, CEL introduces a certainty-equivalent network that directly learns the certainty-equivalent value which is a nonlinear function of the state and control. In contrast, [43] use a 3-dimensional Gauss–Hermite quadrature to calculate the certainty equivalent by integrating against the density of the 3-dimensional shock [3]. This quadrature approach becomes infeasible or inaccurate for shocks with 4 or higher dimension because of curse of dimensionality. Second, CEL does not use Euler equations, first-order conditions, or model-specific residuals as training objectives. Instead, CEL learns the policy by directly maximizing the Bellman objective using the learned certainty-equivalent function. In contrast, [43] learn the policy indirectly by minimizing residuals from a model-specific system of equations, so the policy update depends on the derived system of first-order and other model-specific equations. Third, CEL uses a modular architecture that separates value approximation, policy learning, and certainty-equivalent learning, and updates these objects through an alternating learning procedure. In contrast, [43] use a deep equilibrium network to approximate several model-specific objects jointly, including policies, multipliers, and the value function. Fourth, CEL updates the value function in a separate step by fitting the value network to the Bellman target constructed from the learned policy and certainty-equivalent function. In contrast, [43] include the value function as one output of their deep equilibrium network, together with choice variables and Lagrange multipliers, and discipline it through a value function residual within the joint residual loss rather than through a separate Bellman-consistency objective. Fifth, CEL evaluates the learned value and policy functions using out-of-sample Bellman errors and Euler or first-order residuals across benchmark applications. In contrast, [43] do not report residual-based accuracy errors or compare the learned solution with closed-form or high-accuracy numerical benchmark solutions. Sixth, because CEL does not require multiplier networks, it reduces the number of model-specific objects that must be approximated, whereas [43] explicitly approximate Lagrange multipliers as part of the model-specific system. Seventh, CEL incorporates target networks and delayed policy updates, which substantially improve the stability of the learning process by stabilizing target values and avoiding overly frequent policy updates. In contrast, [43] optimize all these model-specific objects jointly through a single residual loss without these separate stabilization devices. Finally, CEL uses the state transition rule only as a simulator for next-period states and does not require differentiability of the transition function. In contrast, the residual-system construction in [43] requires the transition equations to be differentiable with respect to both state variables and policy variables in order to derive and evaluate the relevant first-order and other model-specific equations.
The rest of the paper is organized as follows. Subsection 1.1 reviews related literature. Section 2 presents the general recursive utility problem and the Bellman equation. Section 3 introduces the CEL algorithm, including the neural-network architectures and training objectives. Section 4 presents the numerical results for discounted linear exponential quadratic Gaussian control, small-noise robust control, Epstein–Zin DSGE, and multivariate strategic asset allocation, and evaluates the accuracy of the learned solutions using closed-form benchmarks, VFI comparisons, out-of-sample Bellman errors, and Euler or first-order residuals. Section 5 concludes.
1.1 Literature Review
This subsection reviews several related strands of literature: recursive utility theory and applications, dynamic programming foundations for recursive preferences, traditional numerical methods for dynamic programming with recursive utility, and machine-learning methods for dynamic programming and equilibrium computation.
The theoretical foundation of recursive utility goes back to [37], [38], and [39]. Recursive preferences have become important in macroeconomics and finance because they disentangle risk aversion from the elasticity of intertemporal substitution. They have been widely used to study the equity premium puzzle [4], long-run risk asset pricing [5], portfolio choice [see, e.g., 6, 7], fiscal policy [8], robust control [see, e.g., 9, 10, 11], and long-horizon policy evaluation. Recursive utility has also been applied to strategic asset allocation with time-varying investment opportunities [see, e.g., 12, 13], integrated assessment models with climate risk [see, e.g., 14, 15], and consumption-investment problems in incomplete markets [see, e.g., 16, 17, 18, 19, 20, 21]. These applications show that recursive preferences are useful for studying dynamic decisions in which long-run risk, uncertainty, and intertemporal substitution play distinct roles.
Recent theoretical work has studied the existence, uniqueness, and dynamic programming foundations of recursive utility. [44] analyze recursive utility in Markov environments with stochastic growth. [45], [46], and [47] provide existence and uniqueness results for recursive utility and asset-pricing models with unbounded states, stochastic volatility, rare disasters, ambiguity, and preference shocks. [48], [49], [50], and [51] further develop dynamic programming methods for recursive preferences. These papers clarify when recursive utility is well defined and when Bellman equations are valid. Our paper builds on this theoretical foundation but focuses on a computational method for solving such models.
Traditional numerical methods provide important tools for dynamic programming and recursive utility models, but in practice they are usually limited to low-dimensional or low- to moderate-dimensional problems. Standard approaches include value function iteration, policy iteration, projection methods, perturbation methods, and Euler-equation methods; see [52], [53], and [54, 55, 56] for general treatments. Accuracy evaluation in this literature often relies on residual-based diagnostics. [1] provides theoretical foundations for using Euler equation residuals to assess numerical solutions by relating the size of Euler residuals to the approximation errors of policy and value functions. In recursive-utility settings, value function iteration and policy iteration have been used by [49], [50], [48], [57], [14], and [15]. Projection-based methods have been applied to portfolio-choice and climate-economy models with recursive preferences [see, e.g., 22, 23, 24, 25]. Perturbation and local approximation methods are common in DSGE models with recursive preferences [see, e.g., 26, 27], while global solution approaches have also been developed for macro-financial applications [see, e.g., 28, 29]. These methods often rely on grids, basis functions, local approximations, or problem-specific structures, and their computational cost grows rapidly with the dimension of the state and control spaces. As a result, they become difficult to apply to high-dimensional recursive utility models.
There is a growing literature that develops machine learning methods for dynamic programming problems with expected utility. These studies use neural networks, Gaussian processes, and simulation-based methods to approximate value functions and policy functions, and to train or evaluate these approximations using model-specific conditions, including Bellman equations, optimality conditions, feasibility restrictions, and other consistency conditions, mainly in expected-utility models.
[58] develop deep learning methods for solving high-dimensional discrete-time finite-horizon dynamic programming problems with expected utility. For representative-agent infinite-horizon models, [59] and [60] use Gaussian process regression to approximate value functions, [61] use neural networks to learn policy functions, and [62] use neural networks to approximate expectation terms in Euler equations. [63] propose a general deep learning framework based on lifetime utility maximization, Euler-equation error minimization, and Bellman-equation error minimization. Related work studies finite-horizon and overlapping-generations models using deep learning to approximate value and policy functions and to minimize residuals from model-specific systems of equations [see, e.g., 30, 2, 31]. [64] and [65] further develop Monte Carlo and Bellman-error-based approaches for dynamic economic models.
Machine learning has also been applied to heterogeneous-agent and continuous-time models. [63] demonstrate deep learning methods in both representative-agent and heterogeneous-agent environments. [66] combine supervised and unsupervised learning using neural networks for value and policy approximation. [67] approximate heterogeneous-agent New Keynesian models with neural networks trained on first-order-condition errors and hard output constraints. In continuous-time heterogeneous-agent settings, [68] and [69] use deep learning methods based on probabilistic formulations and forward-backward stochastic differential equations. Related continuous-time developments in applied mathematics use deep neural networks to solve high-dimensional PDEs, BSDEs, HJB equations, and stochastic control problems [see, e.g., 32, 33, 34, 35, 36]. In continuous-time finance, [70] solve equilibrium models with Epstein–Zin preferences by minimizing Hamilton–Jacobi–Bellman residuals. Their framework is closely tied to continuous-time diffusion or jump-diffusion representations, whereas CEL is designed for discrete-time Bellman equations and only requires forward simulation of next-period states.
A related data-driven and simulation-based literature uses approximate dynamic programming and reinforcement learning to learn value functions, policies, or decision rules from observed or simulated transition data. General references include [71], [72], and [73]. Applications in economics and finance include [74], [75], and [76]. These methods provide flexible tools for learning from transition data, but most of them are designed for expected-utility or time-additive objective functions rather than recursive utility.
Relative to these strands of literature, this paper extends deep learning methods from expected-utility dynamic programming to recursive utility from a certainty-equivalent learning perspective. The key distinction is that CEL directly learns the certainty-equivalent value in the Bellman equation and uses this learned object for policy improvement. This design makes the method mesh-free, simulation-based, and applicable to recursive utility models with nonlinear certainty-equivalent terms, high-dimensional state variables, and general discrete-time stochastic dynamics.
2 The Dynamic Programming Problem with Recursive Utility
We consider a discrete-time infinite-horizon dynamic programming problem with recursive utility. Time is indexed by \( t=0,1,2,…\) . At time period \( t\) , the agent observes a state vector \( s_t\in\mathbb{R}^{n_s}\) and chooses a control vector \( c_t\in\mathbb{R}^{n_c}\) . We write the policy function as
(1)
The control must satisfy the feasibility constraint
(2)
where \( \mathcal{A}(s_t)\) denotes the admissible set of controls conditional on the current state. The state evolves according to
(3)
where \( \psi(\cdot)\) is the transition function and \( z_{t+1}\) is a random shock. Conditional on \( (s_t,c_t)\) , the next state is independent of past states, so the state process is Markovian. Additional history dependence can be incorporated by augmenting the state vector.
We adopt a general recursive utility specification and use \( V(s_t)\) to denote the value function at state \( s_t\) . The key object is the certainty-equivalent value conditional on the current state and control. We define it as
(4)
where \( f\) captures the agent’s attitude toward risk and \( f^{-1}\) is well defined on the relevant range. Thus, \( V_c(s_t,c_t)\) is a state-control certainty-equivalent value: for each current state \( s_t\) and candidate control \( c_t\) , it summarizes the nonlinear conditional certainty-equivalent value generated by the next-period value function.
Recursive utility is then defined by
(5)
where \( w\) is the time aggregator. This notation keeps the state \( s_t\) , the control \( c_t\) , the value function \( V(s_t)\) , the certainty-equivalent value \( V_c(s_t,c_t)\) , and the aggregator \( w\) consistent with the Bellman equation below.
The decision problem is to choose an admissible Markov policy to maximize the initial value,
(6)
The associated Bellman equation is
(7)
where \(V_c(s_t,c_t)\) is the certainty-equivalent value defined in (4), and the next-period state is generated by (3).
Equivalently, using generic state variables \( s\) and \( s'\) , the Bellman equation can be written as
(8)
where
(9)
Here \( s'\) denotes the next-period state generated from the current state \( s\) , the control \( c\) , and the shock realization \( z'\) . Solving the problem amounts to finding a value function, a policy function, and a certainty-equivalent function that jointly satisfy the Bellman equation.
The notation nests several standard specifications in recursive utility and robust control. Risk-sensitive and robust-control formulations commonly use exponential certainty equivalents, as in the risk-sensitive and robust-control literatures [see, e.g., 9, 10, 11, 26]. Kreps–Porteus and Epstein–Zin preferences separate risk aversion from intertemporal substitution and generate power certainty equivalents of continuation utility [see, e.g., 37, 38, 4, 39, 6]. Two important examples can be written as special cases of the same notation. In risk-sensitive control, one commonly uses
(10)
so that the certainty-equivalent value is an exponential certainty equivalent. In Epstein–Zin preferences, for positive certainty-equivalent values,
(11)
and a standard aggregator can be written as
(12)
These examples illustrate that the same Bellman structure covers both exponential risk adjustments and Epstein–Zin certainty-equivalent values.
The key difficulty is the nonlinear certainty-equivalent value \(V_c(s,c)\). The conditional expectation is taken after applying the nonlinear transformation \(f\) to the future value function, and the inverse transformation is then applied to the expectation. This nested nonlinear state-control object is the central quantity that CEL is designed to approximate.
3 The Certainty Equivalent Learning (CEL) Algorithm
This section is organized around the main components of CEL. Section 3.1 explains why nonlinear certainty-equivalent values create a distinct numerical challenge. Section 3.2 introduces the neural-network representations of the value, policy, and certainty-equivalent value. Section 3.3 describes the alternating update strategy. The remaining subsections discuss stabilizing enhancements and then summarize the full training procedure.
3.1 Challenges
Recursive utility changes the computational nature of dynamic programming in a fundamental way. The discussion in the Introduction identifies five related challenges. This subsection explains these challenges in more detail and shows how they motivate the CEL architecture.
The first challenge is that recursive utility generally has no explicit representation in terms of future states and controls. With expected utility, once a policy \( c_t=c(s_t)\) is fixed, the value function can often be represented as the sum of a discounted stream of future utilities generated by that policy. In many benchmark models, this structure either yields an explicit expansion in terms of future states and controls or reduces the problem to a standard conditional-mean regression. With recursive utility, this simplification is generally unavailable. The value function is defined implicitly as the fixed point of the Bellman equation, and the current value depends recursively on the future value function. As a result, the Bellman equation cannot usually be unfolded into a simple discounted sum of future utilities.
The second challenge is that the Bellman equation contains a nonlinear certainty-equivalent term \( V_c(s_t, c_t)= f^{-1} \left( \mathbb{E}_t \left[ f\left(V(s_{t+1})\right) \mid s_t,c_t\right]\right)\) , where \( f\) is a nonlinear function. This term is a nonlinear transformation of the conditional expectation of a nonlinear function of future utility. It is also a function of both the current state and the current control. In high-dimensional models, directly evaluating this state-control certainty-equivalent value is difficult because it requires approximating a nonlinear conditional object over a high-dimensional state-control space. This expression also clarifies how expected utility arises as a special case. If \( f\) is linear and the aggregator over time is additive, the certainty-equivalent term reduces to the ordinary conditional expectation of future value. The numerical difficulty studied here arises when the certainty equivalent is no longer a simple conditional mean.
The third challenge is that finite-sample average is an biased estimate of the certainty equivalent. With expected utility, the future-value term is linear in the conditional expectation, so a sample average of simulated future values is an unbiased estimator of the conditional expectation. With recursive utility, however, the nonlinear transformation makes the plug-in estimator biased in general. Let \( s_{t+1}^{j}, j=1,…, G\) be i.i.d. samples of \( s_{t+1}\) . Define
Although \( \widehat M\) is an unbiased estimator of \( M(s_t,c_t)\) conditional on \( (s_t,c_t)\) , the plug-in estimator \( f^{-1}(\widehat M)\) is generally not unbiased for \( f^{-1}(M(s_t,c_t))=V_c(s_t, c_t)\) , because \( f^{-1}\) is nonlinear. More explicitly,
in general. By a local expansion, the bias is approximately \( \left(f^{-1}\right)''(M(s_t,c_t))\operatorname{Var}_t(\widehat M)/2\) . Hence the bias can become more important when the certainty-equivalent transformation is strongly curved or when the conditional distribution of future values is dispersed, as in high-risk-aversion, risk-sensitive, or robust-control specifications. This is why a naive Monte Carlo treatment of the certainty-equivalent term may be unstable under recursive utility, and this motivates learning the certainty-equivalent value directly. In the expected-utility special case, this finite-sample bias disappears. When \( f^{-1}\) is linear, applying \( f^{-1}\) to the sample mean preserves unbiasedness, so the plug-in estimator remains unbiased for the continuation term.
The fourth challenge is that, during policy improvement, the certainty-equivalent value must be evaluated at candidate controls, including controls that are not optimal, rather than only at the optimal control. The relevant object is not merely a state function; it is a state-control function. To compare alternative controls at the same current state, the algorithm needs the certainty-equivalent value associated with each candidate control. Therefore, a value network evaluated only at future states does not by itself provide all the information needed for the maximization step. This motivates learning the certainty-equivalent value as a separate state-control function, which maps each state-control pair into the corresponding certainty-equivalent value. This is the role of the certainty-equivalent network in the three-network architecture.
The fifth challenge is the tight coupling between certainty-equivalent learning and policy improvement. The certainty-equivalent target depends on the future value approximation, while the policy network is updated using the learned certainty-equivalent value. The updated policy then changes the distribution of future simulated states. As a result, errors in the value approximation can affect certainty-equivalent learning, errors in the certainty-equivalent approximation can affect policy improvement, and the policy update can change the future states on which the next certainty-equivalent targets are constructed. These feedback loops can amplify approximation errors across iterations, especially in models with stochastic volatility, rare events, strong risk sensitivity, or high curvature in preferences. CEL therefore separates the learning problem into functional blocks. The value network approximates the value function, the policy network represents the policy function, and the certainty-equivalent network approximates the nonlinear certainty-equivalent term. In the four-network architecture, the certainty-equivalent value is further decomposed into an expected continuation network and a nonlinear difference network. These architectures should be understood as different decompositions of the Bellman equation under recursive utility, rather than as a literal count of how many neural-network modules are used in a particular implementation.
3.2 Neural Network Framework
CEL represents the objects in the Bellman equation with neural networks and updates them sequentially. Throughout this section, \( s_t\) denotes the current state, \( c_t\) denotes the current control, \( w\) denotes the time aggregator, and \( f^{-1}(\mathbb{E}_t[f(\cdot)])\) denotes the certainty-equivalent transformation. We use the following names consistently. The value network approximates the value function, the policy network represents the policy function, the certainty-equivalent network directly approximates the nonlinear certainty-equivalent value, the expected continuation network approximates the ordinary conditional expectation of next-period value, and the nonlinear difference network approximates the gap between the expected-value component and the certainty-equivalent value. We consider three related architectures. The terms three-network, four-network, and two-network refer to different functional decompositions of the Bellman equation under recursive utility rather than to the literal number of subnetworks used in every application. For example, when a model has several control components, such as consumption and labor or consumption and portfolio weights, these components may be represented by separate policy subnetworks but are treated as one policy-network block in the CEL architecture. The three-network architecture is the baseline specification, the four-network architecture further decomposes the certainty equivalent into an expected continuation component and a nonlinear difference component, and the two-network architecture provides a more compact representation by removing the explicit value network and defining the current value implicitly through the policy and certainty-equivalent networks.
3.2.1 Three-network architecture
The baseline architecture uses one value-network block, one policy-network block, and one certainty-equivalent-network block:
(13)
(14)
(15)
The value network \( V(s_t;\phi)\) approximates the value function, the policy network \( c(s_t;\theta)\) approximates the policy function, and the certainty-equivalent network \( V_c(s_t,c_t;\xi)\) approximates the nonlinear certainty-equivalent value used by the Bellman operator.
3.2.2 Four-network architecture
The four-network architecture separates the certainty-equivalent value into an expected continuation component and a nonlinear difference component:
(16)
(17)
(18)
(19)
Equivalently,
(20)
Here \( V_e\) is the expected continuation network, which learns the ordinary conditional expectation of the future value, while \( D\) is the nonlinear difference network, which learns the Jensen-type gap between the expected-value component and the certainty equivalent. This decomposition is useful when the nonlinear transformation is highly curved.
The sign of \( D\) follows from Jensen’s inequality and the monotonicity of \( f^{-1}\) . Let \( X_{t+1}=V(s_{t+1})\) . If \( f\) is increasing and concave, Jensen’s inequality implies
(21)
Since \( f^{-1}\) is increasing, applying \( f^{-1}\) to both sides gives
(22)
Therefore,
(23)
The same conclusion holds when \( f\) is decreasing and convex, because applying the decreasing inverse \( f^{-1}\) reverses the Jensen inequality. In this case, the four-network implementation can impose non-negativity on \( D\) , for example by using a positive output activation.
Conversely, if \( f\) is increasing and convex, or decreasing and concave, the Jensen inequality is reversed after accounting for the monotonicity of \( f^{-1}\) . Hence,
(24)
3.2.3 Two-network architecture
The two-network architecture removes the explicit value network and keeps only the policy-network block and the certainty-equivalent-network block. The two networks are defined by
(25)
(26)
Here \( V_c(s_t,c_t;\xi)\) is the certainty-equivalent network, and the right-hand side uses the value function \( V\) to describe the population target rather than to define another network. Since there is no independent value network in this architecture, the current value used during training is represented implicitly by
(27)
This specification reduces the number of networks and forces the value used in policy improvement to be consistent with the learned certainty-equivalent function. It is most useful when the certainty-equivalent network is accurate enough to support bootstrapping without an independently fitted value network.
3.3 Alternating Optimization Strategy
CEL updates the neural networks by alternating among certainty-equivalent estimation, policy improvement, and value-function fitting. The precise set of updates depends on whether the implementation uses the three-network, four-network, or two-network architecture.
Three-network architecture. Given samples \( \{(s_t^i,c_t^i,s_{t+1}^{i,j})\}_{i,j}\) , the certainty-equivalent network is trained by
(28)
Given the learned certainty-equivalent function, the policy network is updated by
(29)
and the value network is fitted to the Bellman target:
(30)
Four-network architecture. The four-network version first fits the expected continuation network,
(31)
and then fits the nonlinear difference network,
(32)
The policy and value updates are
(33)
(34)
Two-network architecture. For the two-network version, there is no separate value update. The implicit value is
(35)
Thus, the certainty-equivalent network is trained by replacing the explicit next-period value in (28) with the implicit value \(\widehat V(s_{t+1};\theta,\xi)\). In other words, the update uses the policy network, the certainty-equivalent network, and the implicit value representation in (25)–(27). The policy update remains the same as (29), but the current value is always evaluated through the implicit composition in (35). This compact variant is therefore a two-network counterpart to the three-network CEL scheme.
3.4 Algorithmic Enhancements for Stability and Efficiency
To enhance the numerical robustness and convergence properties of CEL, we introduce several methodological refinements. These improvements address numerical challenges in deep learning for dynamic programming, including instability arising from bootstrapped targets, insufficient exploration in high-dimensional control spaces, and value-estimation errors caused by overly frequent policy updates. The proposed modifications consist of three components: exploratory control perturbation during path simulation, target networks for stabilized value updates, and delayed policy updates for more stable policy improvement.
The subsequent discussion applies to the three-network, four-network, and two-network variants. In the four-network case, the certainty-equivalent value is decomposed as \( V_c=V_e-D\) ; in the three-network case, the same object is learned directly as \( V_c\) ; and in the two-network case, the current value is represented implicitly through \( w(s_t,c(s_t;\theta),V_c(s_t,c(s_t;\theta);\xi))\) . Hence the stabilization devices below can be adapted to all three architectural variants with only minor notational changes.
3.4.1 Exploratory Control Perturbation During Path Simulation
Accurate policy estimation requires not only precise function approximation but also sufficient exploration of the control space. In high-dimensional decision environments, naively following the policy network’s output may lead to premature convergence to local optima or under-exploration of feasible states. To mitigate this issue, we implement a randomized perturbation mechanism that injects structured exploration into the simulation process. This procedure is provided in 1.
The simulation process is designed to cover the relevant regions of the state space. Each simulation epoch begins by sampling initial states \( S_0\) from the estimated stationary distribution of the controlled process, ensuring that our policy evaluation starts from states that are representative of the long-run behavior of the system. This approach provides a more stable foundation for evaluating policy performance compared to fixed or arbitrary initial conditions. For state variables and environmental parameters not explicitly governed by the policy network, we employ a conservative initialization strategy, drawing their values from a predefined uniform distribution over their plausible ranges. This ensures that our exploration covers a diverse set of scenarios while maintaining physiological and economic plausibility.
At each iteration, when we update the network \( V_e(s_t, c_t; \rho)\) and \( D(s_t, c_t; \nu)\) , we simulate the Gaussian path exploration process in reinforcement learning, with a specific focus on adding exploratory noise to the policy actions. The simulation starts from these properly initialized states and generates \( N\) trajectories of length \( T\) . At each time step \( t\) , it first generates the base action using the policy network \( c(s_t; \theta)\) based on the current state. Subsequently it adds a uniformly distributed random perturbation to the action space. This design maintains the policy’s dominance while encouraging the system to explore unknown state regions through targeted exploration. Finally, the state for the next time step is updated according to the transition equation, completing a single iteration. This entire process repeats for \( T\) steps, ultimately outputting the complete trajectories containing both states and actions. This procedure is provided in 1.
Since the input to the neural network \( V_e(s_t, c_t; \rho)\) and \( D(s_t, c_t; \nu)\) includes not only the state \( s_t\) but also the action \( c_t\) , this simulation process updates the values not just for the state-action pairs \( (s_t, a_t)\) on the optimal path, but for all pairs within its neighborhood.
Case 1: Discounted Linear Exponential Quadratic Gaussian Control. In this benchmark case, the control network outputs unbounded real-valued actions (e.g., via linear or ReLU activations), and exploration is enhanced through Gaussian-distributed noise injection. Specifically, for a subset of simulated paths, we perturb the network-generated action \( c_t\) as:
where \( \zeta\) is a scaling hyperparameter and \( I\) denotes the identity matrix. To maintain numerical stability, the perturbed values are clipped to a prescribed feasible range when necessary, ensuring compatibility with the underlying physical or economic constraints.
Case 2: High-Dimensional Control with Low-Discrepancy Sequences. For problems where the control space is high-dimensional and standard Gaussian perturbations may lead to inefficient exploration, we employ a quasi-Monte Carlo approach using Sobol sequences. This strategy generates deterministic, low-discrepancy perturbations that provide more uniform coverage of the control space. The perturbation is defined as:
where \( q_t \in [0,1]^{n_c}\) is a vector drawn from a \( n_c\) -dimensional Sobol sequence, and \( (2q_t - 1)\) linearly transforms the sequence to the interval \( (-1,1)^{n_c}\) . This approach yields a systematic exploration pattern that covers the control space more evenly than random Gaussian noise, particularly in high-dimensional settings, while maintaining the perturbation magnitude controlled by the scaling hyperparameter \( \zeta\) .
Algorithm 1 Exploratory Control Perturbation During Path Simulation
3.4.2 Target Networks for Stabilized Value Updates
A fundamental challenge in solving dynamic programming problems with recursive utility using deep learning stems from the inherent instability of bootstrapping in value function estimation. Recursive utility specifications, which nest nonlinear transformations of future utilities within current decisions, are particularly susceptible to this issue. When the same network parameters are used to both generate and fit temporal-difference targets, approximation errors can propagate and amplify through successive updates, leading to training divergence or oscillatory behavior.
To address this instability, we implement a target network architecture following the methodology pioneered by [77]. This mechanism maintains separate, slowly-evolving copies of key networks that are used exclusively for constructing bootstrapped targets. The targets are thereby decoupled from the rapidly changing online networks, introducing a form of hysteresis that improves learning stability. The parameters of these target networks are updated via Polyak averaging, ensuring a smooth and controlled evolution of the learning targets. This procedure is provided in 2.
Three-Network Architecture. In the three-network framework, we maintain a target network for the certainty-equivalent network. The parameters of this target network, denoted \( \bar{\xi}\) , are updated via soft averaging according to:
(36)
where \( \tau \ll 1\) is a small update rate that controls the degree of smoothing. This target network, \( \bar{V}_c(s_t, c_t; \bar{\xi})\) , is used to provide fixed targets for the value function update, leading to the stabilized objective:
(37)
Here, the certainty equivalent \( \bar{V}_c\) is evaluated using target parameters, effectively freezing the target during the value function optimization.
The same target-network construction applies to the two-network architecture, where the target certainty-equivalent network is used to construct the bootstrapped implicit value, while no separate value-network target is needed.
Four-Network Architecture. The extension to the four-network architecture necessitates maintaining target parameters for both the expected continuation network and nonlinear difference network. The update rules are given by:
(38)
The target certainty-equivalent value is then constructed as the combination:
(39)
This formulation yields the corresponding stabilized objective functions for policy and value learning:
(40)
Unified Update Procedure. The update of all target network parameters follows the same soft averaging scheme. For any online parameter set \( \omega\) and its target copy \( \bar{\omega}\) , the update is
(41)
where the relevant subset of parameters depends on the chosen network architecture. This procedure is summarized in Algorithm 2, which delineates the specific parameters updated in each architectural variant.
Algorithm 2 Target Network Soft Update Procedure
This target network framework follows a stabilization principle: the quantities used for bootstrapping must evolve on a slower timescale than the parameters being updated. By decoupling the target computation from the immediate gradient updates, the learning process benefits from a stationary objective, which is essential for convergent behavior in approximate value iteration schemes. The modifications outlined in this section—encompassing exploratory perturbation, continuation methods, and target networks—collectively form a robust numerical toolkit for solving recursive utility models with deep learning, addressing distinct sources of instability and inefficiency and improving convergence stability across a broad spectrum of economic environments.
3.4.3 Policy Network Delay
Inspired by the delayed policy update mechanism in the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, we implement a similar delayed update schedule for the policy network \( \theta\) . Specifically, we update the policy parameters \( \theta\) only once every \( d\) iterations, where \( d\) is a hyperparameter typically set between 1 and 5. During the intervening iterations, we continue to update the value function and certainty-equivalent networks, allowing for more accurate and stable value estimates before each policy improvement step.
This delay mitigates the accumulation of value estimation errors that can arise when the policy network changes too frequently. By updating the policy less often than the value networks, we ensure that the policy gradient steps are taken based on more reliable value estimates, leading to more stable convergence in training.
3.5 Algorithmic Framework and Training Procedure
To enhance numerical robustness and accelerate convergence in these challenging nonlinear optimization problems, the Certainty Equivalent Learning (CEL) algorithm incorporates several key methodological innovations: (1) exploratory control perturbation during path simulation for structured exploration; (2) target networks for stable bootstrapping in value function learning; and (3) delayed policy updates to mitigate value estimation errors.
CEL cycles through certainty-equivalent estimation, policy improvement, and, when an explicit value network is used, value-function fitting. In the three-network architecture the certainty-equivalent block is parameterized by \( \xi\) ; in the four-network architecture it is decomposed into \( (\rho,\nu)\) ; and in the two-network architecture the value function is represented implicitly and the explicit value-update step is omitted. The complete procedure alternates between these phases until convergence.
The optimization proceeds through a sequential reinforcement learning procedure, where the CEL parameters are updated in a fixed cyclic order to improve stability and convergence. The enhanced update procedure is as follows:
3.5.1 Three-Network Architecture
-
Simulation with exploratory control perturbation: For each optimization iteration \( k\) , we generate \( N\) trajectories of length \( T\) using the exploratory control perturbation mechanism described in Algorithm 1. Starting from initial states sampled from the estimated stationary distribution, we simulate:
\[ \begin{equation} \{\tilde{c}_t^{i}, s_t^{i}\}_{t=0}^{T-1} \sim \mathcal{P}(\cdot|s_0^{i}, \theta^{(k-1)}, \text{Method}, \zeta) \end{equation} \](42)
for \( i = 1, …, N\) , where Method \( \in\) {Gaussian, Sobol}. These perturbed trajectories cover the relevant regions of the state-action space while maintaining the policy’s dominance.
-
Update the certainty-equivalent network (\( \xi\) ): Given fixed value network parameters \( {\phi}^{(k-1)}\) , we update the certainty-equivalent network using the stabilized objective:
\[ \begin{equation} \min_{\xi \in \Theta^\xi} \frac{1}{N}\sum_{i=1}^{N} \left( {f\left(V_c(s_t^i, \tilde{c}_t^i; \xi)\right)} - \frac{1}{G} \sum_{j=1}^G {f\left( V(s_{t+1}^{i,j}; \phi^{(k-1)}) \right)} \right)^2. \end{equation} \](43)
where \( \tilde{c}_t^i\) denotes the perturbed control action generated during simulation using Algorithm 1. Note that the control variable \( \tilde{c}_t^i\) incorporates exploratory noise, ensuring that the certainty-equivalent network is trained on a diverse set of state-action pairs.
-
Update the policy network (\( \theta\) ): Given fixed parameters \( {\xi}^{(k)}\) , we optimize the policy parameters only if \( k \mod d = 0\) , where \( d\) is the delay hyperparameter. When updating, we maximize:
\[ \begin{equation} \max_{\theta \in \Theta^\theta} \frac{1}{N} \sum_{i=1}^{N} w\left( s_t^i, c(s_t^i; \theta), V_c(s_t^i, c(s_t^i; \theta); {\xi}^{(k)}) \right). \end{equation} \](44)
This delayed update schedule prevents the accumulation of value estimation errors and promotes more stable convergence.
-
Update the value network (\( \phi\) ): Given fixed policy parameters \( \theta^{(k)}\) and certainty equivalent parameters \( \bar{\xi}^{(k-1)}\) , we refine the value function by optimizing:
\[ \begin{equation} \min_{\phi \in \Theta^\phi} \frac{1}{N}\sum_{i=1}^{N} \left( {V(s_t^i; \phi)} - w\left( s_t^i, c(s_t^i; {\theta}^{(k)}), V_c(s_t^i, c(s_t^i; {\theta}^{(k)}),\bar{\xi}^{(k-1)})\right) \right)^2. \end{equation} \](45)
This objective uses target networks to provide stable learning targets, preventing training divergence.
-
Soft update target networks: After updating the online networks, we perform soft updates of the target networks using Polyak averaging:
\[ \begin{align} \bar{\xi}^{(k)} = \tau \xi^{(k)} + (1 - \tau)\bar{\xi}^{(k-1)} \\\end{align} \](46)
where \( \tau \ll 1\) is a small update rate controlling the degree of smoothing. This ensures that target parameters evolve slowly, providing stable learning targets.
3.5.2 Four-Network Architecture
The four-network architecture follows the same simulation, delayed policy update, value-fitting, and target-network logic as the three-network architecture. The only difference is that the certainty-equivalent value is decomposed into an expected continuation component and a nonlinear difference component:
(47)
Thus, instead of learning \( V_c\) directly, the algorithm learns \( V_e\) and \( D\) separately.
Given the same perturbed state-control samples \( \{(s_t^i,\tilde c_t^i)\}_{i=1}^N\) generated by Algorithm 1, the expected continuation network is trained by
(48)
Conditional on the updated expected continuation network, the nonlinear difference network is trained by
(49)
The policy network is updated, subject to the same delayed-update schedule as in the three-network architecture, by maximizing
(50)
When an explicit value network is used, it is fitted to the stabilized Bellman target constructed from the target expectation and difference networks:
(51)
The target parameters are updated by Polyak averaging,
(52)
Therefore, relative to the three-network architecture, the four-network architecture changes only the representation of the certainty-equivalent value. The common CEL loop—exploratory simulation, certainty-equivalent learning, delayed policy improvement, value fitting, and target-network updating—remains unchanged.
3.5.3 Two-Network Architecture
The two-network architecture is a compact version of CEL that removes the explicit value network. It contains only a policy network \( c(s_t;\theta)\) and a certainty-equivalent network \( V_c(s_t,c_t;\xi)\) . The value function is represented implicitly by the recursive aggregator:
(53)
This architecture therefore replaces the independent value approximation \( V(s_t;\phi)\) with the implicit value representation \( \widehat V(s_t;\theta,\xi)\) .
At iteration \( k\) , simulated state-control pairs are generated using the same exploratory perturbation procedure as before:
(54)
For each pair \( (s_t^i,\widetilde c_t^i)\) , the next-period states are generated by
(55)
Since there is no value network, the next-period value used in the certainty-equivalent target is evaluated through the implicit value function in (25)–(27). To stabilize the bootstrap, the certainty-equivalent component uses the target certainty-equivalent network:
(56)
The certainty-equivalent network is then trained by
(57)
The policy network is updated, again subject to the delayed-update schedule, by maximizing
(58)
Finally, the target certainty-equivalent network is updated by
(59)
The two-network architecture therefore omits the value-fitting step entirely. Relative to the three-network architecture, it is more compact and enforces consistency between the policy and value representation, but it can be less flexible because the value function is no longer learned as an independent object.
Algorithm 3 CEL Algorithm with Three-Network Architecture
Algorithm 4 CEL Algorithm with Four-Network Architecture
Algorithm 5 CEL Algorithm with Two-Network Architecture
4 Numerical Results
This section evaluates the accuracy and reliability of the value and policy functions learned by the proposed CEL variants. Assessing solution quality in dynamic programming problems requires problem-specific metrics, as the appropriate validation approach depends on the availability of closed-form benchmarks. When an analytical solution exists—common in linear-quadratic or exponential-utility specifications—we directly compare the learned functions against the ground truth, reporting pointwise errors and global deviation measures. When the reference is a numerical benchmark rather than the exact solution, we report differences rather than errors. Such comparisons offer the most transparent evidence of numerical accuracy.
In the more typical case where no closed-form solution is available, we instead measure the extent to which the learned solution satisfies the model’s Bellman equation and optimality conditions. We focus on problem-specific diagnostic metrics, including Bellman errors, Euler residuals, FOC residuals, and static-condition errors. The Bellman error quantifies the discrepancy between the two sides of the Bellman equation. Euler and FOC residuals evaluate violations of the model’s intertemporal or intratemporal optimality conditions, while static-condition errors provide additional model-specific consistency checks. These measures are zero at the exact solution; hence, small values indicate that the learned policy and value functions approximately satisfy the relevant optimality restrictions.
Computing these diagnostics involves evaluating conditional expectations of future values under the learned policy. We employ a nested simulation approach: for each state encountered along a test trajectory, we simulate multiple independent future paths, average the relevant next-period quantities, and evaluate the right-hand side of the Bellman equation. This simulation-based expectation estimation is flexible and model-agnostic, avoiding strong parametric assumptions about state transitions or noise distributions. By using a sufficiently large number of inner simulations, we obtain precise estimates of the Bellman errors and the relevant optimality-condition residuals, thereby obtaining reliable evidence on the accuracy of our numerical solution.
To further validate the learned value function, we use a nested-simulation diagnostic inspired by [52]. The idea is to simulate multiple future states from the current state, compute the simulated certainty-equivalent value, and compare the value-network output with this simulation-based estimate. By calculating the standard error of the simulated values, we construct a confidence interval for the value obtained from the right-hand side of the Bellman equation. If the learned value function lies within this interval, the result provides additional evidence that the computed solution is consistent with the Bellman equation.
Building on the evaluation framework above, we now present the numerical results of the proposed deep learning algorithm for dynamic programming problems with recursive utility. We consider four main applications that cover linear-quadratic control, robust control with a nonlinear certainty-equivalent term, DSGE models with stochastic volatility, and strategic portfolio choice with predictable returns.
The first application is a Gaussian linear-quadratic control problem with a closed-form benchmark. This experiment is used to evaluate whether the proposed method can accurately recover both the value function and the policy function when the true solution is known. We first study a benchmark case with \( n_s=8\) and \( n_c=4\) , and then consider a high-dimensional case with \( n_s=100\) and \( n_c=50\) to examine scalability.
The second application is a small-noise robust control problem with nonlinear recursive utility and model uncertainty. We first consider a general nonlinear case without an analytical solution and compare the learned solution with value function iteration (VFI). We then study a homothetic benchmark case with an analytical solution, which allows us to directly evaluate the approximation errors of the value and policy functions. This application is designed to test whether the proposed method remains accurate and stable when robustness concerns and nonlinear certainty-equivalent values become important.
The third application is a DSGE model with recursive preferences and stochastic volatility. This environment is richer than the previous benchmark problems because it contains a multidimensional state vector, multiple controls, and a nonlinear equilibrium structure. We use this example to examine whether the proposed algorithm can solve macroeconomic dynamic programming problems with recursive preferences in which traditional grid-based methods become computationally expensive and sensitive to interpolation errors.
The fourth application is a multivariate strategic asset allocation problem with predictable returns. In this model, the investor jointly chooses the consumption-wealth ratio and a vector of portfolio weights in response to a multivariate predictive state vector. This problem allows us to evaluate the performance of the proposed framework in a high-dimensional portfolio-choice environment with recursive preferences. We assess the learned normalized value function, consumption policy, portfolio policy, and several internal accuracy measures, including Bellman errors and static-condition errors.
Unless otherwise stated, the figure legends in this section use the following notation. closedform denotes the analytical benchmark. CEL denotes the direct deep-learning value or policy network output learned by CEL. CEL-td denotes the value obtained from a one-period evaluation of the right-hand side of the Bellman equation, where the certainty-equivalent term is evaluated through Monte Carlo simulation of next-period states and then plugged into the Bellman right-hand side. CEL-vc denotes the corresponding value obtained from the same one-period evaluation, but with the certainty-equivalent term replaced by the learned certainty-equivalent network. Thus, the closeness among CEL, CEL-td, and CEL-vc provides an internal consistency check for the learned solution.
The results are organized as follows. Section 4.1 reports the Gaussian linear-quadratic control results. Section 4.2 reports the small-noise robust control results. Section 4.3 reports the DSGE results with recursive preferences and stochastic volatility. Section 4.4 reports the multivariate strategic asset allocation results.
4.1 Gaussian Control: Closed-Form Benchmark and Scalability
The first example builds on the discounted risk-sensitive linear regulator framework developed by [9], which extends the classical linear-quadratic Gaussian (LQG) control problem by incorporating exponential risk adjustments into the cost functional. This model is widely used in engineering and robust control, and provides a tractable benchmark for testing the accuracy of CEL under Gaussian dynamics and convex quadratic costs.
4.1.1 Problem Setup and Network Representation
The system evolves according to a linear stochastic difference equation:
(60)
where \( s_t \in \mathbb{R}^{n_s}\) denotes the state vector, \( c_t \in \mathbb{R}^{n_c}\) denotes the control (or action) vector, and \( \{w_t\}\) is an i.i.d. sequence of standard Gaussian shocks. The initial state \( s_0\) is given exogenously.
The decision maker evaluates outcomes under a risk-sensitive recursive utility specification with an exponential certainty-equivalent term. The objective is to minimize the resulting exponential-quadratic cost functional:
(61)
where \( Q \succeq 0\) and \( R \succ 0\) are penalty matrices on controls and states, respectively, \( \beta \in (0,1)\) is the discount factor, and \( \sigma > 0\) parameterizes risk sensitivity. The specification retains the quadratic structure of the standard LQG problem while introducing recursive risk aversion via the exponential certainty equivalent.
For this benchmark, the state is the vector \( s_t\in\mathbb{R}^{n_s}\) , the control is \( c_t\in\mathbb{R}^{n_c}\) , and the recursive utility is the exponential-quadratic specification in (61). Here \( c_t\) denotes the generic control vector in the benchmark, rather than consumption. This example is a canonical validation case because the infinite-horizon solution is linear and time invariant, so the learned value and policy functions can be compared directly with the closed-form benchmark.
To solve this problem using deep learning, we implement the three-network CEL architecture:
where \( \phi\) , \( \theta\) , and \( \xi\) are the parameters of the value network, policy network, and certainty-equivalent network, respectively. This matches the structure of the Gaussian benchmark, where the nonlinear risk-sensitive certainty-equivalent value can be learned directly by \( V_c\) without introducing a separate nonlinear difference network.
For the benchmark experiment, we set \( n_s=8\) , \( n_c=4\) , \( \sigma=1\) , and \( \beta=0.96\) .
In this experiment, the system is specified by the transition matrices \( A \in \mathbb{R}^{8 \times 8}\) and \( B \in \mathbb{R}^{8 \times 4}\) , the covariance-related matrix \( C \in \mathbb{R}^{8 \times 8}\) , the state cost matrix \( R \in \mathbb{R}^{8 \times 8}\) , and the control cost matrix \( Q \in \mathbb{R}^{4 \times 4}\) . The full numerical specification is reported in Appendix 6.2.
4.1.2 Benchmark Case: \( n_s=8\) , \( n_c=4\)
Value function approximation. Figure 1 reports the learned value function \( V(s_t)\) for the Gaussian benchmark case with \( n_s=8\) and \( n_c=4\) . In this experiment, we vary the first component of \( s_t\) while fixing all other state variables at their steady-state values. The figure compares the direct value network from the deep learning method with the analytical closed-form benchmark and the one-step certainty-equivalent estimates.
The value-function curves are almost indistinguishable across the evaluation domain. In particular, the CEL value function closely matches the closed-form benchmark, indicating that the value network accurately captures the level, curvature, and scale of the true value function. The certainty-equivalent estimates are also closely aligned with the direct value network, which suggests that the certainty-equivalent components learned by the auxiliary networks are internally consistent with the Bellman equation.
To further examine Bellman consistency, we conduct a nested-simulation diagnostic test inspired by [52], and derive the 95% confidence interval using the delta method. The result is shown in Figure 2. The learned value function and the certainty-equivalent function both lie within the confidence band over the evaluation range. This provides additional evidence that the deep learning solution satisfies the Bellman equation.
Figure 3 reports the relative error between the learned value function and the closed-form benchmark. The relative error remains below \( 10^{-3}\) across the state range. This confirms that the learned value function is not only visually close to the analytical benchmark, but also quantitatively accurate. Therefore, the proposed method provides an accurate approximation to the value function in the moderate-dimensional Gaussian benchmark case.
Policy function approximation. The policy function \( c(s_t)\) is evaluated by varying the first component of \( s_t\) while fixing the remaining state variables at their steady-state values. Figure 4 compares the learned CEL policy with the analytical closed-form benchmark.
The learned policy closely tracks the closed-form benchmark over the entire evaluation range. This indicates that the policy network successfully learns the structure of the optimal control. Therefore, the proposed method is able to recover not only the value function, but also the economically relevant policy function.
Figure 5 reports the absolute error between the learned policy and the closed-form benchmark. The policy error remains small across the state range, showing that the policy approximation is quantitatively accurate.
4.1.3 High-Dimensional Case: \( n_s=100\) , \( n_c=50\)
We next consider a high-dimensional Gaussian control problem with \( n_s=100\) and \( n_c=50\) . This experiment is designed to examine whether the proposed method remains accurate when both the state and control dimensions become large. Such a setting is challenging for traditional grid-based methods because the number of grid points grows exponentially with the dimension. Therefore, this experiment provides a direct test of the scalability of the proposed deep learning algorithm.
Value function approximation. Figure 6 reports the learned value function in the high-dimensional case. We vary the first component of \( s_t\) while fixing all other state variables at their steady-state values. The figure compares the learned CEL value function with the closed-form benchmark and the one-step certainty-equivalent estimates.
The value-function curves are closely aligned across the full evaluation range. In particular, the learned CEL value function remains almost indistinguishable from the closed-form benchmark. The certainty-equivalent estimates also stay close to the direct value-network output, suggesting that the certainty-equivalent components of the learned solution remain internally consistent even in the high-dimensional environment.
To further examine whether the learned value function satisfies the Bellman equation, we conduct the same nested-simulation diagnostic test as in the benchmark case. The result is reported in Figure 7. The learned value function and certainty-equivalent approximation lie within the 95% confidence band. This provides additional evidence that the learned solution remains Bellman-consistent when the state dimension increases to 100.
Figure 8 reports the relative error between the learned value function and the closed-form benchmark. The relative error remains below \( 10^{-4}\) across the evaluation range. This shows that the value approximation is not only visually close to the analytical solution, but also quantitatively accurate in a high-dimensional state space.
Policy function approximation. Figure 9 presents the learned policy function in the high-dimensional case. We again vary the first component of \( s_t\) while fixing the remaining state variables at their steady-state values. The learned policy closely follows the closed-form benchmark, indicating that the policy network can still recover the optimal policy function when the control dimension increases to 50.
Figure 10 reports the absolute error between the learned policy and the closed-form benchmark. The policy error remains small over the state range, confirming that the learned neural network solution produces accurate control decisions even in the high-dimensional environment.
4.2 Small Noise Robust Control: General and Homothetic Cases
Our second example builds on the work of [78], who study robust decision-making under model misspecification and uncertainty. In this framework, a representative agent possesses risk-sensitive preferences as introduced by [9]. This approach departs from standard expected utility theory by introducing a penalty for potential misperceptions about the stochastic structure of the economy. The penalty is implemented through a log-exponential transformation of future utility, thereby internalizing a form of robustness in intertemporal planning.
4.2.1 Problem Setup and Network Representation
The agent’s problem is characterized by the following recursive utility specification:
(62)
where \( \gamma > 0\) denotes the coefficient of relative risk aversion, \( \sigma > 0\) parameterizes the sensitivity to model uncertainty (with higher \( \sigma\) implying greater concern for robustness), and \( \beta \in (0,1)\) is the subjective discount factor. The agent faces a standard Cobb-Douglas resource constraint:
(63)
where \( C_t\) is consumption, \( K_t\) is the end-of-period capital stock, \( \alpha \in (0,1)\) is the capital share, \( \delta \in [0,1]\) is the depreciation rate, and \( P\) scales the technology shock. The exogenous technology shock \( a_t\) follows a linear Gaussian process:
(64)
with persistence parameter \( \Omega_a \in (-1,1)\) , unconditional mean \( \Omega_0 / (1 - \Omega_a)\) , and conditional volatility \( \sqrt{\epsilon} \, \Omega_v\) .
For the deep learning implementation, the state is \( s_t=(K_{t-1},a_t)^\top\) , and the control is the relative consumption share
(65)
where \( W_t\) denotes total resources. The constraint \( c_t\in(0,1)\) guarantees positive consumption and investment, and the capital law of motion becomes \( K_t=(1-c_t)W_t\) .
To capture the nested recursive utility structure, we employ a four-network architecture:
This problem provides a tractable yet rich test case for CEL. In the general nonlinear specification, no analytical solution is available, so we use VFI as a numerical benchmark. In the homothetic benchmark specification, the model admits an analytical solution, which allows us to evaluate the numerical errors of both CEL and VFI directly.
4.2.2 General Nonlinear Case without Analytical Solution
We first consider the general nonlinear case without an analytical solution. In this setting, value function iteration is used as a numerical benchmark. Since VFI is itself a numerical approximation rather than the exact solution, the comparison below is reported as a relative difference rather than an approximation error. The purpose of this comparison is to examine whether the deep learning solution delivers value and policy functions that are close to the VFI benchmark and whether the residual-based diagnostics are of a similar magnitude.
Results for baseline robustness: \( \sigma=1\) .
Value function approximation. The value function \( V(s_t)\) is evaluated across varying capital \( K_{t-1}\) , fixing the technology shock at its steady state \( a_{ss}\) . Figure 11 compares the learned value function and its one-step expansion from the CEL method with the VFI benchmark.
The learned value function is close to the VFI benchmark over the whole capital range. The one-step expansion is also closely aligned with the direct value network, indicating that the value function approximation is internally consistent. This suggests that the proposed method captures the nonlinear curvature induced by recursive preferences, capital accumulation, and model uncertainty.
Figure 12 reports the relative difference between the CEL value network and the VFI benchmark. The relative difference remains small across the evaluation range. This shows that the CEL value function is quantitatively close to the VFI benchmark.
Policy function approximation. Figure 13 displays the learned consumption policy as capital varies, with the current technology shock fixed at its steady state. Figure 14 reports the policy function in terms of the consumption-to-wealth ratio.
The level of consumption increases with capital, which is consistent with economic intuition. The consumption-to-wealth ratio provides a normalized view of the consumption-saving decision and shows how the agent adjusts the policy as capital changes. The nonlinear shape of the policy reflects the role of model uncertainty and recursive preferences in the intertemporal trade-off.
Figure 15 reports the relative difference between the learned policy and the VFI policy. Since the VFI solution is used as a numerical benchmark, this figure should be interpreted as a comparison between two numerical methods rather than as an error relative to the true solution. The small relative difference indicates that the proposed method approximates not only the value function but also the economically relevant policy function.
Euler residual and relative Bellman error. We further evaluate the learned solution using two diagnostic measures: the Euler residual and the relative Bellman error. The Euler residual evaluates whether the learned consumption-saving policy satisfies the intertemporal optimality condition, while the relative Bellman error evaluates whether the learned value function is consistent with the Bellman equation after normalizing the Bellman discrepancy by the right-hand side of the Bellman equation. The formulas used to compute these diagnostic measures are reported in Appendix 6.3.
Figure 16 reports the Euler residual. The Euler residual from the CEL method is comparable to that from VFI across the evaluation range. This suggests that the learned policy satisfies the intertemporal optimality condition to a similar degree as the VFI benchmark.
Figure 17 reports the Bellman error. The Bellman error from the CEL method is also comparable to that obtained from VFI. This indicates that the learned value function is consistent with the Bellman equation at a level similar to the reported grid-based benchmark.
Results for high robustness: \( \sigma=50\) .
We next consider a higher-robustness case with \( \sigma=50\) . A larger value of \( \sigma\) strengthens the recursive exponential adjustment and makes the problem numerically more demanding. This case therefore provides a useful robustness check for the proposed method in the general nonlinear environment.
Value function approximation. Figure 18 compares the learned value function and its one-step expansion with the VFI benchmark for \( \sigma=50\) .
The learned value function remains close to the VFI benchmark under high robustness. The direct value network and the one-step expansion also remain closely aligned, showing that the recursive approximation remains stable when the robustness parameter increases.
Figure 19 reports the relative difference between the CEL value network and the VFI benchmark. As before, this figure is interpreted as a relative difference between two numerical solutions rather than as an error relative to the exact solution. The small relative difference indicates that the CEL method remains close to the VFI benchmark in the high-robustness case.
Policy function approximation. Figures 20 and 21 report the learned policy function and the consumption-to-wealth ratio for \( \sigma=50\) .
The policy functions remain smooth across the capital range. This suggests that the policy network continues to produce stable consumption decisions even when the recursive utility adjustment becomes stronger.
Figure 22 reports the relative difference between the CEL policy and the VFI policy. The difference remains controlled across the state range, suggesting that the proposed method produces policy functions that are close to the VFI benchmark under high robustness.
Euler residual and Bellman error. Figure 23 reports the Euler residual for \( \sigma=50\) . The Euler residual produced by the CEL method is of a similar magnitude to that obtained from VFI. This indicates that the learned policy satisfies the first-order optimality condition at a level comparable to the VFI benchmark.
Figure 24 reports the Bellman error. The Bellman error from the CEL method is also comparable to that of VFI, suggesting that the learned value function satisfies the Bellman equation to a similar degree as the reported grid-based benchmark. Together, the Euler residual and Bellman error indicate that the proposed method remains numerically stable under stronger robustness concerns.
Overall, the general nonlinear small-noise results show that the proposed deep learning method provides value and policy functions that are close to the VFI benchmark. The residual-based diagnostics further show that the Euler residuals and Bellman errors of the CEL and VFI solutions are of similar magnitude. Since the general nonlinear case does not admit an analytical solution, these comparisons should be interpreted as validation against a standard numerical benchmark rather than as a direct ranking relative to the true solution.
4.2.3 Homothetic Benchmark Case with Analytical Solution
We next consider a homothetic benchmark case with an analytical solution. This case is useful because it allows us to compare both the CEL and VFI solutions directly with the analytical benchmark. Instead of describing it simply as a case with a closed-form solution, we refer to it as a homothetic benchmark case because the analytical tractability comes from the special homothetic structure of the model.
A closed-form solution exists for this optimization problem under the following special case:
(66)
where the logarithmic utility and the simplified production and shock structure generate a homothetic solution. This analytical benchmark allows us to evaluate the numerical accuracy of the value function and policy function more directly than in the general nonlinear case.
The model is trained with state \( s_t=(K_{t-1},a_t)\) , control \( c_t\in(0,1)\) , discount factor \( \beta=0.9\) , and robustness parameters \( \sigma=50\) and \( \sigma=200\) . In the error plots below, the CEL solution, the VFI solution, and the analytical solution are evaluated at the VFI capital grid points. Therefore, the reported relative errors compare the numerical methods with the analytical solution at the same evaluation points.
Results for moderate robustness: \( \sigma=50\) .
Figures 25, 26, and 27 report the value function, policy function, and consumption-to-wealth ratio for the case \( \sigma=50\) . The learned value function closely follows the analytical solution, and the policy function also tracks the benchmark well.
Figure 25 shows that the CEL value function is very close to the analytical benchmark. The one-step certainty-equivalent expansion is also closely aligned with the direct value network, indicating that the learned value function approximation is internally consistent. Compared with the VFI benchmark, the CEL value function is visibly closer to the analytical solution over most of the evaluation range.
Figures 26 and 27 show that both CEL and VFI recover the policy function. The CEL policy is close to the analytical benchmark and appears smoother over the evaluation range. The comparison suggests that the proposed method can accurately recover the consumption-saving decision in this homothetic benchmark case.
Figures 28 and 29 plot the relative errors with respect to the analytical solution for the value function and policy function. The errors are evaluated at the VFI capital grid points, and the vertical axis uses a logarithmic scale to display the magnitude of the errors.
Figure 28 shows that the CEL value-function error is generally lower than that of the VFI benchmark. Over most of the capital grid, the CEL error is around or below \( 10^{-4}\) , whereas the VFI error is typically between \( 10^{-4}\) and \( 10^{-3}\) . There is a localized deviation near the lower boundary of the grid, but away from this boundary region the CEL value approximation remains closer to the analytical solution.
Figure 29 shows that the CEL policy-function error is generally lower and smoother than that of the VFI benchmark. The VFI policy error exhibits visible oscillations across the capital grid, while the CEL policy error remains smoother over most of the evaluation range. This pattern suggests that the neural policy approximation can reduce some of the local oscillations associated with grid-based policy approximation.
Results for high robustness: \( \sigma=200\) .
We further consider a high-robustness case with \( \sigma=200\) . A larger value of \( \sigma\) increases the strength of the recursive exponential adjustment and makes the problem numerically more challenging. This case therefore provides a useful robustness check for the proposed method.
Figures 30, 31, and 32 report the value function, policy function, and consumption-to-wealth ratio. The learned value and policy functions remain close to the analytical benchmark, showing that the method is stable even when the robustness parameter is large.
Figure 30 shows that the CEL value function remains close to the analytical benchmark under high robustness. The certainty-equivalent expansion also stays aligned with the direct value network, suggesting that the recursive structure is still well captured when the robustness parameter becomes large.
Figures 31 and 32 show that the learned policy remains stable and close to the analytical benchmark. Similar to the \( \sigma=50\) case, the CEL policy is smooth and tracks the analytical solution over the capital range. This suggests that the proposed method can recover the main shape of the policy function even under stronger robustness concerns.
Figures 33 and 34 report the relative errors with respect to the analytical solution. As in the \( \sigma=50\) case, all relative errors are evaluated at the VFI capital grid points.
Figure 33 shows that the CEL value-function error remains generally lower than that of the VFI benchmark. The CEL error is mostly around or below \( 10^{-4}\) , while the VFI value-function error is typically between \( 10^{-4}\) and \( 10^{-3}\) . Thus, even in the high-robustness case, the proposed method provides a more accurate value-function approximation in most regions of the evaluation grid.
Figure 34 shows that the CEL policy-function error is generally lower and smoother than the VFI policy-function error. The oscillatory pattern in the VFI policy error is less pronounced in the CEL solution, indicating that the learned policy network provides a smoother approximation to the analytical policy.
Accuracy comparison.
The homothetic benchmark results show a consistent pattern across both \( \sigma=50\) and \( \sigma=200\) . Because an analytical solution is available, the reported quantities are true relative errors with respect to the closed-form benchmark, evaluated at the VFI capital grid points. For the value function, the CEL approximation is generally more accurate than the VFI benchmark, with relative errors mostly around or below \( 10^{-4}\) , compared with VFI errors that are typically between \( 10^{-4}\) and \( 10^{-3}\) . For the policy function, the CEL approximation also delivers lower and smoother relative errors over most of the evaluation range, while the VFI policy error exhibits visible local oscillations.
Overall, the homothetic benchmark case confirms that the proposed deep learning method can accurately solve recursive utility problems when an analytical benchmark is available. In contrast to the general nonlinear case, where VFI is only a numerical benchmark, this closed-form case permits a direct error comparison. The results suggest that CEL achieves accuracy comparable to, and often better than, the VFI benchmark in both value-function and policy-function approximation, especially away from localized boundary regions.
4.3 DSGE Model with Recursive Preferences and Stochastic Volatility
Our third example is adapted from [79], who study a dynamic stochastic general equilibrium (DSGE) model with recursive preferences and time-varying volatility in total factor productivity (TFP). This framework extends the standard neoclassical growth model by incorporating [38, 80]-style preferences and a stochastic volatility process, making it particularly suited for studying business cycle fluctuations and long-run asset pricing phenomena. The model’s key innovation lies in its ability to separate risk aversion from intertemporal substitution while capturing time-varying economic uncertainty.
4.3.1 Problem Setup and Network Representation
The representative agent’s preferences are specified through the following recursive utility function:
(67)
where \( \nu\) governs the consumption-leisure trade-off, \( \gamma\) measures risk aversion, and \( \theta = \frac{1-\gamma}{1-1/\psi}\) captures the deviation from standard CRRA preferences, with \( \psi\) denoting the elasticity of intertemporal substitution (EIS). This recursive formulation allows for distinct roles of risk aversion and EIS, enabling more flexible modeling of investor behavior under uncertainty.
The production side follows a standard Cobb-Douglas technology:
(68)
where \( \zeta\) represents the capital share and \( \delta\) is the depreciation rate. The total factor productivity shock \( z_t\) evolves as:
(69)
Here, \( \sigma_t\) follows a stationary AR(1) process with persistence \( \rho\) and innovation volatility \( \eta\) , introducing stochastic volatility into the economic system. This specification captures the time-varying uncertainty in productivity shocks, which helps generate realistic asset pricing implications and business cycle dynamics.
For the deep learning implementation, the state is \( s_t=(K_t,z_t,\sigma_t)^\top\) , and the controls are the consumption-to-resources ratio \( c_t\) and labor supply \( L_t\) . The consumption ratio is
(70)
We impose \( c_t,L_t\in(0,1)\) through the final-layer activation functions, which guarantees positive consumption and feasible labor allocations.
To capture the nested recursive utility structure, we employ a four-network architecture:
The two policy components, \( c(s_t;\theta_c)\) and \( L(s_t;\theta_L)\) , are grouped as the policy-network block in the four-network CEL architecture.
This DSGE framework presents a computationally challenging test case due to its three-dimensional state space and the interaction between recursive preferences and stochastic volatility. The model’s solution requires accurately capturing precautionary saving motives and time-varying risk premia, making it an ideal benchmark for evaluating CEL’s performance in a rich economic environment.
4.3.2 Numerical Results
In this subsection, we evaluate the learned value function, policy functions, and several performance measures for the DSGE model. Let \( K_{ss}\) denote the steady-state capital stock. In all figures below, we vary capital \( K_t\) over the interval \( [0.5K_{ss},1.5K_{ss}]\) , while fixing the other state variables at their steady-state values. This allows us to examine how the learned solution changes with capital while keeping productivity and volatility constant.
Figure 35 reports the learned value function. The figure compares the direct value network, denoted by CEL, with the value estimates obtained from the temporal-difference update and the certainty-equivalent component, denoted by CEL-td and CEL-vc, respectively. The three curves are very close to each other over the full range of capital. This indicates that the value function has converged well and that the different components of the certainty-equivalent approximation are internally consistent.
Figure 36 reports the learned policy functions. The two policy components are the consumption-to-wealth ratio \( c_t\) and labor supply \( L_t\) . The consumption-to-wealth ratio describes how much of total available resources is allocated to current consumption, while the labor policy describes the optimal labor supply decision. The smoothness of both policy functions suggests that the neural network learns stable policy functions over the capital range.
Figure 37 reports the relative Bellman error. The relative Bellman error measures whether the learned value function is consistent with the equation after normalizing the Bellman discrepancy by the right-hand side of the Bellman equation. In this experiment, the relative Bellman error remains below \( 10^{-4}\) across the evaluation range. This provides strong evidence that the learned value function satisfies the Bellman equation with high numerical accuracy.
Figures 38 and 39 report two optimality-condition diagnostics. The consumption Euler residual evaluates the intertemporal optimality condition for consumption and investment, while the labor FOC residual evaluates the intratemporal first-order condition for labor supply. The formulas used to compute the relative Bellman error and these optimality-condition residuals are reported in Appendix 6.4. Both residuals remain below \( 10^{-3}\) throughout the capital range, indicating that the learned consumption, investment, and labor decisions are close to satisfying the model’s first-order optimality conditions.
Finally, Figure 40 reports the static condition error. This condition is derived from the homothetic structure of the DSGE model, which implies a relationship between the value function and consumption. Therefore, the static condition provides an additional performance measure for checking the internal consistency of the learned solution. The error remains below \( 10^{-3}\) over the full capital range, indicating that the learned value and policy functions also satisfy this homotheticity-based restriction with high accuracy.
Overall, the DSGE results show that the proposed deep learning method can accurately solve a model with recursive preferences, stochastic volatility, and two controls. The close agreement among the value-function approximations, together with small Bellman errors, consumption Euler residuals, labor FOC residuals, and static condition errors, suggests that the learned solution is both numerically accurate and internally consistent.
4.4 Multivariate Strategic Asset Allocation with Predictable Returns
Our fourth example considers a multivariate strategic asset allocation problem with predictable returns, following the framework of [13]. The investor has Epstein–Zin recursive preferences and chooses consumption and portfolio weights in an environment where asset returns are driven by a vector autoregression. This example is useful for evaluating whether the proposed method can solve portfolio-choice problems with recursive preferences, multiple predictive state variables, and multiple controls.
4.4.1 Problem Setup and Network Representation
The investor’s value function is
(71)
where \( \gamma\) is the coefficient of relative risk aversion, \( \delta\) is the discount factor, and \( \theta\) controls the elasticity of intertemporal substitution. Wealth evolves according to
(72)
The predictive state vector follows a vector autoregression (VAR(1)) process:
(73)
The state vector \( \mathbf{z}_t\) includes the short-term return, excess returns on risky assets, and predictive variables such as dividend-price ratios and yield spreads.
Given portfolio weights \( \boldsymbol{\alpha}_t\) on risky assets, the gross portfolio return is
(74)
where \( R_{1,t+1}\) is the gross return on the short-term benchmark asset and \( \mathbf{R}_{t+1}\) is the vector of gross returns on risky assets.
The original problem is written in terms of both wealth \( W_t\) and the predictive state \( \mathbf{z}_t\) . However, Epstein–Zin preferences are homothetic, so the value function is homogeneous of degree one in wealth. We can therefore write
(75)
where \( V(\mathbf{z}_t)\) is the normalized value function per unit of wealth. Define the consumption-wealth ratio
(76)
Then the normalized wealth transition is
(77)
Substituting (75) and (77) into the Bellman equation and dividing both sides by \( W_t\) removes wealth from the state space. The normalized stationary problem becomes
(78)
subject to the VAR law of motion (73) and the feasibility constraints
(79)
Equivalently, since \( c_t\) is chosen at time \( t\) , the term \( (1-c_t)\) can be taken outside the conditional expectation:
(80)
This is the formulation used in the numerical implementation. The key simplification is that the state variable is now only \( \mathbf{z}_t\) , rather than \( (W_t,\mathbf{z}_t)\) . The policy functions are also normalized: the algorithm learns the consumption-wealth ratio \( c(\mathbf{z}_t)\) and the portfolio policy function \( \boldsymbol{\alpha}(\mathbf{z}_t)\) .
To approximate the solution, we use a four-network architecture:
The consumption policy function \( c(\mathbf{z}_t;\theta_c)\) and the portfolio policy function \( \boldsymbol{\alpha}(\mathbf{z}_t;\theta_{\alpha})\) are grouped as the policy-network block in the four-network CEL architecture.
4.4.2 Numerical Results
In this subsection, we report the numerical results for the multivariate strategic asset allocation problem. The results are computed under the calibration with risk aversion \( \gamma=20\) . All figures are presented in a \( 2\times 3\) subplot format. In each panel, one state variable is varied over a range of \( \pm 3\) stationary standard deviations around its steady-state mean, while all other state variables are fixed at their means. This design allows us to isolate the dependence of the value function, consumption policy, and portfolio policy on each component of the predictive state vector.
Value function approximation. Figure 41 reports the normalized value function. Each of the six panels varies one dimension of the state vector while keeping the remaining state variables fixed at their means. The value function is smooth across all dimensions, indicating that the value network captures the dependence of continuation utility on the predictive state variables in a stable manner. The shape of the value function also reflects the effect of time-varying investment opportunities encoded in the VAR state vector.
Overall, Figure 41 shows that the learned normalized value function behaves regularly across the state space and exhibits a stable response to changes in the predictive variables. This suggests that the proposed deep learning method is able to approximate the value function accurately in a multivariate asset allocation environment.
Policy function approximation: consumption ratio. Figure 42 reports the policy function for the consumption-wealth ratio \( c_t\) . As in the value-function figure, each panel varies one state variable while fixing the remaining state variables at their means. The learned consumption policy function is smooth across all dimensions, which indicates that the policy network produces stable consumption decisions over the relevant region of the state space.
The smooth variation of the consumption-wealth ratio across the six dimensions suggests that the model successfully captures how intertemporal consumption decisions respond to changes in expected returns and other predictive state variables. Since the consumption policy function is a key object in the Epstein–Zin problem, this result provides additional support for the accuracy of the learned policy function.
Policy function approximation: portfolio weights. Figure 43 reports the portfolio-weight policy function. Each panel shows how the optimal asset allocation responds to changes in one state variable, while all other state variables are held fixed at their means. The resulting portfolio policy functions are state-dependent and respond systematically to changes in the predictive variables, reflecting the fact that the investor adjusts the asset allocation in response to time-varying expected returns and risks.
Figure 43 shows that the learned portfolio policy functions respond systematically to the predictive state vector. This is economically important because the main feature of strategic asset allocation is precisely that portfolio weights vary with the investment opportunity set. The systematic variation of the learned policy suggests that the neural network approximation captures economically meaningful state dependence in portfolio choice.
Performance measures. To further assess the numerical quality of the solution, Figure 44 reports two performance measures: the Bellman error and the static condition error. The Bellman error evaluates how well the learned value function satisfies the Bellman equation. The static condition error provides an additional internal-consistency check for the policy functions and the value-function approximation. Since the value function is small in magnitude in this calibration, the Bellman error is reported in absolute terms.
Both performance measures are very small over most of the evaluation range, with the majority of the reported values below \( 10^{-4}\) . This indicates that the networks are well trained and that the learned solution is highly consistent with the underlying problem with recursive utility. In particular, the small Bellman errors suggest that the value function approximation is accurate, while the small static condition errors indicate that the policy functions are also internally coherent.
Overall, the results for the multivariate strategic asset allocation problem show that the proposed deep learning framework can successfully solve a high-dimensional portfolio-choice problem with recursive preferences and predictable returns. The learned value function, consumption policy function, and portfolio weights all respond regularly to the predictive state vector, and the small Bellman and static condition errors provide strong evidence of numerical accuracy.
5 Conclusion
This paper proposes the first Certainty Equivalent Learning (CEL) algorithm, a deep learning framework for solving high-dimensional discrete-time dynamic programming problems with recursive utility. The central idea is to approximate the certainty-equivalent value directly as a separate state-control function that enters the Bellman equation. Instead of treating the nonlinear certainty-equivalent term as an object that must be recomputed repeatedly during policy and value updates, CEL represents the relevant value functions, policy functions, and certainty-equivalent values with neural networks. This provides a mesh-free and simulation-based approach for nonlinear, stochastic, and high-dimensional decision problems with recursive utility.
The proposed framework includes several architectural variants. The three-network version learns the value function, policy function, and certainty-equivalent value. The four-network version further decomposes the certainty-equivalent value into an expected continuation component and a nonlinear difference component, which is useful when the certainty-equivalent term involves strong nonlinear transformations. The two-network version removes the explicit value network and defines the current value implicitly through the policy and certainty-equivalent networks. These variants should be viewed as different decompositions of the same problem. The algorithm can also be combined with target-network updates for certainty-equivalent components and randomized exploration to stabilize bootstrapped training and policy search. Importantly, CEL does not rely on Euler equations, first-order conditions, perturbation around a steady state, or problem-specific projection bases.
The numerical results lead to several main conclusions. First, in the Gaussian linear-quadratic benchmark, the proposed method accurately recovers both the value function and the policy function. In the benchmark case with \( n_s=8\) and \( n_c=4\) , the learned value function closely matches the closed-form solution, and the relative value-function error remains small across the evaluation range. The learned policy also tracks the analytical benchmark well, indicating that the policy network successfully recovers the optimal-control structure. In the high-dimensional case with \( n_s=100\) and \( n_c=50\) , the method continues to perform well: the value function remains close to the closed-form benchmark, the relative error stays below \( 10^{-4}\) , and the learned policy remains accurate. These results demonstrate both the accuracy and scalability of the proposed algorithm.
Second, in the general nonlinear small-noise robust control problem, where no analytical solution is available, the learned value and policy functions are close to the VFI benchmark. Since VFI is itself a numerical approximation, the comparison is interpreted as a relative difference rather than as an error against the true solution. For both the baseline case with \( \sigma=1\) and the high-robustness case with \( \sigma=50\) , the CEL solution remains close to the VFI benchmark in value and policy functions. The Euler residuals and Bellman errors are also of a similar magnitude to those obtained from the VFI benchmark. These results suggest that the proposed method remains stable as robustness concerns become stronger and can recover solutions consistent with a standard low-dimensional numerical benchmark.
Third, in the homothetic small-noise benchmark case with an analytical solution, both the CEL and VFI solutions can be evaluated directly against the closed-form benchmark. The relative errors are computed at the VFI capital grid points, so the two numerical methods are compared with the analytical solution at the same evaluation points. Across both \( \sigma=50\) and \( \sigma=200\) , the CEL value-function approximation is generally more accurate than the VFI benchmark, with relative errors mostly around or below \( 10^{-4}\) , compared with VFI errors that are typically between \( 10^{-4}\) and \( 10^{-3}\) . The CEL policy-function approximation also displays lower and smoother relative errors over most of the evaluation range, while the VFI policy error exhibits visible local oscillations. These results indicate that, in the analytical homothetic benchmark, CEL achieves accuracy comparable to, and often better than, the VFI benchmark, especially away from localized boundary regions.
Fourth, in the DSGE model with recursive preferences and stochastic volatility, the proposed method successfully handles a richer macroeconomic environment with a three-dimensional state vector and two controls. The learned value function is internally consistent across the direct value network, the temporal-difference update, and the certainty-equivalent approximation. The learned consumption and labor policies are smooth over the capital range. Moreover, the Bellman error remains below \( 10^{-4}\) , while the consumption Euler residual and the labor FOC residual remain below \( 10^{-3}\) . The former evaluates the intertemporal optimality condition for consumption and investment, whereas the latter evaluates the intratemporal first-order condition for labor supply. The static condition error also remains below \( 10^{-3}\) . These results suggest that the learned solution satisfies the Bellman equation, the intertemporal and intratemporal optimality conditions, and the homotheticity-based static restriction with high numerical accuracy.
Finally, in the multivariate strategic asset allocation problem with predictable returns, the proposed framework is also able to solve a high-dimensional portfolio-choice problem with recursive preferences. The learned normalized value function, consumption-wealth ratio, and portfolio weights respond systematically to the predictive state variables. The Bellman error and static condition error are mostly below \( 10^{-4}\) , indicating that the learned value function and policy functions are internally consistent with the Bellman equation and the model-specific optimality restrictions.
Overall, these results show that the proposed deep learning framework can solve a wide range of high-dimensional discrete-time dynamic programming problems with recursive utility and nonlinear certainty-equivalent values. The method accurately approximates both value and policy functions, remains stable under stronger robustness concerns, scales to high-dimensional state and control spaces, and performs well in richer DSGE and portfolio-choice environments where traditional grid-based methods become computationally expensive or infeasible. In particular, the small Bellman errors, consumption Euler residuals, labor FOC residuals, and static condition errors provide evidence that the learned solutions are not only regular in shape but also internally consistent with the underlying optimality conditions.
Taken together, the findings suggest that CEL provides a practical computational tool for high-dimensional discrete-time dynamic programming with recursive utility in macroeconomics and finance. By learning certainty-equivalent functions directly, the method addresses a key numerical difficulty created by nonlinear certainty-equivalent terms. CEL’s mesh-free neural-network structure allows the method to scale to high-dimensional state spaces and complex stochastic environments where conventional value function iteration and projection methods can become computationally expensive or sensitive to interpolation choices.
The CEL algorithm can also be applied for dynamic programming problems with expected utility, as expected utility is a special case of recursive utility.
6 Appendix
This appendix collects supplementary material that supports the model, algorithm, and numerical analysis in the main text. It first summarizes the neural-network notation and parameter choices, then provides analytical derivations for the benchmark models, and finally defines the diagnostic measures used to evaluate Bellman errors and optimality residuals in the numerical experiments.
A.1 Notation for Neural-Network Objects
Table 1 summarizes the main neural-network objects and parameter symbols used in the CEL framework. In applications with multiple control components, the policy-network entry represents the policy block, which may contain several policy subnetworks.
| Symbol | Meaning | Parameter |
| \( V(s_t;\phi)\) | Value network | \( \phi\) |
| \( c(s_t;\theta)\) | Policy network | \( \theta\) |
| \( V_c(s_t,c_t;\xi)\) | Certainty-equivalent network | \( \xi\) |
| \( V_e(s_t,c_t;\rho)\) | Expected continuation network | \( \rho\) |
| \( D(s_t,c_t;\nu)\) | Nonlinear difference network in the decomposition \( V_c=V_e-D\) | \( \nu\) |
| \( \bar V_c\) , \( \bar V_e\) , \( \bar D\) | Target-network versions used for stabilized updates | \( \bar\xi,\bar\rho,\bar\nu\) |
| \( \widehat V(s_t;\theta,\xi)\) | Implicit value in the two-network architecture | \( (\theta,\xi)\) |
A.2 Parameters
This appendix reports the full numerical specification used in the high-dimensional experiment in Section 4.1. The state dimension is \( n_s=8\) , the control dimension is \( n_c=4\) , and the noise covariance matrix is \( \Sigma = I_8\) .
The state transition matrix is
(81)
The control matrix is
(82)
The covariance-related matrix is
(83)
The state cost matrix is
(84)
The control cost matrix is
(85)
A.3 Euler Residual and Relative Bellman Error for the Small-Noise Robust Control Model
This appendix defines the diagnostic residuals used to evaluate the learned solution of the small-noise robust control model. These residuals are used only as ex-post accuracy measures and are not used as training objectives in CEL.
Let the state be \(s_t=(K_{t-1},a_t)\), where \(K_{t-1}\) is predetermined capital and \(a_t\) is the current technology state. The production technology and resource constraint are given by
(86)
In the numerical implementation, the policy network outputs the consumption-to-resources ratio \(c_t\in(0,1)\). Hence,
(87)
The technology state follows
(88)
The recursive utility specification is
(89)
Define the risk-sensitive change-of-measure term by
(90)
The gross marginal return on capital is
(91)
The standard Euler equation for the small-noise robust control model can be written as
(92)
Accordingly, we define the Euler residual as
(93)
At the exact solution, \(\mathcal R^E(s_t)=0\) for all states.
We also define the relative Bellman error. Given the learned value function and policy function, define the Bellman right-hand side as
(94)
The relative Bellman error is then defined as the Bellman discrepancy normalized by the right-hand side of the Bellman equation:
(95)
At the exact solution, \(\mathcal R^B(s_t)=0\) for all states.
In the numerical implementation, the conditional expectations in (90), (93), and (94) are approximated by sample means over simulated next-period shocks. We report the absolute residuals \(|\mathcal R^E(s_t)|\) and \(|\mathcal R^B(s_t)|\), or their logarithmic transformations, as diagnostic measures of solution accuracy.
6.4 Relative Bellman Error, Euler Residual, and Labor Optimality Residual for the DSGE Model
This appendix defines the diagnostic residuals used to evaluate the learned solution of the DSGE model with recursive utility. These residuals are used only as ex-post accuracy measures and are not used as training objectives in CEL.
Let the state be \(s_t=(K_t,z_t,\sigma_t)\), and let \(C_t\) and \(L_t\) denote consumption and labor implied by the learned policy functions. The production technology and resource constraint are given by
(96)
The one-period utility composite is
(97)
We write the Epstein–Zin aggregator as
(98)
Define
(99)
Given the learned policy functions and value function, define the Bellman right-hand side by
(100)
The relative Bellman error is the Bellman discrepancy normalized by the right-hand side of the Bellman equation:
(101)
At the exact solution, \(\mathcal R^B(s_t)=0\) for all states.
Define the current marginal utility term by
(102)
where
(103)
The gross marginal return on capital is
(104)
The standard Euler equation for capital with Epstein–Zin preferences can be written as
(105)
Accordingly, we define the capital Euler residual as
(106)
At the exact solution, \(\mathcal R^K(s_t)=0\) for all states.
We also evaluate the intratemporal labor optimality condition. The marginal product of labor is
(107)
The labor first-order condition equates the marginal product of labor to the marginal rate of substitution between leisure and consumption:
(108)
Equivalently,
(109)
We therefore define the labor FOC error, or labor optimality residual, as
(110)
At the exact solution, \(\mathcal R^L(s_t)=0\) for all states.
In the numerical implementation, the conditional expectations in \(M_t\), in the relative Bellman error (101), and in the Euler residual (106) are approximated by sample means over simulated next-period shocks. We report the absolute residuals \(|\mathcal R^B(s_t)|\), \(|\mathcal R^K(s_t)|\), and \(|\mathcal R^L(s_t)|\), or their logarithmic transformations, as diagnostic measures of solution accuracy.
References
[1] Santos, M. S. (2000) . Accuracy of numerical solutions using the Euler equation residuals, Econometrica 68(6): 1377–1402.
[2] Azinovic, M., Gaegauf, L. and Scheidegger, S. ( 2022) . Deep equilibrium nets, International Economic Review 63(4): 1471–1525.
[3] Scheidegger, S. (2026) .
Deep learning for solving and estimating dynamic models in economics
and finance.
https://arxiv.org/abs/2605.14493
[4] Weil, P. (1989) . The equity premium puzzle and the risk-free rate puzzle, Journal of Monetary Economics 24(3): 401–421.
[5] Bansal, R. and Yaron, A. (2004) . Risks for the long run: A potential resolution of asset pricing puzzles, The Journal of Finance 59(4): 1481–1509.
[6] Duffie, D. and Epstein, L. G. (1992) . Stochastic differential utility, Econometrica 60: 353–394.
[7] Schroder, M. and Skiadas, C. (1999) . Optimal consumption and portfolio selection with stochastic differential utility, Journal of Economic Theory 89(1): 68–126.
[8] Kaplan, G. and Violante, G. L. (2014) . A model of the consumption response to fiscal stimulus payments, Econometrica 82(4): 1199–1239.
[9] Hansen, L. P. and Sargent, T. J. ( 1995) . Discounted linear exponential quadratic gaussian control, IEEE Transactions on Automatic control 40(5): 968–971.
[10] Hansen, L. P. and Sargent, T. J. ( 2008) . Robustness, Princeton university press.
[11] Hansen, L. P. and Sargent, T. J. ( 2013) . Recursive Models of Dynamic Linear Economies, Princeton University Press.
[12] Campbell, J. Y. and Viceira, L. M. ( 1999) . Consumption and portfolio decisions when expected returns are time varying, The Quarterly Journal of Economics 114(2): 433–495.
[13] Campbell, J. Y., Chan, Y. L. and Viceira, L. M. ( 2003) . A multivariate model of strategic asset allocation, Journal of financial economics 67(1): 41–80.
[14] Cai, Y. and Lontzek, T. S. (2019) . The social cost of carbon with economic and climate risks, Journal of Political Economy 127(6): 2684–2734.
[15] Zhao, Y., Basu, A., Lontzek, T. S. and Schmedders, K. (2023) . The social cost of carbon when we wish for full-path robustness, Management Science 69(12): 7585–7606.
[16] Kraft, H., Seifried, F. T. and Steffensen, M. ( 2013) . Consumption-portfolio optimization with recursive utility in incomplete markets, Finance and Stochastics 17: 161–196.
[17] Kraft, H., Seiferling, T. and Seifried, F. T. ( 2017) . Optimal consumption and investment with epstein–zin recursive utility, Finance and Stochastics 21: 187–226.
[18] Xing, H. (2017) . Consumption–investment optimization with epstein–zin utility in incomplete markets, Finance and Stochastics 21: 227–262.
[19] Matoussi, A. and Xing, H. (2018) . Convex duality for epstein–zin stochastic differential utility, Mathematical Finance 28(4): 991–1019.
[20] Feng, Z. and Tian, D. (2021) . Optimal consumption and portfolio selection with epstein-zin utility under general constraints, arXiv preprint arXiv:2111.09032 .
[21] Feng, Z., Tian, D. and Zheng, H. ( 2024) . Consumption-investment optimization with epstein-zin utility in unbounded non-markovian markets, arXiv preprint arXiv:2407.19995 .
[22] Garlappi, L. and Skoulakis, G. (2010) . Solving consumption and portfolio choice problems: The state variable decomposition method, The Review of Financial Studies 23(9): 3346–3400.
[24] Cai, Y., Judd, K. and Xu, R. (2020) .
Numerical solution of dynamic portfolio optimization with transaction
costs.
https://arxiv.org/abs/2003.01809
[25] Dixon, M., Gvozdanovic, I. and O’Kane, D. ( 2025) . Time-consistent reinforcement learning for optimal consumption under Epstein–Zin preferences, SIAM Journal on Financial Mathematics 16(3): 1139–1175.
[26] Tallarini, T. D. (2000) . Risk-sensitive real business cycles, Journal of Monetary Economics 45(3): 507–532.
[27] Andreasen, M. M. (2012) . On the effects of rare disasters and uncertainty shocks for risk premia in non-linear dsge models, Review of Economic Dynamics 15(3): 295–316.
[28] Cao, D., Luo, W. and Nie, G. (2023a) . Global DSGE models, Review of Economic Dynamics 51: 199–225.
[29] Cao, D., Luo, W. and Nie, G. (2023b) . Uncovering the effects of the zero lower bound with an endogenous financial wedge, American Economic Journal: Macroeconomics 15(1): 135–172.
[30] Duarte, V., Fonseca, J., Goodman, A. S. and Parker, J. A. (2021) . Simple allocation rules and optimal portfolio choice over the lifecycle, Technical report, National Bureau of Economic Research.
[31] Azinovic, M. and Jan Žemlička ( 2023) . Economics-inspired neural networks with stabilizing homotopies, arXiv . Available at https://arxiv.org/abs/2303.14802 .
[32] E, W., Han, J. and Jentzen, A. (2017) . Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations, Communications in Mathematics and Statistics 5(4): 349–380.
[33] Beck, C., E, W. and Jentzen, A. ( 2019) . Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations, Journal of Nonlinear Science 29(4): 1563–1619.
[34] Huré, C., Pham, H., Bachouch, A. and Langrené, N. (2021) . Deep neural networks algorithms for stochastic control problems on finite horizon: Convergence analysis, SIAM Journal on Numerical Analysis 59(1): 525–557.
[35] Bachouch, A., Huré, C., Langrené, N. and Pham, H. (2022) . Deep neural networks algorithms for stochastic control problems on finite horizon: Numerical applications, Methodology and Computing in Applied Probability 24(1): 143–178.
[36] Reppen, A. M., Soner, H. M. and Tissot-Daguette, V. ( 2023) . Deep stochastic optimization in finance, Digital Finance 5(1): 91–111.
[37] Kreps, D. M. and Porteus, E. L. ( 1978) . Temporal resolution of uncertainty and dynamic choice theory, Econometrica 46(1): 185–200.
[38] Epstein, L. G. and Zin, S. E. (1989) . Substitution, risk aversion, and the temporal behavior of consumption and asset returns: a theoretical framework, Econometrica 57(4): 937–969.
[39] Weil, P. (1990) . Nonexpected utility in macroeconomics, The Quarterly Journal of Economics 105(1): 29–42.
[40] Stokey, N. L., Lucas, R. E. and Prescott, E. C. ( 1989) . Recursive Methods in Economic Dynamics, Harvard University Press.
[41] Ljungqvist, L. and Sargent, T. J. ( 2018) . Recursive Macroeconomic Theory, 4 edn, MIT Press.
[42] Miao, J. (2020) . Economic Dynamics in Discrete Time, MIT Press, Cambridge, Massachusetts.
[43] Friedl, A., Kübler, F., Scheidegger, S. and Usui, T. (2023) . Deep uncertainty quantification: with an application to integrated assessment models, Technical report, Working Paper University of Lausanne.
[44] Hansen, L. P. and Scheinkman, J. A. ( 2012) . Recursive utility in a markov environment with stochastic growth, Proceedings of the National Academy of Sciences 109(30): 11967–11972.
[45] Christensen, T. M. (2022) . Existence and uniqueness of recursive utilities without boundedness, Journal of Economic Theory 200: 105413.
[46] Pohl, W., Schmedders, K. and Wilms, O. ( 2024) . Existence of the wealth-consumption ratio in asset pricing models with recursive preferences, The Review of Financial Studies 37(3): 989–1028.
[47] Stachurski, J., Wilms, O. and Zhang, J. ( 2024) . Asset pricing with time preference shocks: Existence and uniqueness, Journal of Economic Theory 216: 105781.
[48] Stachurski, J. and Zhang, J. (2021) . Dynamic programming with state-dependent discounting, Journal of Economic Theory 192: 105190.
[49] Ma, Q. and Stachurski, J. (2021) . Dynamic programming deconstructed: Transformations of the bellman equation and computational efficiency, Operations Research 69(5): 1591–1607.
[50] Ma, Q., Stachurski, J. and Toda, A. A. ( 2022) . Unbounded dynamic programming via the q-transform, Journal of Mathematical Economics 100: 102652.
[51] Jaśkiewicz, A. and Nowak, A. S. (
2024) .
Stochastic dynamic programming under recursive epstein-zin
preferences.
https://arxiv.org/abs/2410.19181
[52] Judd, K. L. (1998) . Numerical Methods in Economics, MIT Press, Cambridge, MA.
[53] Miranda, M. J. and Fackler, P. L. ( 2002) . Applied Computational Economics and Finance, MIT Press, Cambrideg, MA.
[54] Carroll, C. (2006) . The method of endogenous gridpoints for solving dynamic stochastic optimization problems, Economics Letters 91(3): 312–320.
[55] Carroll, C. (2011) . Lecture notes on solution methods for representative agent dynamic stochastic optimization problems, Preprint, Johns Hopkins University.
[56] Carroll, C. (2022) . Solution methods for microeconomic dynamic stochastic optimization problems, Preprint, Johns Hopkins University.
[57] Coeurdacier, N., Rey, H. and Winant, P. ( 2020) . Financial integration and growth in a risky world, Journal of Monetary Economics 112: 1–21.
[58] Han, J. and E, W. (2016) . Deep learning approximation for stochastic control problems.
[59] Renner, P. and Scheidegger, S. (2018) . Machine learning for dynamic incentive problems, Available at SSRN 3282487 .
[60] Scheidegger, S. and Bilionis, I. ( 2019) . Machine learning for high-dimensional dynamic stochastic economies, Journal of Computational Science 33: 68–82.
[61] Lepetyuk, V., Maliar, L. and Maliar, S. ( 2020) . When the US catches a cold, Canada sneezes: A lower-bound tale told by deep learning, Journal of Economic Dynamics and Control 117: 103926.
[62] Valaitis, V. and Villa, A. T. (2024) . A machine learning projection method for macro-finance models, Quantitative Economics 15(1): 145–173.
[63] Maliar, L., Maliar, S. and Winant, P. ( 2021) . Deep learning for solving dynamic economic models., Journal of Monetary Economics 122: 76–101.
[64] Pascal, J. (2024) . Artificial neural networks to solve dynamic programming problems: A bias-corrected Monte Carlo operator, Journal of Economic Dynamics and Control 162: 104853.
[65] Skavysh, V., Priazhkina, S., Guala, D. and Bromley, T. R. (2023) . Quantum Monte Carlo for economics: Stress testing and macroeconomic deep learning, Journal of Economic Dynamics and Control 153: 104680.
[66] Han, J., Yang, Y. and E, W. (2022) . Deepham: A global solution method for heterogeneous agent models with aggregate shocks, arXiv . Available at https://arxiv.org/abs/2112.14377.
[67] Hall-Hoffarth, E. (2023) . Non-linear approximations of DSGE models with neural-networks and hard-constraints, arXiv . Available at https://arxiv.org/abs/2310.13436.
[68] Huang, J. (2023) . A probabilistic solution to high-dimensional continuous-time macro and finance models, Available at SSRN 4538048 .
[69] Huang, J. (2024) . Breaking the curse of dimensionality in heterogeneous-agent models: A deep learning-based probabilistic approach, Available at SSRN 4649043 .
[70] Duarte, V., Duarte, D. and Silva, D. H. ( 2024) . Machine learning for continuous-time finance, The Review of Financial Studies 37(11): 3217–3271.
[71] Sutton, R. S. and Barto, A. G. (1998) . Reinforcement Learning: An Introduction, MIT Press, Cambridge, Massachusetts.
[72] Powell, W. B. (2011) . Approximate Dynamic Programming: Solving the Curses of Dimensionality, 2 edn, John Wiley and Sons, Hoboken, New Jersey.
[73] Bertsekas, D. P. (2012) . Dynamic Programming and Optimal Control: Approximate Dynamic Programming, Vol. II, 4 edn, Athena Scientific, Belmont, Massachusetts.
[74] Feng, Z., Han, J. and Zhu, S. (2023) . Optimal taxation with incomplete markets: an exploration via reinforcement learning, Available at SSRN 4758552 .
[75] Aboussalah, A. M., Xu, Z. and Lee, C.-G. ( 2022) . What is the value of the cross-sectional approach to deep reinforcement learning?, Quantitative Finance 22(6): 1091–1111.
[76] Jiang, Y., Olmo, J. and Atwi, M. ( 2025) . High-dimensional multi-period portfolio allocation using deep reinforcement learning, International Review of Economics & Finance 98: 103996.
[77] Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D. and Wierstra, D. (2015) . Continuous control with deep reinforcement learning.
[78] Anderson, E. W., Hansen, L. P. and Sargent, T. J. ( 2012) . Small noise methods for risk-sensitive/robust economies, Journal of Economic Dynamics and Control 36(4): 468–500.
[79] Caldara, D., Fernandez-Villaverde, J., Rubio-Ramirez, J. F. and Yao, W. (2012) . Computing dsge models with recursive preferences and stochastic volatility, Review of Economic Dynamics 15(2): 188–206.
[80] Epstein, L. G. and Zin, S. E. (1991) . Substitution, risk aversion, and the temporal behavior of consumption and asset returns: An empirical analysis, Journal of political Economy 99(2): 263–286.