Post-training assessment

The function assess can be used to assess a trained estimator. The resulting Assessment object contains ground-truth parameters, estimates, and other quantities that can be used to compute quantitative and qualitative diagnostics.

NeuralEstimators.assess Function

assess(estimator, θ, Z; ...)
assess(estimators::Vector, θ, Z; ...)
assess(estimator::PriorEstimator, θ; N = size(θ, 2), ...)
assess(estimator::PriorEstimator; N = 1000, ...)

Assesses an estimator (or a collection of estimators) based on true parameters θ and corresponding simulated data Z.

The parameters θ should be given as a × matrix, where is the parameter dimension and is the number of sampled parameter vectors.

When Z contains more simulated data sets than the number of sampled parameter vectors, θ will be recycled via horizontal concatenation: θ = repeat(θ, outer = (1, J)), where J = numobs(Z) ÷ K is the number of simulated data sets for each parameter vector. This allows assessment of the estimator's sampling distribution under fixed parameters.

The return value is of type Assessment.

A PriorEstimator is assessed without data. Instead, N draws from the learned distribution are compared with test samples θ from the true distribution, if given as a × matrix (by default, N is , or 1000 if no test samples are given), and the risk is the average negative log density of the test samples under the learned distribution (see risk). The draws can be visualised with plot(assessment). The keyword arguments parameter_names, estimator_name, device, and use_gpu (the latter two passed to sample and logdensity) also apply.

Keyword arguments

• estimator_name::String (or estimator_names::Vector{String} for multiple estimators): name(s) of the estimator(s) (sensible defaults provided).

• parameter_names::Vector{String}: names of the parameters (sensible default provided).

• use_gpu = true: Bool or Vector{Bool} with length equal to the number of estimators.

• probs = nothing (applicable only to PointEstimator): probability levels taking values between 0 and 1. By default, no bootstrap uncertainty quantification is done; if probs is provided, it must be a two-element vector specifying the lower and upper probability levels for non-parametric bootstrap intervals (note that parametric bootstrap is not currently supported with assess()).

• B::Integer = 400 (applicable only to PointEstimator): number of bootstrap samples.

• pointsummary::Function = mean (applicable only to estimators that yield posterior samples): a function that summarises a vector of posterior samples into a single point estimate for each marginal; any function mapping a vector to a scalar is valid (e.g., median for the posterior median).

• kwargs... (applicable only to estimators that yield posterior samples): additional keyword arguments passed to sampleposterior.

source

NeuralEstimators.Assessment Type

Assessment

A type for storing the output of assess(). The field runtime contains the total time taken for each estimator. The field estimates is a long-form DataFrame with columns:

• parameter: the name of the parameter

• truth: the true value of the parameter

• estimate: the estimated value of the parameter

• k: the index of the parameter vector

• j: the index of the data set (only relevant in the case that multiple data sets are associated with each parameter vector)

If the estimator is a PosteriorEstimator or a RatioEstimator, in addition to the fields listed above, the field samples stores the posterior samples as a long-form DataFrame with the columns parameter, truth, k, j (as given above), as well as:

• draw: the index of the draw within the posterior samples

• value: the value of the posterior sample for a given parameter and draw.

If the estimator is an IntervalEstimator, the column estimate will be replaced by the columns lower and upper, containing the lower and upper bounds of the interval, respectively.

If the estimator is a QuantileEstimator, there will also be a column prob indicating the probability level of the corresponding quantile estimate.

If the estimator is a PriorEstimator, the field estimates is nothing, and the field samples is a long-form DataFrame with the columns parameter, draw, and value (as given above), and distribution, which is "learned" for draws from the learned distribution and "true" for test samples from the true distribution. The field risk then contains the average negative log density of the test samples under the learned distribution (or nothing if no test samples were given).

Use merge() to combine assessments from multiple estimators of the same type or join() to combine assessments from a PointEstimator and an IntervalEstimator.

source

Makie.plot Method

plot(assessment::Assessment; plots = nothing, ...)

Visualise the performance of a neural estimator, given the Assessment object returned by assess. Returns a Makie Figure with one panel for each parameter.

Extension

This function is defined in the NeuralEstimatorsPlottingMakieExt extension and requires CairoMakie (or another Makie backend) to be loaded.

The plots that are available depend on the type of estimator that was assessed. By default all of them are drawn; use the keyword argument plots to select a subset.

Point and interval estimates (PointEstimator, IntervalEstimator):

• :recovery: estimates against the true values. Accurate estimates lie close to the dashed identity line. For an IntervalEstimator, the intervals are drawn as vertical line segments; when the assessment contains both point estimates and intervals, the intervals are drawn only if intervals = true.

Quantile estimates (QuantileEstimator):

• :calibration: the proportion of estimated quantiles that are greater than the true value, against the nominal probability level τ. Specifically, for k = 1,…,K, sample pairs (θᵏ, Zᵏ) with θᵏ ∼ p(θ) and Zᵏ ∼ p(Z ∣ θᵏ), so that θᵏ is a draw from the posterior p(θ ∣ Zᵏ); then, for each τ, plot the proportion of the estimated quantiles Q(Zᵏ, τ) that are greater than θᵏ. A well-calibrated estimator follows the dashed identity line.

Posterior samples (PosteriorEstimator, RatioEstimator, TelescopingRatioEstimator):

• :recovery: point estimates (the pointsummary given to assess, by default the posterior mean) against the true values. If intervals = true, central 95% credible intervals are drawn as vertical line segments.

• :ecdf: simulation-based calibration. For each parameter, the empirical distribution function of the fractional rank of the true value among the posterior draws, together with a simultaneous prob-level confidence band (Säilynoja et al., 2022). A well-calibrated posterior gives a curve that stays within the band.

• :zscore: the posterior z-score, (posterior mean − true value) / posterior standard deviation, against the posterior contraction, 1 − posterior variance / prior variance. Ideally the z-scores are centred on zero and the contractions are close to one.

Learned distributions (PriorEstimator): draws from the learned distribution (filled histograms and coloured points), compared with test samples from the true distribution (outlined histograms and points in the text colour) if these were given to assess; otherwise, the learned distribution is shown alone.

• :pairs (default): with one parameter, a histogram; with two parameters, the two marginal histograms and a scatter plot of the joint draws; with three or more parameters, a pairs plot with the marginal histograms on the diagonal and scatter plots of the joint draws off the diagonal (see overlay).

• :marginals: the marginal histograms only, in a single row (see ncols).

Keyword arguments

• plots = nothing: the plots to draw, given as a Symbol or a collection of Symbols from those listed above, in the order in which they should appear (e.g., plots = (:recovery, :ecdf)). By default, all available plots are drawn. For a PriorEstimator, give one of :pairs (the default) or :marginals.

• intervals = false: whether to draw intervals behind the point estimates in the :recovery plot. They are not drawn by default, since they tend to hide the points when there are many test data sets.

• prob = 0.99: simultaneous coverage of the confidence band in the :ecdf plot.

• difference = true: if true, the :ecdf plot shows the difference between the empirical distribution function and that of the uniform distribution, which makes departures from calibration easier to see; if false, it shows the empirical distribution function itself.

• grid = false: when the assessment contains several estimators (see merge()), they are by default drawn in the same panels in different colours. If grid = true, each estimator is instead given its own row of panels, which is easier to read with more than three estimators.

• ncols = nothing: the number of panels in each row, after which the parameters wrap onto a new row. By default, at most four, balanced across rows; for the :marginals plot of a PriorEstimator, all panels are in one row.

• transpose = nothing: by default, the figure has one row of panels for each plot and one column for each parameter. If transpose = true, it instead has one row for each parameter and one column for each plot (e.g., 2 × 3 for two parameters and three plots), and ncols has no effect. With a single parameter, the figure is transposed unless transpose = false.

• overlay = false: for the :pairs plot of a PriorEstimator with three or more parameters, the learned draws are drawn below the diagonal and the test samples above it. If overlay = true, both are instead drawn below the diagonal, and the panels above it are left empty.

• figure = (;), axis = (;): attributes passed to the Figure and to every Axis, respectively. By default the panels are of a fixed size and the figure is sized to fit them; give figure = (; size = (w, h)) to fix the size of the figure instead. Colours and fonts are taken from the current Makie theme.

Examples

using NeuralEstimators, CairoMakie

# Given an estimator and test parameters and data (see `assess`)
assessment = assess(estimator, θ_test, Z_test)

plot(assessment)                               # all available plots
plot(assessment; plots = :recovery)            # a single plot
plot(assessment; plots = (:recovery, :ecdf))   # a subset (posterior samples)
plot(assessment; transpose = true)             # one row per parameter, one column per plot

# Learned distributions (PriorEstimator), with or without test samples from the true distribution
assessment = assess(estimator, θ_test)
plot(assessment)                               # pairs plot
plot(assessment; plots = :marginals)           # marginal histograms only

source

NeuralEstimators.risk Function

risk(assessment::Assessment; ...)

Computes a Monte Carlo approximation of an estimator's Bayes risk,

where    denotes a set of parameter vectors sampled from the prior and, for each , data are simulated from the statistical model conditional on .

If the Assessment object corresponds to an estimator with a self-defined loss (e.g., PosteriorEstimator), the precomputed risk is returned directly. For a PriorEstimator, this is the average negative log density of the test samples under the learned distribution. Otherwise, the risk is computed from the estimates and true parameters using the provided loss function.

Keyword arguments

• loss = (x, y) -> abs(x - y): a binary operator defining the loss function (default: absolute-error loss)

• average_over_parameters::Bool = false: if true, the loss is averaged over all parameters; otherwise (default), it is computed separately for each parameter.

source

NeuralEstimators.bias Function

bias(assessment::Assessment; average_over_parameters = false)

Computes a Monte Carlo approximation of an estimator's bias,

where    denotes a set of parameter vectors sampled from the prior and, for each , data are simulated from the statistical model conditional on .

source

NeuralEstimators.rmse Function

rmse(assessment::Assessment; average_over_parameters = false)

Computes a Monte Carlo approximation of an estimator's root-mean-squared error,

where    denotes a set of parameter vectors sampled from the prior and, for each , data are simulated from the statistical model conditional on .

source

NeuralEstimators.coverage Function

coverage(assessment::Assessment; ...)

Computes a Monte Carlo approximation of an interval estimator's expected coverage, as defined in Hermans et al. (2022, Definition 2.1), and the proportion of parameters below and above the lower and upper bounds, respectively.

Keyword arguments

• average_over_parameters::Bool = false: if true, the coverage is averaged over all parameters; otherwise (default), it is computed over each parameter separately.

source